caroline-tot-chrome-pfq-informational failed HWTest security_OpenFDs |
|||||||||
Issue descriptioncaroline-tot-chrome-pfq-informational builds failed security_OpenFDs constantly starting from 11/29 (PFQ builds are quite red these days, so it's been covered by other various issues), see the starting build: https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/caroline-tot-chrome-pfq-informational/builds/405 Selected error messages: 11-29-2017 [20:07:30] Output below this line is for buildbot consumption: @@@STEP_LINK@[Test-Logs]: security_OpenFDs: FAIL: Unexpected open file descriptors.@http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/159380693-chromeos-test/@@@ @@@STEP_LINK@[Flake-Dashboard]: security_OpenFDs@https://wmatrix.googleplex.com/retry_teststats/?days_back=30&tests=security_OpenFDs@@@ @@@STEP_LINK@[Test-History]: security_OpenFDs@https://wmatrix.googleplex.com/unfiltered?hide_missing=True&tests=security_OpenFDs@@@ Will return from run_suite with status: ERROR Also see the test suitcase: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/159380693-chromeos-test/chromeos6-row2-rack23-host13/security_OpenFDs/ Some logs from the test log: 2017-11-30T03:26:31.936409+00:00 NOTICE autotest[18585]: 19:26:31.933 ERROR| security_OpenFDs:0144| Some filter(s) failed to match any fds: set(['0[57]00 /dev/shm/..*', '0700 /dev/dri/renderD129', '0700 anon_inode:dmabuf']) 2017-11-30T03:26:31.939598+00:00 NOTICE autotest[18586]: 19:26:31.937 ERROR| security_OpenFDs:0147| Found unexpected fds in chrome type=renderer: set(['0700 /memfd: (deleted)']) 2017-11-30T03:26:31.944108+00:00 NOTICE autotest[18587]: 19:26:31.942 ERROR| security_OpenFDs:0144| Some filter(s) failed to match any fds: set(['0[57]00 /dev/shm/..*', '0700 /dev/dri/renderD129', '0700 anon_inode:dmabuf']) 2017-11-30T03:26:31.946757+00:00 NOTICE autotest[18588]: 19:26:31.944 ERROR| security_OpenFDs:0147| Found unexpected fds in chrome type=renderer: set(['0700 /memfd: (deleted)'])
,
Dec 4 2017
Autotest would have to be doing something very unexpected to be using the test runner code in //build/android/pylib...
,
Dec 5 2017
This started on the Chrome pfq with R64-10171.0.0-b405 https://stainless.corp.google.com/search?exclude_cts=true&exclude_non_release=false&board=%5Ecaroline%24&test=%5Esecurity%5C_OpenFDs%24&view=matrix&col=build&row=board&first_date=20171129&last_date=20171205 Found unexpected fds in chrome type=renderer: set(['0700 /memfd: (deleted)']) Time to bisect Chrome? (Then again this is a change detector test and 99% of the cases one can just add the newly detected issue to the whitelist.)
,
Dec 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/5cff7ff71242e0ecd983e19b6063b3a72c1db8bd commit 5cff7ff71242e0ecd983e19b6063b3a72c1db8bd Author: xdai <xdai@chromium.org> Date: Tue Dec 05 01:27:59 2017 Move the test security_OpenFDs from bvt-inline to bvt-perbuild. The test security_OpenFDs constantly failed on caroline-tot-chrome-pfq-informational. Move it to bvt-perbuild to unblock PFQ while I'm bisecting Chrome to figure out what might cause it. It will be reverted later. BUG= chromium:791786 TEST=none Change-Id: Id7e4ad60e98eb61a8aa96cabdd8ba76576f13a8c Reviewed-on: https://chromium-review.googlesource.com/807288 Trybot-Ready: Xiaoqian Dai <xdai@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> Tested-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/5cff7ff71242e0ecd983e19b6063b3a72c1db8bd/client/site_tests/security_OpenFDs/control
,
Dec 5 2017
Start bisecting
,
Dec 5 2017
Bisect is done. Culprit CL is https://chromium-review.googlesource.com/781160. Assign to the owner. dvallet@, could you take a look at it? Please check if it's expected.
,
Dec 6 2017
+security folks This security_OpenFDs test appears to be failing on the R64 release branch on systems with ARC++ at least so far. Do we know if this failure is benign or if this is something we would want to hold up the 64 dev release for?
,
Dec 6 2017
Currently we will operate under the assumption that this is not a critical failure that would block release based on the response to this bug being that we remove the test from the PFQ (e.g. this was considered an invalid blocking of the PFQ and thus not a blocker for Chrome). If this is not a valid assessment please let us know.
,
Dec 6 2017
The culprit CL does not seem to be ARC++ related. But more importantly Xiaoqian removed the test from PFQ *temporarily* while bisecting -- I don't think that means an invalid blocking of the PFQ. The CL looks benign, we should probably just update the test.
,
Dec 6 2017
Actually it is not just ARC++ systems as we see it in BayTrail, it may be systems on newer kernels? Removing the test temporarily would allow a potentially bad Chrome in though, so I would think that implies some belief the failures is benign. In any case if you think it is safe, then we can move forward with the 64 dev today so I think we are good.
,
Dec 6 2017
Newer kernels makes sense -- memfd does not exist on older kernels. We should update the test.
,
Dec 6 2017
I was not sure if this test is important or not. But since the failure only happened on caroline, and the culprit CL was landed in 11/29 which means there might be subsequent CLs that depending on the culprit CL. So I would prefer to remove the test temporarily other than reverting the CL without confirming with the owner. dvallet@, could you provide your opinion here?
,
Dec 6 2017
It's fine to use memfd's for shared memory here. We should update the test expectation.
,
Dec 6 2017
My apologies for all the extra work in bisecting the issue! The CL has already been reverted and it was deemed insecure , see https://bugs.chromium.org/p/chromium/issues/detail?id=792117 The CL affected Chrome on Linux based systems (including Chrome OS), so if your tests covered any of these I'd say they were effective. I'll mark this as fixed, feel free to reopen if there's anything else to do.
,
Dec 6 2017
Ah good point, I forgot that we couldn't really seal the memfd's. This is not fully done until the revert makes it to the branch and the test gets put back in the PFQ. Xiaoqian, is the test back in the PFQ? Daniel, did the revert land in M64 or M65?
,
Dec 6 2017
I'll bring the test back to PFQ. Danial, you might need to merge your revert CL to M64 if your revert CL landed in M65.
,
Dec 6 2017
Btw: I don't have permission to view Issue 792117
,
Dec 6 2017
792117 is the security bug tied to memfd's, I added you to it but I don't think it adds a lot of context here.
,
Dec 7 2017
FYI CL to re-enable the test security_OpenFDs: https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/812085
,
Dec 7 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/aa048119bc0cac2cde121e5f9574689dfc18e1b0 commit aa048119bc0cac2cde121e5f9574689dfc18e1b0 Author: Xiaoqian Dai <xdai@chromium.org> Date: Thu Dec 07 18:39:26 2017 Revert "Move the test security_OpenFDs from bvt-inline to bvt-perbuild." This reverts commit 5cff7ff71242e0ecd983e19b6063b3a72c1db8bd. Reason for revert: The CL that caused the failure has been reverted. Move this test back to bvt-inlie. Original change's description: > Move the test security_OpenFDs from bvt-inline to bvt-perbuild. > > The test security_OpenFDs constantly failed on caroline-tot-chrome-pfq-informational. > Move it to bvt-perbuild to unblock PFQ while I'm bisecting Chrome to figure out what > might cause it. It will be reverted later. > > BUG= chromium:791786 > TEST=none > > Change-Id: Id7e4ad60e98eb61a8aa96cabdd8ba76576f13a8c > Reviewed-on: https://chromium-review.googlesource.com/807288 > Trybot-Ready: Xiaoqian Dai <xdai@chromium.org> > Reviewed-by: Ilja H. Friedel <ihf@chromium.org> > Tested-by: Ilja H. Friedel <ihf@chromium.org> Bug: chromium:791786 Change-Id: I3fdf7eb2838bab616eb43b9a788db8ec7ed92b4d Reviewed-on: https://chromium-review.googlesource.com/812085 Reviewed-by: Ilja H. Friedel <ihf@chromium.org> Tested-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/aa048119bc0cac2cde121e5f9574689dfc18e1b0/client/site_tests/security_OpenFDs/control
,
Dec 7 2017
Thanks for the revert! Daniel, please confirm the state of your CL's revert on the M64 branch. Thanks!
,
Dec 8 2017
https://bugs.chromium.org/p/chromium/issues/detail?id=792117#c16 has the revert landed in the M64 branch, so the test should not fail anymore on M64. |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by x...@chromium.org
, Dec 4 2017