New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Dec 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

caroline-tot-chrome-pfq-informational failed HWTest security_OpenFDs

Project Member Reported by x...@chromium.org, Dec 4 2017

Issue description

caroline-tot-chrome-pfq-informational builds failed security_OpenFDs constantly starting from 11/29 (PFQ builds are quite red these days, so it's been covered by other various issues), see the starting build:  https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/caroline-tot-chrome-pfq-informational/builds/405 

Selected error messages:
   11-29-2017 [20:07:30] Output below this line is for buildbot consumption:
  @@@STEP_LINK@[Test-Logs]: security_OpenFDs: FAIL: Unexpected open file descriptors.@http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/159380693-chromeos-test/@@@
  @@@STEP_LINK@[Flake-Dashboard]: security_OpenFDs@https://wmatrix.googleplex.com/retry_teststats/?days_back=30&tests=security_OpenFDs@@@
  @@@STEP_LINK@[Test-History]: security_OpenFDs@https://wmatrix.googleplex.com/unfiltered?hide_missing=True&tests=security_OpenFDs@@@
  Will return from run_suite with status: ERROR

Also see the test suitcase: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/159380693-chromeos-test/chromeos6-row2-rack23-host13/security_OpenFDs/

Some logs from the test log:
2017-11-30T03:26:31.936409+00:00 NOTICE autotest[18585]: 19:26:31.933 ERROR|  security_OpenFDs:0144| Some filter(s) failed to match any fds: set(['0[57]00 /dev/shm/..*', '0700 /dev/dri/renderD129', '0700 anon_inode:dmabuf'])
2017-11-30T03:26:31.939598+00:00 NOTICE autotest[18586]: 19:26:31.937 ERROR|  security_OpenFDs:0147| Found unexpected fds in chrome type=renderer: set(['0700 /memfd: (deleted)'])
2017-11-30T03:26:31.944108+00:00 NOTICE autotest[18587]: 19:26:31.942 ERROR|  security_OpenFDs:0144| Some filter(s) failed to match any fds: set(['0[57]00 /dev/shm/..*', '0700 /dev/dri/renderD129', '0700 anon_inode:dmabuf'])
2017-11-30T03:26:31.946757+00:00 NOTICE autotest[18588]: 19:26:31.944 ERROR|  security_OpenFDs:0147| Found unexpected fds in chrome type=renderer: set(['0700 /memfd: (deleted)'])
 

Comment 1 by x...@chromium.org, Dec 4 2017

Cc: jbudorick@chromium.org
By looking at the history, it seems https://chromium-review.googlesource.com/c/chromium/src/+/797680 might be related. jbudorick@, could you help take a look to see if it might cause the failure? 
Autotest would have to be doing something very unexpected to be using the test runner code in //build/android/pylib...

Comment 3 by ihf@chromium.org, Dec 5 2017

This started on the Chrome pfq with R64-10171.0.0-b405
https://stainless.corp.google.com/search?exclude_cts=true&exclude_non_release=false&board=%5Ecaroline%24&test=%5Esecurity%5C_OpenFDs%24&view=matrix&col=build&row=board&first_date=20171129&last_date=20171205

Found unexpected fds in chrome type=renderer: set(['0700 /memfd: (deleted)'])

Time to bisect Chrome? (Then again this is a change detector test and 99% of the cases one can just add the newly detected issue to the whitelist.)
Project Member

Comment 4 by bugdroid1@chromium.org, Dec 5 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/5cff7ff71242e0ecd983e19b6063b3a72c1db8bd

commit 5cff7ff71242e0ecd983e19b6063b3a72c1db8bd
Author: xdai <xdai@chromium.org>
Date: Tue Dec 05 01:27:59 2017

Move the test security_OpenFDs from bvt-inline to bvt-perbuild.

The test security_OpenFDs constantly failed on caroline-tot-chrome-pfq-informational.
Move it to bvt-perbuild to unblock PFQ while I'm bisecting Chrome to figure out what
might cause it. It will be reverted later.

BUG= chromium:791786 
TEST=none

Change-Id: Id7e4ad60e98eb61a8aa96cabdd8ba76576f13a8c
Reviewed-on: https://chromium-review.googlesource.com/807288
Trybot-Ready: Xiaoqian Dai <xdai@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>

[modify] https://crrev.com/5cff7ff71242e0ecd983e19b6063b3a72c1db8bd/client/site_tests/security_OpenFDs/control

Comment 5 by x...@chromium.org, Dec 5 2017

Cc: -x...@chromium.org
Owner: x...@chromium.org
Status: Started (was: Untriaged)
Start bisecting

Comment 6 by x...@chromium.org, Dec 5 2017

Cc: x...@chromium.org
Owner: dvallet@chromium.org
Bisect is done. Culprit CL is https://chromium-review.googlesource.com/781160. Assign to the owner.
dvallet@, could you take a look at it? Please check if it's expected.
Cc: mnissler@chromium.org jorgelo@chromium.org
+security folks

This security_OpenFDs test appears to be failing on the R64 release branch on systems with ARC++ at least so far. 

Do we know if this failure is benign or if this is something we would want to hold up the 64 dev release for?
Cc: bhthompson@chromium.org kbleicher@chromium.org
Currently we will operate under the assumption that this is not a critical failure that would block release based on the response to this bug being that we remove the test from the PFQ (e.g. this was considered an invalid blocking of the PFQ and thus not a blocker for Chrome).

If this is not a valid assessment please let us know.
The culprit CL does not seem to be ARC++ related.

But more importantly Xiaoqian removed the test from PFQ *temporarily* while bisecting -- I don't think that means an invalid blocking of the PFQ.

The CL looks benign, we should probably just update the test.
Actually it is not just ARC++ systems as we see it in BayTrail, it may be systems on newer kernels? 

Removing the test temporarily would allow a potentially bad Chrome in though, so I would think that implies some belief the failures is benign.

In any case if you think it is safe, then we can move forward with the 64 dev today so I think we are good.
Newer kernels makes sense -- memfd does not exist on older kernels.

We should update the test.

Comment 12 by x...@chromium.org, Dec 6 2017

I was not sure if this test is important or not. But since the failure only happened on caroline, and the culprit CL was landed in 11/29 which means there might be subsequent CLs that depending on the culprit CL. So I would prefer to remove the test temporarily other than reverting the CL without confirming with the owner. 
dvallet@, could you provide your opinion here? 
It's fine to use memfd's for shared memory here. We should update the test expectation.
Status: Fixed (was: Started)
My apologies for all the extra work in bisecting the issue!

The CL has already been reverted and it was deemed insecure , see https://bugs.chromium.org/p/chromium/issues/detail?id=792117

The CL affected Chrome on Linux based systems (including Chrome OS), so if your tests covered any of these I'd say they were effective.

I'll mark this as fixed, feel free to reopen if there's anything else to do.
Cc: dvallet@chromium.org
Owner: x...@chromium.org
Status: Started (was: Fixed)
Ah good point, I forgot that we couldn't really seal the memfd's.

This is not fully done until the revert makes it to the branch and the test gets put back in the PFQ.

Xiaoqian, is the test back in the PFQ?

Daniel, did the revert land in M64 or M65?

Comment 16 by x...@chromium.org, Dec 6 2017

I'll bring the test back to PFQ. Danial, you might need to merge your revert CL to M64 if your revert CL landed in M65.

Comment 17 by x...@chromium.org, Dec 6 2017

Btw: I don't have permission to view  Issue 792117 
792117 is the security bug tied to memfd's, I added you to it but I don't think it adds a lot of context here.

Comment 19 by x...@chromium.org, Dec 7 2017

FYI CL to re-enable the test security_OpenFDs:
https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/812085
Project Member

Comment 20 by bugdroid1@chromium.org, Dec 7 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/aa048119bc0cac2cde121e5f9574689dfc18e1b0

commit aa048119bc0cac2cde121e5f9574689dfc18e1b0
Author: Xiaoqian Dai <xdai@chromium.org>
Date: Thu Dec 07 18:39:26 2017

Revert "Move the test security_OpenFDs from bvt-inline to bvt-perbuild."

This reverts commit 5cff7ff71242e0ecd983e19b6063b3a72c1db8bd.

Reason for revert: The CL that caused the failure has been reverted. Move this test back to bvt-inlie.

Original change's description:
> Move the test security_OpenFDs from bvt-inline to bvt-perbuild.
> 
> The test security_OpenFDs constantly failed on caroline-tot-chrome-pfq-informational.
> Move it to bvt-perbuild to unblock PFQ while I'm bisecting Chrome to figure out what
> might cause it. It will be reverted later.
> 
> BUG= chromium:791786 
> TEST=none
> 
> Change-Id: Id7e4ad60e98eb61a8aa96cabdd8ba76576f13a8c
> Reviewed-on: https://chromium-review.googlesource.com/807288
> Trybot-Ready: Xiaoqian Dai <xdai@chromium.org>
> Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
> Tested-by: Ilja H. Friedel <ihf@chromium.org>

Bug:  chromium:791786 
Change-Id: I3fdf7eb2838bab616eb43b9a788db8ec7ed92b4d
Reviewed-on: https://chromium-review.googlesource.com/812085
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>

[modify] https://crrev.com/aa048119bc0cac2cde121e5f9574689dfc18e1b0/client/site_tests/security_OpenFDs/control

Owner: dvallet@chromium.org
Thanks for the revert!

Daniel, please confirm the state of your CL's revert on the M64 branch. Thanks!
Owner: x...@chromium.org
Status: Fixed (was: Started)
https://bugs.chromium.org/p/chromium/issues/detail?id=792117#c16 has the revert landed in the M64 branch, so the test should not fail anymore on M64.

Sign in to add a comment