security_OpenFDs recent test flakes |
||||||||||||||||||||
Issue descriptionWe have seen this issue on falco-chrome-pfq twice this month. https://bugs.chromium.org/p/chromium/issues/list?q=security_OpenFDs+Failure The most recent run failure had this in the logs: 04/27 20:09:21.441 ERROR| security_OpenFDs:0147| Found unexpected fds in chrome type=renderer: set(['0700 anon_inode:dmabuf']) 04/27 20:09:21.446 DEBUG| base_utils:0176| Running 'logger "autotest finished iteration /usr/local/autotest/results/default/security_OpenFDs/sysinfo/iteration.1"' 04/27 20:09:21.451 WARNI| test:0606| Autotest caught exception when running test: Traceback (most recent call last): [...] File "/usr/local/autotest/tests/security_OpenFDs/security_OpenFDs.py", line 211, in run_once raise error.TestFail("Unexpected open file descriptors.") [jschuh@, please feel free to reassign]
,
Apr 28 2016
,
May 26 2016
,
May 26 2016
,
Jun 1 2016
Is this the same issue as is blocking veyron_rialto HW Test, or should I split a new bug for this? Output below this line is for buildbot consumption: @@@STEP_LINK@[Test-Logs]: Suite job: ABORT@http://cautotest/tko/retrieve_logs.cgi?job=/results/65328898-chromeos-test/@@@ @@@STEP_LINK@[Flake-Dashboard]: Suite job@https://wmatrix.googleplex.com/retry_teststats/?days_back=30&tests=Suite job@@@ @@@STEP_LINK@[Test-Logs]: security_OpenFDs: FAIL: Unexpected open file descriptors.@http://cautotest/tko/retrieve_logs.cgi?job=/results/65328913-chromeos-test/@@@ @@@STEP_LINK@[Flake-Dashboard]: security_OpenFDs@https://wmatrix.googleplex.com/retry_teststats/?days_back=30&tests=security_OpenFDs@@@ Will return from run_suite with status: ERROR https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_rialto-release/builds/77/steps/HWTest%20%5Bbvt-inline%5D/logs/stdio
,
Jun 1 2016
(https://groups.google.com/a/google.com/d/msg/chromeos-team/dl40yDXh40g/GtLCD_RjNwAJ mentioned this issue is impacting peach-pit too)
,
Jun 4 2016
,
Jun 7 2016
,
Jun 13 2016
,
Jun 14 2016
,
Jun 17 2016
Justin is a busy man.
,
Jun 17 2016
I assume it is an FD leak which is caused by one of the test that run before.
,
Jun 17 2016
Looks like it does not really depend on previous tests test_that --iterations 20 $DUT security_OpenFDs -------------------------------------------------------------------------------------- /tmp/test_that_results_dCoKvb/results-01-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-01-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-02-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-02-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-03-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-03-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-04-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-04-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-05-security_OpenFDs [ FAILED ] /tmp/test_that_results_dCoKvb/results-05-security_OpenFDs FAIL: Unexpected open file descriptors. /tmp/test_that_results_dCoKvb/results-05-security_OpenFDs/security_OpenFDs [ FAILED ] /tmp/test_that_results_dCoKvb/results-05-security_OpenFDs/security_OpenFDs FAIL: Unexpected open file descriptors. /tmp/test_that_results_dCoKvb/results-05-security_OpenFDs/security_OpenFDs 06/17 16:25:11.353 ERROR| security_OpenFDs:0144| Some filter(s) failed to match any fds: set(['0700 /dev/dri/renderD129']) /tmp/test_that_results_dCoKvb/results-05-security_OpenFDs/security_OpenFDs 06/17 16:25:11.357 ERROR| security_OpenFDs:0147| Found unexpected fds in chrome type=renderer: set(['0700 anon_inode:dmabuf']) /tmp/test_that_results_dCoKvb/results-06-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-06-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-07-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-07-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-08-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-08-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-09-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-09-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-10-security_OpenFDs [ FAILED ] /tmp/test_that_results_dCoKvb/results-10-security_OpenFDs FAIL: Unexpected open file descriptors. /tmp/test_that_results_dCoKvb/results-10-security_OpenFDs/security_OpenFDs [ FAILED ] /tmp/test_that_results_dCoKvb/results-10-security_OpenFDs/security_OpenFDs FAIL: Unexpected open file descriptors. /tmp/test_that_results_dCoKvb/results-10-security_OpenFDs/security_OpenFDs 06/17 16:26:11.659 ERROR| security_OpenFDs:0144| Some filter(s) failed to match any fds: set(['0700 /dev/dri/renderD129']) /tmp/test_that_results_dCoKvb/results-10-security_OpenFDs/security_OpenFDs 06/17 16:26:11.662 ERROR| security_OpenFDs:0147| Found unexpected fds in chrome type=renderer: set(['0700 anon_inode:dmabuf']) /tmp/test_that_results_dCoKvb/results-11-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-11-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-12-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-12-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-13-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-13-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-14-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-14-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-15-security_OpenFDs [ FAILED ] /tmp/test_that_results_dCoKvb/results-15-security_OpenFDs FAIL: Unexpected open file descriptors. /tmp/test_that_results_dCoKvb/results-15-security_OpenFDs/security_OpenFDs [ FAILED ] /tmp/test_that_results_dCoKvb/results-15-security_OpenFDs/security_OpenFDs FAIL: Unexpected open file descriptors. /tmp/test_that_results_dCoKvb/results-15-security_OpenFDs/security_OpenFDs 06/17 16:27:11.991 ERROR| security_OpenFDs:0144| Some filter(s) failed to match any fds: set(['0700 /dev/dri/renderD129']) /tmp/test_that_results_dCoKvb/results-15-security_OpenFDs/security_OpenFDs 06/17 16:27:11.995 ERROR| security_OpenFDs:0147| Found unexpected fds in chrome type=renderer: set(['0700 anon_inode:dmabuf']) /tmp/test_that_results_dCoKvb/results-16-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-16-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-17-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-17-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-18-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-18-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-19-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-19-security_OpenFDs/security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-20-security_OpenFDs [ PASSED ] /tmp/test_that_results_dCoKvb/results-20-security_OpenFDs/security_OpenFDs [ PASSED ] -------------------------------------------------------------------------------------- Total PASS: 34/40 (85%)
,
Jun 18 2016
It is a racy test. I will kick it off the cq for now. Notice pretty reliably on Falco after starting chrome first it passes, then it fails twice, then it passes:
localhost autotest # stop ui; start ui; bin/autotest tests/security_OpenFDs/control; bin/autotest tests/security_OpenFDs/control; bin/autotest tests/security_OpenFDs/control; bin/autotest tests/security_OpenFDs/control;
ui stop/waiting
ui start/running, process 28561
DEBUG:root:Failed to import elasticsearch. Mock classes will be used and calls to Elasticsearch server will be no-op. Test run is not affected by the missing elasticsearch module.
17:11:18 INFO | Writing results to /usr/local/autotest/results/default
17:11:18 ERROR| Symlink init scripts failed with [Errno 30] Read-only file system
17:11:18 INFO | START ---- ---- timestamp=1466208678 localtime=Jun 17 17:11:18
17:11:18 INFO | START security_OpenFDs security_OpenFDs timestamp=1466208678 localtime=Jun 17 17:11:18
17:11:18 INFO | ChromeOS BOARD = falco_1.4GHz_4GB
17:11:18 ERROR| Some filter(s) failed to match any fds: set(['0500 /usr/share/fonts/.*', '0700 /dev/dri/renderD129'])
17:11:20 INFO | GOOD security_OpenFDs security_OpenFDs timestamp=1466208680 localtime=Jun 17 17:11:20 completed successfully
17:11:20 INFO | END GOOD security_OpenFDs security_OpenFDs timestamp=1466208680 localtime=Jun 17 17:11:20
17:11:20 INFO | END GOOD ---- ---- timestamp=1466208680 localtime=Jun 17 17:11:20
DEBUG:root:Failed to import elasticsearch. Mock classes will be used and calls to Elasticsearch server will be no-op. Test run is not affected by the missing elasticsearch module.
17:11:21 INFO | Writing results to /usr/local/autotest/results/default
17:11:21 ERROR| Symlink init scripts failed with [Errno 30] Read-only file system
17:11:21 INFO | START ---- ---- timestamp=1466208681 localtime=Jun 17 17:11:21
17:11:21 INFO | START security_OpenFDs security_OpenFDs timestamp=1466208681 localtime=Jun 17 17:11:21
17:11:21 INFO | ChromeOS BOARD = falco_1.4GHz_4GB
17:11:21 ERROR| Some filter(s) failed to match any fds: set(['0500 /usr/share/fonts/.*', '0700 /dev/dri/renderD129'])
17:11:21 ERROR| Some filter(s) failed to match any fds: set(['0700 /dev/dri/renderD129'])
17:11:21 ERROR| Found unexpected fds in chrome type=renderer: set(['0700 anon_inode:dmabuf'])
17:11:21 WARNI| Autotest caught exception when running test:
Traceback (most recent call last):
File "/usr/local/autotest/common_lib/test.py", line 600, in _exec
_call_test_function(self.execute, *p_args, **p_dargs)
File "/usr/local/autotest/common_lib/test.py", line 804, in _call_test_function
return func(*args, **dargs)
File "/usr/local/autotest/common_lib/test.py", line 461, in execute
dargs)
File "/usr/local/autotest/common_lib/test.py", line 347, in _call_run_once_with_retry
postprocess_profiled_run, args, dargs)
File "/usr/local/autotest/common_lib/test.py", line 376, in _call_run_once
self.run_once(*args, **dargs)
File "/usr/local/autotest/tests/security_OpenFDs/security_OpenFDs.py", line 211, in run_once
raise error.TestFail("Unexpected open file descriptors.")
TestFail: Unexpected open file descriptors.
17:11:22 ERROR| child process failed
17:11:23 INFO | FAIL security_OpenFDs security_OpenFDs timestamp=1466208682 localtime=Jun 17 17:11:22 Unexpected open file descriptors.
17:11:23 INFO | END FAIL security_OpenFDs security_OpenFDs timestamp=1466208683 localtime=Jun 17 17:11:23
17:11:23 INFO | END GOOD ---- ---- timestamp=1466208683 localtime=Jun 17 17:11:23
DEBUG:root:Failed to import elasticsearch. Mock classes will be used and calls to Elasticsearch server will be no-op. Test run is not affected by the missing elasticsearch module.
17:11:23 INFO | Writing results to /usr/local/autotest/results/default
17:11:23 ERROR| Symlink init scripts failed with [Errno 30] Read-only file system
17:11:23 INFO | START ---- ---- timestamp=1466208683 localtime=Jun 17 17:11:23
17:11:23 INFO | START security_OpenFDs security_OpenFDs timestamp=1466208683 localtime=Jun 17 17:11:23
17:11:23 INFO | ChromeOS BOARD = falco_1.4GHz_4GB
17:11:23 ERROR| Some filter(s) failed to match any fds: set(['0500 /usr/share/fonts/.*', '0700 /dev/dri/renderD129'])
17:11:23 ERROR| Some filter(s) failed to match any fds: set(['0700 /dev/dri/renderD129'])
17:11:23 ERROR| Found unexpected fds in chrome type=renderer: set(['0700 anon_inode:dmabuf'])
17:11:23 WARNI| Autotest caught exception when running test:
Traceback (most recent call last):
File "/usr/local/autotest/common_lib/test.py", line 600, in _exec
_call_test_function(self.execute, *p_args, **p_dargs)
File "/usr/local/autotest/common_lib/test.py", line 804, in _call_test_function
return func(*args, **dargs)
File "/usr/local/autotest/common_lib/test.py", line 461, in execute
dargs)
File "/usr/local/autotest/common_lib/test.py", line 347, in _call_run_once_with_retry
postprocess_profiled_run, args, dargs)
File "/usr/local/autotest/common_lib/test.py", line 376, in _call_run_once
self.run_once(*args, **dargs)
File "/usr/local/autotest/tests/security_OpenFDs/security_OpenFDs.py", line 211, in run_once
raise error.TestFail("Unexpected open file descriptors.")
TestFail: Unexpected open file descriptors.
17:11:24 ERROR| child process failed
17:11:25 INFO | FAIL security_OpenFDs security_OpenFDs timestamp=1466208684 localtime=Jun 17 17:11:24 Unexpected open file descriptors.
17:11:25 INFO | END FAIL security_OpenFDs security_OpenFDs timestamp=1466208685 localtime=Jun 17 17:11:25
17:11:25 INFO | END GOOD ---- ---- timestamp=1466208685 localtime=Jun 17 17:11:25
DEBUG:root:Failed to import elasticsearch. Mock classes will be used and calls to Elasticsearch server will be no-op. Test run is not affected by the missing elasticsearch module.
17:11:25 INFO | Writing results to /usr/local/autotest/results/default
17:11:25 ERROR| Symlink init scripts failed with [Errno 30] Read-only file system
17:11:25 INFO | START ---- ---- timestamp=1466208685 localtime=Jun 17 17:11:25
17:11:25 INFO | START security_OpenFDs security_OpenFDs timestamp=1466208685 localtime=Jun 17 17:11:25
17:11:26 INFO | ChromeOS BOARD = falco_1.4GHz_4GB
17:11:26 ERROR| Some filter(s) failed to match any fds: set(['0500 /usr/share/fonts/.*', '0700 /dev/dri/renderD129'])
17:11:26 ERROR| Some filter(s) failed to match any fds: set(['0700 /dev/dri/renderD129'])
17:11:27 INFO | GOOD security_OpenFDs security_OpenFDs timestamp=1466208686 localtime=Jun 17 17:11:26 completed successfully
17:11:27 INFO | END GOOD security_OpenFDs security_OpenFDs timestamp=1466208687 localtime=Jun 17 17:11:27
17:11:27 INFO | END GOOD ---- ---- timestamp=1466208687 localtime=Jun 17 17:11:27
,
Jun 18 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/37aee5eaff86749891ebcde3454a4e9617c7a214 commit 37aee5eaff86749891ebcde3454a4e9617c7a214 Author: Ilja H. Friedel <ihf@chromium.org> Date: Sat Jun 18 00:40:33 2016 security_OpenFDs: whitelist anon_inode:dmabuf. This FD shows up for 1 or 2 seconds after starting Chrome and disappears again. Still need to figure out where it came from. BUG= chromium:607632 TEST=looped security_OpenFDs on Falco. Change-Id: Ib8080999add113cd222eb99efec7166af676aa94 Reviewed-on: https://chromium-review.googlesource.com/353812 Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/37aee5eaff86749891ebcde3454a4e9617c7a214/client/site_tests/security_OpenFDs/security_OpenFDs.py
,
Jun 18 2016
,
Jun 20 2016
The source of these dmabuf FDs is the recent addition of zero-copy texture uploads. In other words, this is WAI and the test just needed to be updated, so this bug is fixed.
,
Jun 20 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/fe14f5801ce9e0a27e227b5e20de73a0f34d7a6f commit fe14f5801ce9e0a27e227b5e20de73a0f34d7a6f Author: Ilja H. Friedel <ihf@chromium.org> Date: Mon Jun 20 22:35:39 2016 security_OpenFDs: clarify comment. BUG= chromium:607632 TEST=None. Change-Id: I3e6326fb29a90da1f26991502de4e874a5472228 Reviewed-on: https://chromium-review.googlesource.com/354230 Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Stéphane Marchesin <marcheu@chromium.org> [modify] https://crrev.com/fe14f5801ce9e0a27e227b5e20de73a0f34d7a6f/client/site_tests/security_OpenFDs/security_OpenFDs.py
,
Jun 20 2016
,
Jul 1 2016
,
Aug 29 2016
,
Oct 7 2016
,
Nov 19 2016
,
Jan 21 2017
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Aug 3 2017
Closing. Please reopen it if its not fixed. Thanks! |
||||||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||||||
Comment 1 by lpique@chromium.org
, Apr 28 2016