security_SandboxedServices is very flaky: 'adb' failed sandboxing |
||||||||
Issue descriptionsecurity_SandboxedServices failed in 3 of 4 runs of veyron_minnie-tot-chrome-pfq-informational builds. It started showing up from https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/veyron_minnie-tot-chrome-pfq-informational/builds/6787. Selected error messages: 12/05 22:18:24.992 DEBUG| utils:0212| Running 'for f in /proc/[1-9]*/status ; do awk '$1 ~ "^(Pid|CapInh|CapPrm|CapEff|CapBnd|CapAmb|NoNewPrivs|Seccomp):" {printf "%s%s ", $1, $NF; if ($1 == "Seccomp:") printf "\n"}' $f ; done' 12/05 22:18:26.439 DEBUG|security_Sandboxed:0141| output of awk: 12/05 22:18:26.461 DEBUG| utils:0212| Running 'scanelf -qF'%s#F' -gs __asan_init `which debugd`' 12/05 22:18:26.579 DEBUG| asan:0026| running_on_asan(): symbol: '', _ASAN_SYMBOL: '__asan_init' 12/05 22:18:26.595 WARNI|security_Sandboxed:0320| Stale baselines: set(['thermal.sh', 'attestationd', 'easy_unlock', 'arc-networkd', 'netfilter-queue', 'timberslide', 'wimax-manager', 'esif_ufd', 'cromo', 'tpm_managerd', 'upstart-udev-br', 'arc-obb-mounter', 'lid_touchpad_he']) 12/05 22:18:26.603 WARNI|security_Sandboxed:0323| New services: set(['adb', 'main', 'arc-oemcrypto', 'boot_latch']) 12/05 22:18:26.610 ERROR|security_Sandboxed:0334| New services are not allowed to run as root, but these are: ['adb'] 12/05 22:18:26.618 ERROR|security_Sandboxed:0338| Failed sandboxing: ['adb'] 12/05 22:18:26.706 WARNI| test:0637| The test failed with the following exception Traceback (most recent call last): File "/usr/local/autotest/common_lib/test.py", line 631, in _exec _call_test_function(self.execute, *p_args, **p_dargs) File "/usr/local/autotest/common_lib/test.py", line 831, in _call_test_function return func(*args, **dargs) File "/usr/local/autotest/common_lib/test.py", line 495, in execute dargs) File "/usr/local/autotest/common_lib/test.py", line 362, in _call_run_once_with_retry postprocess_profiled_run, args, dargs) File "/usr/local/autotest/common_lib/test.py", line 400, in _call_run_once self.run_once(*args, **dargs) File "/usr/local/autotest/tests/security_SandboxedServices/security_SandboxedServices.py", line 339, in run_once raise error.TestFail('One or more processes failed sandboxing') TestFail: One or more processes failed sandboxing
,
Jan 5 2018
Still failing in HWTest [bvt-inline] at least once a day on veyron_minnie-tot-chrome-pfq-informational: https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/veyron_minnie-tot-chrome-pfq-informational?numbuilds=100
,
Jan 7 2018
jorgelo@ - you seem to be a frequent contributor to this test file. Could you take a quick look at this and triage it appropriately?
,
Jan 11 2018
Happened again on https://luci-milo.appspot.com/buildbot/chromeos.chrome/veyron_minnie-tot-chrome-pfq-informational/7053 +linben@, hidehiko@ since this seems like a possible adb issue.
,
Jan 12 2018
Drive-by. According to the /var/log/messages, https://pantheon.corp.google.com/storage/browser/chromeos-autotest-esults/169419563-chromeos-test/chromeos4-row9-rack10-host9/security_SandboxedServices/sysinfo/var/log/ it looks like other ARC tests ran before the security test. It started adb server, and the security_SandboxedService test seemed to find it is running and reported an error. Some random ideas to fix; - Should we add an entry for "adb" to baseline or exclusion? Looks like a easiest fix, assuming it is ok from security perspective. - Maybe should we kill adb server in ARC test teardown? Though, TBH, it looks difficult to "100% ensure" it is always killed correctly. - Reboot the machine before testing? WDYT?
,
Jan 12 2018
Tests must not assume that they get a clean/freshly installed/freshly rebooted DUT for testing, except if they set these conditions up for themselves. There is nothing in infra land that would promise that.
,
Jan 16 2018
,
Jan 16 2018
Still see this on Jan 15: https://sheriff-o-matic.appspot.com/gardener#
,
Jan 16 2018
Here is the correct link: https://luci-milo.appspot.com/buildbot/chromeos.chrome/veyron_minnie-tot-chrome-pfq-informational/7073#
,
Jan 17 2018
adb should be added to the exclusion list, probably. The only problem with adding to the excl list is that if we ever start running adb constantly, we won't find that.
,
Jan 17 2018
Re #6, security_SandboxedServices presumably has a built-in expectation that it's running on a Chrome OS device that's in some "normal" state, right? I'm a bit uneasy about carving out holes for processes that other tests might've left lying around.
,
Jan 17 2018
It is a change detector test which captures the state of a DUT at a single but random time and compares it with a baseline. Such tests have a use in that they automate work that otherwise would have to be performed manually by the test team. That said they are incredibly difficult to get into a shape that is fit for the CQ (=fast + stable). I do believe that adding adb should be fine, as this is a CrOS test and adb lives in the container. The container is meant to be tested by CTS.
,
Jan 23 2018
Flaky tests don't help anyone, we should add adb to the exclude list at least for the immediate future: https://chromium-review.googlesource.com/#/c/chromiumos/third_party/autotest/+/881666
,
Jan 23 2018
if CTS/whatever is leaking adb processes, those should be fixed to not leave the device in a bad state
,
Feb 5 2018
The CL landed, I'm gonna call this fixed. Things look green: https://stainless.corp.google.com/search?test=%5Esecurity%5C_SandboxedServices%24&exclude_non_release=true&exclude_cts=true&col=build&row=model&view=matrix&first_date=2018-01-30&last_date=2018-02-05
,
Mar 17 2018
,
Mar 17 2018
Issue 799669 has been merged into this issue. |
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by michae...@chromium.org
, Dec 14 2017Labels: Hotlist-CrOS-Sheriffing
Summary: security_SandboxedServices is very flaky: 'adb' failed sandboxing (was: security_SandboxedServices is very flaky on veyron_minnie-tot-chrome-pfq-informational)