New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 656903 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Oct 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Blocking:
issue 656870



Sign in to add a comment

security_SandboxedServices failure "One or more processes failed sandboxing"

Project Member Reported by cychiang@chromium.org, Oct 18 2016

Issue description

The failures happens starting 8907 https://bugs.chromium.org/p/chromium/issues/detail?id=656870

Test log from

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/81298194-chromeos-test/chromeos4-row4-rack11-host19/debug/

10/17 22:11:53.912 WARNI|security_Sandboxed:0266| Stale baselines: set(['thermal.sh', '# Since udev creates device nodes and changes owners/perms', 'attestationd', '# launch new shells via login.  Would be nice if it integrated things.', '# firewalld will fork+exec iptables to handle requests', '# Frecon needs to run as root and in the original namespace because it might', '# TODO: We can fix this when minijail supports ambient caps.  http://b/32066154', '# root.  TODO: We should namespace it.', 'timberslide', 'wimax-manager', '# We need to run as root due to caps not preserving across execs.', '# takes care of dropping root/caps for those commands.', 'arc-networkd', 'X', 'cromo', 'esif_ufd', 'easy_unlock', 'arc-obb-mounter', 'lid_touchpad_he'])
10/17 22:11:53.921 WARNI|security_Sandboxed:0269| New services: set(['avahi-daemon', 'arc_camera_serv', 'nacl_helper_non', 'brcm_patchram_p'])
10/17 22:11:53.929 ERROR|security_Sandboxed:0280| New services are not allowed to run as root, but these are: ['brcm_patchram_p']
10/17 22:11:53.938 ERROR|security_Sandboxed:0284| Failed sandboxing: ['brcm_patchram_p']
10/17 22:11:53.951 DEBUG|        base_utils:0185| Running 'logger "autotest finished iteration /usr/local/autotest/results/default/security_SandboxedServices/sysinfo/iteration.1"'
10/17 22:11:53.966 WARNI|              test:0606| Autotest caught exception when running test:
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 600, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/common_lib/test.py", line 804, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/common_lib/test.py", line 461, in execute
    dargs)
  File "/usr/local/autotest/common_lib/test.py", line 347, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/common_lib/test.py", line 376, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/tests/security_SandboxedServices/security_SandboxedServices.py", line 285, in run_once
    raise error.TestFail("One or more processes failed sandboxing")
TestFail: One or more processes failed sandboxing


Maybe this is related to recent change to the test: https://bugs.chromium.org/p/chromium/issues/detail?id=652969
Still need to check.
 
Blocking: 656870
https://wmatrix.googleplex.com/matrix/unfiltered?tests=security_SandboxedServices&days_back=20&hide_missing=True shows the test was quite stable until

R56-8907.0.0-rc2 on tricky
R56-8909.0.0 on veyron_mickey	veyron_minnie	veyron_rialto	veyron_speedy

This makes https://chromium-review.googlesource.com/#/c/395730/ which introduced in 8908.0 quite suspicious.
Oh ok that explains. I think we just need to add brcm_patchram_p to whitelist in the test.
It seems that other than brcm_patchram_plus, there is another process running cat as root.

  273     1 cat                              root           root           root             root             -268435457 -268435456 -268432121 -268435460 -268435459 -268435458 /usr/bin/coreutils --coreutils-prog-shebang=cat /bin/cat

But I am not sure where does that come from.
That seems to be coming from old tcsd.conf.
On tricky, the services to be added are ['ping', 'check_ethernet.']
Also something to fix:
1. remove services that does not run on tricky and veyron
2. we should not put comments in csv file, or should let csv reader handle comment correctly.


10/17 19:51:45.571 WARNI|security_Sandboxed:0266| Stale baselines: set(['# Since udev creates device nodes and changes owners/perms', 'attestationd', '# launch new shells via login.  Would be nice if it integrated things.', 'frecon', '# Frecon needs to run as root and in the original namespace because it might', 'timberslide', '# We need to run as root due to caps not preserving across execs.', 'arc-obb-mounter', 'thermal.sh', '# firewalld will fork+exec iptables to handle requests', 'easy_unlock', 'wimax-manager', 'daisydog', 'sslh-fork', 'X', '# takes care of dropping root/caps for those commands.', '# TODO: We can fix this when minijail supports ambient caps.  http://b/32066154', '# root.  TODO: We should namespace it.', 'cromo', 'esif_ufd', 'arc-networkd', 'lid_touchpad_he'])
10/17 19:51:45.575 WARNI|security_Sandboxed:0269| New services: set(['avahi-daemon', 'ping', 'nacl_helper_non', 'check_ethernet.'])
10/17 19:51:45.579 ERROR|security_Sandboxed:0280| New services are not allowed to run as root, but these are: ['ping', 'check_ethernet.']
10/17 19:51:45.583 ERROR|security_Sandboxed:0284| Failed sandboxing: ['ping', 'check_ethernet.']
I think check_ethernet and ping should be added into "exclude" instead.
They are test utils that runs periodically.

https://cs.corp.google.com/chromeos_public/src/platform/crostestutils/recover_duts/recover_duts.py?dr
Cc: briannorris@chromium.org
Status: Started (was: Assigned)
Fix uploaded https://chromium-review.googlesource.com/400318.
This should fix the missing process (brcm_patchram_plus) on veyron, and the flake (check_ethernet, ping) caused by test utils.
Cc: semenzato@chromium.org
The fix may be good, but I think it might have caused some other new failures? At least, I'm seeing this same test fail on other builders now (canary and lakitu-paladin I think? I'll double check).
e.g.:

10/18 06:03:36.665 WARNI|security_Sandboxed:0269| New services: set(['avahi-daemon', 'ping', 'nacl_helper_non', 'check_ethernet.'])
10/18 06:03:36.671 ERROR|security_Sandboxed:0280| New services are not allowed to run as root, but these are: ['ping', 'check_ethernet.']
10/18 06:03:36.676 ERROR|security_Sandboxed:0284| Failed sandboxing: ['ping', 'check_ethernet.']

https://uberchromegw.corp.google.com/i/chromeos/builders/butterfly-release/builds/3064
Labels: -Restrict-View-Google
Status: Fixed (was: Started)
Project Member

Comment 13 by bugdroid1@chromium.org, Oct 18 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/875c65e18d01debb2540942f081f321b9566ea9b

commit 875c65e18d01debb2540942f081f321b9566ea9b
Author: Cheng-Yi Chiang <cychiang@chromium.org>
Date: Tue Oct 18 17:37:07 2016

security_SandboxedServices: add brcm_patchram_plus for veyron

Broadcom bluetooth firmware patch downloader brcm_patchram_plus is run
on some veyron_* boards.

BUG= chromium:656903 
TEST=run the test on veyron_minnie

Change-Id: I6c4cdbd082c49169e8ede1f0c919ebc73543fe67
Reviewed-on: https://chromium-review.googlesource.com/400318
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Brian Norris <briannorris@chromium.org>

[modify] https://crrev.com/875c65e18d01debb2540942f081f321b9566ea9b/client/site_tests/security_SandboxedServices/baseline

Project Member

Comment 14 by bugdroid1@chromium.org, Oct 18 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/835e229a590372d2504f122bf84c6035fe11dcc7

commit 835e229a590372d2504f122bf84c6035fe11dcc7
Author: Cheng-Yi Chiang <cychiang@chromium.org>
Date: Tue Oct 18 17:32:30 2016

security_SandboxedServices: ignore ping/check_ethernet from autotest

Add 'ping' and 'check_ethernet' to exclude list since they will be run
periodically as autotest utils.

BUG= chromium:656903 
TEST=run the test on veyron_minnie

Change-Id: I6f6a3bfc0d6b748d24c31de6940dc23c9ed200cb
Reviewed-on: https://chromium-review.googlesource.com/400438
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Luigi Semenzato <semenzato@chromium.org>

[modify] https://crrev.com/835e229a590372d2504f122bf84c6035fe11dcc7/client/site_tests/security_SandboxedServices/exclude

Project Member

Comment 15 by bugdroid1@chromium.org, Oct 25 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/257e4b9e4d7a281482f5ef0c9a7011adf50924f4

commit 257e4b9e4d7a281482f5ef0c9a7011adf50924f4
Author: Mike Frysinger <vapier@chromium.org>
Date: Mon Oct 24 20:49:15 2016

security_SandboxedServices: fix check_ethernet name & add flock/laptop_mode

The name of the script is "check_ethernet.hook", so add the missing
dot to the exclude name so it gets filtered out.

Some scripts run themselves through the `flock` tool, so exclude that
from our checks too.

The laptop_mode script is run based on events (like power changes), so
filter that out too.

BUG= chromium:656903 
TEST=precq passes

Change-Id: I9fcefd033f94a9128b492a5b33034da04d23246a
Reviewed-on: https://chromium-review.googlesource.com/402408
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Luigi Semenzato <semenzato@chromium.org>

[modify] https://crrev.com/257e4b9e4d7a281482f5ef0c9a7011adf50924f4/client/site_tests/security_SandboxedServices/exclude

Project Member

Comment 16 by bugdroid1@chromium.org, Oct 25 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/257e4b9e4d7a281482f5ef0c9a7011adf50924f4

commit 257e4b9e4d7a281482f5ef0c9a7011adf50924f4
Author: Mike Frysinger <vapier@chromium.org>
Date: Mon Oct 24 20:49:15 2016

security_SandboxedServices: fix check_ethernet name & add flock/laptop_mode

The name of the script is "check_ethernet.hook", so add the missing
dot to the exclude name so it gets filtered out.

Some scripts run themselves through the `flock` tool, so exclude that
from our checks too.

The laptop_mode script is run based on events (like power changes), so
filter that out too.

BUG= chromium:656903 
TEST=precq passes

Change-Id: I9fcefd033f94a9128b492a5b33034da04d23246a
Reviewed-on: https://chromium-review.googlesource.com/402408
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Luigi Semenzato <semenzato@chromium.org>

[modify] https://crrev.com/257e4b9e4d7a281482f5ef0c9a7011adf50924f4/client/site_tests/security_SandboxedServices/exclude

Comment 17 by dchan@google.com, Jan 21 2017

Labels: VerifyIn-57

Comment 18 by dchan@google.com, Mar 4 2017

Labels: VerifyIn-58

Comment 19 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 20 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61

Comment 22 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)

Sign in to add a comment