New issue
Advanced search Search tips

Issue 918134 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Jan 3
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Tast "cros" test bundle is killed due to security.StatefulPartitionHardening holding open files in /home

Project Member Reported by derat@chromium.org, Dec 28

Issue description

In the betty-pre-cq run at http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8925927000507178512, both TastVMTest attempts failed due to the test bundle receiving SIGTERM:

...
2018/12/27 22:59:12 Completed test security.SymlinkRestrictions in 1.145s with 0 error(s)
2018/12/27 22:59:12 Started test security.UserFilesGuest
2018/12/27 22:59:12 [22:59:10.424] Restarting ui job
2018/12/27 22:59:13 Got global error mid-test: runner.go:135: Bundle /usr/local/libexec/tast/bundles/local/cros failed: signal: terminated (status 5)
2018/12/27 22:59:13 Failed to run tests: runner.go:135: Bundle /usr/local/libexec/tast/bundles/local/cros failed: signal: terminated (status 5)
...

Looking at the system logs at http://pantheon/storage/browser/chromeos-image-archive/betty-pre-cq/R73-11481.0.0-b3284311/tast_vm_test_results_1/tast_vm_paladin/system_logs/, I don't see any obvious clues in the messages file. The last entry is from 2018-12-28T06:59:11.605947+00:00, and there doesn't appear to be anything there about why the process would be getting killed.

There *is* a shutdown_force_kill_processes file containing the following, though:

COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
local_tes 2480 root  txt    REG    8,1  5595240 284232 /usr/local/bin/local_test_runner
cros      2511 root  txt    REG    8,1 12839672 277154 /usr/local/libexec/tast/bundles/local/cros
cros      2511 root    6u   REG    8,1        0  25937 /mnt/stateful_partition/tast.security.StatefulPartitionHardening.symlink.255227741 (deleted)
cros      2511 root    7u   REG    8,1        0  25939 /mnt/stateful_partition/tast.security.StatefulPartitionHardening.symlink.574375704 (deleted)
cros      2511 root    8u   REG    8,1        0  25938 /mnt/stateful_partition/tast.security.StatefulPartitionHardening.fifo.552048535 (deleted)
cros      2511 root   12u   REG    8,1        0 144365 /mnt/stateful_partition/tast.security.StatefulPartitionHardening.854684949/tast.security.StatefulPartitionHardening.symlink.641784507 (deleted)
cros      2511 root   13u   REG    8,1        0 144367 /mnt/stateful_partition/tast.security.StatefulPartitionHardening.854684949/tast.security.StatefulPartitionHardening.symlink.629720798 (deleted)
cros      2511 root   14u   REG    8,1        0 144366 /mnt/stateful_partition/tast.security.StatefulPartitionHardening.854684949/tast.security.StatefulPartitionHardening.fifo.091106981 (deleted)
cros      2511 root   24u   REG    8,1        0 144368 /home/tast.security.StatefulPartitionHardening.symlink.110117799 (deleted)
cros      2511 root   25u   REG    8,1        0 144370 /home/tast.security.StatefulPartitionHardening.symlink.192994522 (deleted)
cros      2511 root   34u   REG    8,1        0 144369 /home/tast.security.StatefulPartitionHardening.766532130/tast.security.StatefulPartitionHardening.symlink.488402489 (deleted)
cros      2511 root   35u   REG    8,1        0 144372 /home/tast.security.StatefulPartitionHardening.766532130/tast.security.StatefulPartitionHardening.symlink.001686084 (deleted)

I think that what this is saying is that the cros test bundle process is still holding some deleted files in /home open while the ui job is being restarted and getting killed as a result. The ui-post-stop script calls:

  kill_with_open_files_on_path_and_mountpoints \
    '/home/[^.]|/data' /home/chronos/u-*

Those files are opened by security.StatefulPartitionHardening, which ran earlier:

...
2018/12/27 23:03:31 Completed test security.StatefulFiles in 27ms with 0 error(s)
2018/12/27 23:03:31 Started test security.StatefulPartitionHardening
2018/12/27 23:03:31 [23:03:30.102] Checking that symlinks and FIFOs are blocked in /mnt/stateful_partition
2018/12/27 23:03:31 [23:03:30.105] Checking that symlinks and FIFOs are blocked in /var
2018/12/27 23:03:31 [23:03:30.107] Checking that symlinks and FIFOs are blocked in /mnt/stateful_partition/tast.security.StatefulPartitionHardening.633218356
2018/12/27 23:03:31 [23:03:30.111] Checking that symlinks and FIFOs are blocked in /var/tast.security.StatefulPartitionHardening.348650243
2018/12/27 23:03:31 [23:03:30.113] Checking that symlinks and FIFOs are allowed in /tmp
2018/12/27 23:03:31 [23:03:30.114] Checking that symlinks and FIFOs are allowed in /tmp/tast.security.StatefulPartitionHardening.452632198
2018/12/27 23:03:31 [23:03:30.115] Checking that symlinks are allowed in /home
2018/12/27 23:03:31 [23:03:30.115] Checking that symlinks are allowed in /var/cache/echo
2018/12/27 23:03:31 [23:03:30.116] Checking that symlinks are allowed in /var/cache/vpd
2018/12/27 23:03:31 [23:03:30.117] Checking that symlinks are allowed in /var/lib/timezone
2018/12/27 23:03:31 [23:03:30.118] Checking that symlinks are allowed in /var/log
2018/12/27 23:03:31 [23:03:30.119] Checking that symlinks are allowed in /home/tast.security.StatefulPartitionHardening.589683757
2018/12/27 23:03:31 [23:03:30.120] Checking that symlinks are allowed in /var/cache/echo/tast.security.StatefulPartitionHardening.148001192
2018/12/27 23:03:31 [23:03:30.121] Checking that symlinks are allowed in /var/cache/vpd/tast.security.StatefulPartitionHardening.963854055
2018/12/27 23:03:31 [23:03:30.121] Checking that symlinks are allowed in /var/lib/timezone/tast.security.StatefulPartitionHardening.755686682
2018/12/27 23:03:31 [23:03:30.123] Checking that symlinks are allowed in /var/log/tast.security.StatefulPartitionHardening.280249265
2018/12/27 23:03:32 Completed test security.StatefulPartitionHardening in 24ms with 0 error(s)
2018/12/27 23:03:32 Started test security.SymlinkRestrictions
...

It's not immediately obvious to me how it's leaking files.
 
Summary: Tast "cros" test bundle is killed due to security.StatefulPartitionHardening holding open files in /home (was: Tast "cros" test bundle is killed due to )
Whoops, I was looking in the wrong place. Uploaded https://crrev.com/c/1390950 to fix the leak.
Project Member

Comment 3 by bugdroid1@chromium.org, Jan 3

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/f01c923cc48baeea1d27997ec2dbe207c0dbc8dd

commit f01c923cc48baeea1d27997ec2dbe207c0dbc8dd
Author: Daniel Erat <derat@chromium.org>
Date: Thu Jan 03 02:14:05 2019

tast-tests: Fix leak in security.StatefulPartitionHardening.

Fix a file leak in security.StatefulPartitionHardening that
causes the test bundle to get killed by ui-post-stop due to
it still holding open files in /home.

BUG= chromium:918134 
TEST=used lsof to verify that files are no longer open

Change-Id: Ie0625889f400b2156bc8ee8603e0b1822d9a159d
Reviewed-on: https://chromium-review.googlesource.com/1390950
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Eric Caruso <ejcaruso@chromium.org>

[modify] https://crrev.com/f01c923cc48baeea1d27997ec2dbe207c0dbc8dd/src/chromiumos/tast/local/bundles/cros/security/stateful_partition_hardening.go

Status: Fixed (was: Started)

Sign in to add a comment