Tast "cros" test bundle is killed due to security.StatefulPartitionHardening holding open files in /home |
||
Issue descriptionIn the betty-pre-cq run at http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8925927000507178512, both TastVMTest attempts failed due to the test bundle receiving SIGTERM: ... 2018/12/27 22:59:12 Completed test security.SymlinkRestrictions in 1.145s with 0 error(s) 2018/12/27 22:59:12 Started test security.UserFilesGuest 2018/12/27 22:59:12 [22:59:10.424] Restarting ui job 2018/12/27 22:59:13 Got global error mid-test: runner.go:135: Bundle /usr/local/libexec/tast/bundles/local/cros failed: signal: terminated (status 5) 2018/12/27 22:59:13 Failed to run tests: runner.go:135: Bundle /usr/local/libexec/tast/bundles/local/cros failed: signal: terminated (status 5) ... Looking at the system logs at http://pantheon/storage/browser/chromeos-image-archive/betty-pre-cq/R73-11481.0.0-b3284311/tast_vm_test_results_1/tast_vm_paladin/system_logs/, I don't see any obvious clues in the messages file. The last entry is from 2018-12-28T06:59:11.605947+00:00, and there doesn't appear to be anything there about why the process would be getting killed. There *is* a shutdown_force_kill_processes file containing the following, though: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME local_tes 2480 root txt REG 8,1 5595240 284232 /usr/local/bin/local_test_runner cros 2511 root txt REG 8,1 12839672 277154 /usr/local/libexec/tast/bundles/local/cros cros 2511 root 6u REG 8,1 0 25937 /mnt/stateful_partition/tast.security.StatefulPartitionHardening.symlink.255227741 (deleted) cros 2511 root 7u REG 8,1 0 25939 /mnt/stateful_partition/tast.security.StatefulPartitionHardening.symlink.574375704 (deleted) cros 2511 root 8u REG 8,1 0 25938 /mnt/stateful_partition/tast.security.StatefulPartitionHardening.fifo.552048535 (deleted) cros 2511 root 12u REG 8,1 0 144365 /mnt/stateful_partition/tast.security.StatefulPartitionHardening.854684949/tast.security.StatefulPartitionHardening.symlink.641784507 (deleted) cros 2511 root 13u REG 8,1 0 144367 /mnt/stateful_partition/tast.security.StatefulPartitionHardening.854684949/tast.security.StatefulPartitionHardening.symlink.629720798 (deleted) cros 2511 root 14u REG 8,1 0 144366 /mnt/stateful_partition/tast.security.StatefulPartitionHardening.854684949/tast.security.StatefulPartitionHardening.fifo.091106981 (deleted) cros 2511 root 24u REG 8,1 0 144368 /home/tast.security.StatefulPartitionHardening.symlink.110117799 (deleted) cros 2511 root 25u REG 8,1 0 144370 /home/tast.security.StatefulPartitionHardening.symlink.192994522 (deleted) cros 2511 root 34u REG 8,1 0 144369 /home/tast.security.StatefulPartitionHardening.766532130/tast.security.StatefulPartitionHardening.symlink.488402489 (deleted) cros 2511 root 35u REG 8,1 0 144372 /home/tast.security.StatefulPartitionHardening.766532130/tast.security.StatefulPartitionHardening.symlink.001686084 (deleted) I think that what this is saying is that the cros test bundle process is still holding some deleted files in /home open while the ui job is being restarted and getting killed as a result. The ui-post-stop script calls: kill_with_open_files_on_path_and_mountpoints \ '/home/[^.]|/data' /home/chronos/u-* Those files are opened by security.StatefulPartitionHardening, which ran earlier: ... 2018/12/27 23:03:31 Completed test security.StatefulFiles in 27ms with 0 error(s) 2018/12/27 23:03:31 Started test security.StatefulPartitionHardening 2018/12/27 23:03:31 [23:03:30.102] Checking that symlinks and FIFOs are blocked in /mnt/stateful_partition 2018/12/27 23:03:31 [23:03:30.105] Checking that symlinks and FIFOs are blocked in /var 2018/12/27 23:03:31 [23:03:30.107] Checking that symlinks and FIFOs are blocked in /mnt/stateful_partition/tast.security.StatefulPartitionHardening.633218356 2018/12/27 23:03:31 [23:03:30.111] Checking that symlinks and FIFOs are blocked in /var/tast.security.StatefulPartitionHardening.348650243 2018/12/27 23:03:31 [23:03:30.113] Checking that symlinks and FIFOs are allowed in /tmp 2018/12/27 23:03:31 [23:03:30.114] Checking that symlinks and FIFOs are allowed in /tmp/tast.security.StatefulPartitionHardening.452632198 2018/12/27 23:03:31 [23:03:30.115] Checking that symlinks are allowed in /home 2018/12/27 23:03:31 [23:03:30.115] Checking that symlinks are allowed in /var/cache/echo 2018/12/27 23:03:31 [23:03:30.116] Checking that symlinks are allowed in /var/cache/vpd 2018/12/27 23:03:31 [23:03:30.117] Checking that symlinks are allowed in /var/lib/timezone 2018/12/27 23:03:31 [23:03:30.118] Checking that symlinks are allowed in /var/log 2018/12/27 23:03:31 [23:03:30.119] Checking that symlinks are allowed in /home/tast.security.StatefulPartitionHardening.589683757 2018/12/27 23:03:31 [23:03:30.120] Checking that symlinks are allowed in /var/cache/echo/tast.security.StatefulPartitionHardening.148001192 2018/12/27 23:03:31 [23:03:30.121] Checking that symlinks are allowed in /var/cache/vpd/tast.security.StatefulPartitionHardening.963854055 2018/12/27 23:03:31 [23:03:30.121] Checking that symlinks are allowed in /var/lib/timezone/tast.security.StatefulPartitionHardening.755686682 2018/12/27 23:03:31 [23:03:30.123] Checking that symlinks are allowed in /var/log/tast.security.StatefulPartitionHardening.280249265 2018/12/27 23:03:32 Completed test security.StatefulPartitionHardening in 24ms with 0 error(s) 2018/12/27 23:03:32 Started test security.SymlinkRestrictions ... It's not immediately obvious to me how it's leaking files.
,
Dec 28
Whoops, I was looking in the wrong place. Uploaded https://crrev.com/c/1390950 to fix the leak.
,
Jan 3
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/f01c923cc48baeea1d27997ec2dbe207c0dbc8dd commit f01c923cc48baeea1d27997ec2dbe207c0dbc8dd Author: Daniel Erat <derat@chromium.org> Date: Thu Jan 03 02:14:05 2019 tast-tests: Fix leak in security.StatefulPartitionHardening. Fix a file leak in security.StatefulPartitionHardening that causes the test bundle to get killed by ui-post-stop due to it still holding open files in /home. BUG= chromium:918134 TEST=used lsof to verify that files are no longer open Change-Id: Ie0625889f400b2156bc8ee8603e0b1822d9a159d Reviewed-on: https://chromium-review.googlesource.com/1390950 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Eric Caruso <ejcaruso@chromium.org> [modify] https://crrev.com/f01c923cc48baeea1d27997ec2dbe207c0dbc8dd/src/chromiumos/tast/local/bundles/cros/security/stateful_partition_hardening.go
,
Jan 3
|
||
►
Sign in to add a comment |
||
Comment 1 by derat@chromium.org
, Dec 28