VMTests: failure in dev_mode_test leaves broken state behind |
||
Issue descriptionSee for example: https://luci-milo.appspot.com/buildbot/chromeos/betty-arc64-release/145 The first VMTest attempt fails because the device runs out of space in dev_mode_test. However the failure in the second attempt is different: " 08:11:16: INFO: Running VM test dev_mode_test. 08:11:16: INFO: RunCommand: /b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py --verbose betty-arc64 /b/c/cbuild/repository/src/build/images/betty-arc64/latest-cbuildbot/chromiumos_test_image.bin 2017/10/02 08:11:17 - devinstall_test.py - INFO : Setting up the image /b/c/cbuild/repository/src/build/images/betty-arc64/R63-9995.0.0/chromiumos_test_image.bin for vm testing. 2017/10/02 08:11:17 - devinstall_test.py - INFO : Making copy of the vm image /b/c/cbuild/repository/src/build/images/betty-arc64/R63-9995.0.0/chromiumos_qemu_image.bin to manipulate. 2017/10/02 08:11:27 - devinstall_test.py - DEBUG : Copy of vm image stored at /tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/chromiumos_qemu_image.bin. 2017/10/02 08:11:27 - devinstall_test.py - INFO : Wiping /usr/local/bin from the image. 2017/10/02 08:11:35 - devinstall_test.py - WARNING : Received error during cleanup Traceback (most recent call last): File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 73, in Cleanup self.vm.Stop() AttributeError: 'NoneType' object has no attribute 'Stop' Traceback (most recent call last): File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 188, in <module> main() File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 179, in main test.PrepareTest() File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 106, in PrepareTest self._WipeDevInstall() File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 87, in _WipeDevInstall safe=True) File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/../../crostestutils/lib/mount_helper.py", line 23, in MountImage cmd, print_cmd=False, cwd=constants.CROSUTILS_DIR, capture_output=True) File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 653, in RunCommand raise RunCommandError(msg, cmd_result) chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: ./mount_gpt_image.sh '--from=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest' '--image=chromiumos_qemu_image.bin' '--rootfs_mountpt=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/m' '--stateful_mountpt=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/s' --safe mount: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so ERROR : mount failed: device=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/chromiumos_qemu_image.bin target=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/s format= ro/rw=rw options=offset=4456448000 ERROR : Mon Oct 2 08:11:35 PDT 2017 ERROR : PGID PPID PID ELAPSED TIME %CPU COMMAND ERROR : Arguments of 9410: ./mount_gpt_image.sh '--from=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest' '--image=chromiumos_qemu_image.bin' '--rootfs_mountpt=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/m' '--stateful_mountpt=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/s' '--safe' ERROR : Backtrace: (most recent call is last) ERROR : mount_gpt_image.sh:427:main(), called: mount_image ERROR : mount_gpt_image.sh:346:mount_image(), called: die 'Failed to mount all partitions in /tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/chromiumos_qemu_image.bin' ERROR : ERROR : Error was: ERROR : Failed to mount all partitions in /tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/chromiumos_qemu_image.bin Current loopback device status: /dev/loop0: [0804]:56887108 (/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/chromiumos_qemu_image.bin), offset 2308964352 cmd=['./mount_gpt_image.sh', '--from=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest', '--image=chromiumos_qemu_image.bin', '--rootfs_mountpt=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/m', '--stateful_mountpt=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/s', '--safe'], cwd=/b/c/cbuild/repository/src/scripts 08:11:35: ERROR: !!!VMTests failed!!! " It looks as if the failure in the first attempt has left the system in a bad state that breaks (all?) further attempts.
,
Oct 31 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/crostestutils/+/eea4fed5c97f487b57204d33c12dba2b1a5cf42f commit eea4fed5c97f487b57204d33c12dba2b1a5cf42f Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Oct 31 22:15:27 2017 devmode-test: Fix cleanup failure when VM setup fails VM setup can fail while preparing the test, leaving the vm attribute None. Deal with it. BUG= chromium:777461 BUG= chromium:770764 TEST=None Change-Id: Id189343cf4f21cacb702507c0a6b9c249e0eaf30 Reviewed-on: https://chromium-review.googlesource.com/741684 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Nicolas Norvez <norvez@chromium.org> [modify] https://crrev.com/eea4fed5c97f487b57204d33c12dba2b1a5cf42f/devmode-test/devinstall_test.py
,
Mar 31 2018
Bulk closing old unconfirmed issues. |
||
►
Sign in to add a comment |
||
Comment 1 by norvez@chromium.org
, Oct 2 2017