New issue
Advanced search Search tips

Issue 770764 link

Starred by 1 user

Issue metadata

Status: Archived
Owner: ----
Closed: Mar 2018
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

VMTests: failure in dev_mode_test leaves broken state behind

Project Member Reported by norvez@chromium.org, Oct 2 2017

Issue description

See for example:

https://luci-milo.appspot.com/buildbot/chromeos/betty-arc64-release/145

The first VMTest attempt fails because the device runs out of space in dev_mode_test.
However the failure in the second attempt is different:
"
08:11:16: INFO: Running VM test dev_mode_test.
08:11:16: INFO: RunCommand: /b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py --verbose betty-arc64 /b/c/cbuild/repository/src/build/images/betty-arc64/latest-cbuildbot/chromiumos_test_image.bin
2017/10/02 08:11:17 - devinstall_test.py - INFO    : Setting up the image /b/c/cbuild/repository/src/build/images/betty-arc64/R63-9995.0.0/chromiumos_test_image.bin for vm testing.
2017/10/02 08:11:17 - devinstall_test.py - INFO    : Making copy of the vm image /b/c/cbuild/repository/src/build/images/betty-arc64/R63-9995.0.0/chromiumos_qemu_image.bin to manipulate.
2017/10/02 08:11:27 - devinstall_test.py - DEBUG   : Copy of vm image stored at /tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/chromiumos_qemu_image.bin.
2017/10/02 08:11:27 - devinstall_test.py - INFO    : Wiping /usr/local/bin from the image.
2017/10/02 08:11:35 - devinstall_test.py - WARNING : Received error during cleanup
Traceback (most recent call last):
  File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 73, in Cleanup
    self.vm.Stop()
AttributeError: 'NoneType' object has no attribute 'Stop'
Traceback (most recent call last):
  File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 188, in <module>
    main()
  File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 179, in main
    test.PrepareTest()
  File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 106, in PrepareTest
    self._WipeDevInstall()
  File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 87, in _WipeDevInstall
    safe=True)
  File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/../../crostestutils/lib/mount_helper.py", line 23, in MountImage
    cmd, print_cmd=False, cwd=constants.CROSUTILS_DIR, capture_output=True)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 653, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: ./mount_gpt_image.sh '--from=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest' '--image=chromiumos_qemu_image.bin' '--rootfs_mountpt=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/m' '--stateful_mountpt=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/s' --safe
mount: wrong fs type, bad option, bad superblock on /dev/loop1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

ERROR   : mount failed: device=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/chromiumos_qemu_image.bin target=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/s format= ro/rw=rw options=offset=4456448000
ERROR   : Mon Oct  2 08:11:35 PDT 2017
ERROR   :  PGID  PPID   PID     ELAPSED     TIME %CPU COMMAND
ERROR   : Arguments of 9410: ./mount_gpt_image.sh '--from=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest' '--image=chromiumos_qemu_image.bin' '--rootfs_mountpt=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/m' '--stateful_mountpt=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/s' '--safe'
ERROR   : Backtrace:  (most recent call is last)
ERROR   :  mount_gpt_image.sh:427:main(), called: mount_image  
ERROR   :  mount_gpt_image.sh:346:mount_image(), called: die 'Failed to mount all partitions in /tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/chromiumos_qemu_image.bin' 
ERROR   : 
ERROR   : Error was:
ERROR   :   Failed to mount all partitions in /tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/chromiumos_qemu_image.bin

Current loopback device status:
    /dev/loop0: [0804]:56887108 (/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/chromiumos_qemu_image.bin), offset 2308964352

cmd=['./mount_gpt_image.sh', '--from=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest', '--image=chromiumos_qemu_image.bin', '--rootfs_mountpt=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/m', '--stateful_mountpt=/tmp/cbuildbot-tmpJYrHjH/tmp5k0K6WDevModeTest/s', '--safe'], cwd=/b/c/cbuild/repository/src/scripts
08:11:35: ERROR: 
!!!VMTests failed!!!
"

It looks as if the failure in the first attempt has left the system in a bad state that breaks (all?) further attempts.

 
 Issue 770765  tracks the failure in the first VMTests attempt.
Project Member

Comment 2 by bugdroid1@chromium.org, Oct 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/crostestutils/+/eea4fed5c97f487b57204d33c12dba2b1a5cf42f

commit eea4fed5c97f487b57204d33c12dba2b1a5cf42f
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Tue Oct 31 22:15:27 2017

devmode-test: Fix cleanup failure when VM setup fails

VM setup can fail while preparing the test, leaving the vm attribute
None. Deal with it.

BUG= chromium:777461 
BUG= chromium:770764 
TEST=None

Change-Id: Id189343cf4f21cacb702507c0a6b9c249e0eaf30
Reviewed-on: https://chromium-review.googlesource.com/741684
Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Nicolas Norvez <norvez@chromium.org>

[modify] https://crrev.com/eea4fed5c97f487b57204d33c12dba2b1a5cf42f/devmode-test/devinstall_test.py

Status: Archived (was: Unconfirmed)
Bulk closing old unconfirmed issues.

Sign in to add a comment