New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 852029 link

Starred by 2 users
Status: Verified
Owner:
norvez@chromium.org
Closed: Jun 2018
Cc:
xzhou@chromium.org
█ cra...@chromium.org
hywu@chromium.org
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

betty-release canary failing in VM test "dev_mode_test"

Project Member Reported by aaboagye@chromium.org, Jun 12 2018 
The betty-release builder has been failing for quite some time. The latest error that I see is that when it starts running "VM test dev_mode_test", it fails with the following traceback.



15:58:21: INFO: Running VM test dev_mode_test.
15:58:21: INFO: RunCommand: /b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py --verbose betty /b/c/cbuild/repository/src/build/images/betty/latest-cbuildbot/chromiumos_test_image.bin
2018/06/11 15:58:22 - devinstall_test.py - INFO    : Setting up the image /b/c/cbuild/repository/src/build/images/betty/R69-10773.0.0/chromiumos_test_image.bin for vm testing.
2018/06/11 15:58:22 - devinstall_test.py - INFO    : Making copy of the vm image /b/c/cbuild/repository/src/build/images/betty/R69-10773.0.0/chromiumos_qemu_image.bin to manipulate.
2018/06/11 15:58:53 - devinstall_test.py - DEBUG   : Copy of vm image stored at /tmp/cbuildbot-tmpvo9Myy/tmpywS6eODevModeTest/chromiumos_qemu_image.bin.
2018/06/11 15:58:53 - devinstall_test.py - INFO    : Wiping /usr/local/bin from the image.
2018/06/11 15:58:57 - devinstall_test.py - ERROR   : Received error during cleanup
Traceback (most recent call last):
  File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 84, in Cleanup
    osutils.RmDir(self.tmpdir, ignore_missing=True)
  File "/b/c/cbuild/repository/chromite/lib/osutils.py", line 365, in RmDir
    shutil.rmtree(path)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 256, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python2.7/shutil.py", line 254, in rmtree
    os.rmdir(path)
OSError: [Errno 30] Read-only file system: '/tmp/cbuildbot-tmpvo9Myy/tmpywS6eODevModeTest/m/proc'
Traceback (most recent call last):
  File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 192, in <module>
    main()
  File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 189, in main
    options.binhost).Run()
  File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 168, in Run
    self.PrepareTest()
  File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 115, in PrepareTest
    self._WipeDevInstall()
  File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/devinstall_test.py", line 96, in _WipeDevInstall
    safe=True)
  File "/b/c/cbuild/repository/src/platform/crostestutils/devmode-test/../../crostestutils/lib/mount_helper.py", line 23, in MountImage
    cmd, print_cmd=False, cwd=constants.CROSUTILS_DIR, capture_output=True)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 665, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: ./mount_gpt_image.sh '--from=/tmp/cbuildbot-tmpvo9Myy/tmpywS6eODevModeTest' '--image=chromiumos_qemu_image.bin' '--rootfs_mountpt=/tmp/cbuildbot-tmpvo9Myy/tmpywS6eODevModeTest/m' '--stateful_mountpt=/tmp/cbuildbot-tmpvo9Myy/tmpywS6eODevModeTest/s' --safe
mount: wrong fs type, bad option, bad superblock on /dev/loop0p1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
ERROR   : mount failed: image=/tmp/cbuildbot-tmpvo9Myy/tmpywS6eODevModeTest/chromiumos_qemu_image.bin device=/dev/loop0p1 target=/tmp/cbuildbot-tmpvo9Myy/tmpywS6eODevModeTest/s format= ro/rw=rw

Logs here: http://shortn/_OShTggF9mJ

Assigning to norvez@ as he's listed as the owner of this "board".

I did some searching and found that similar cases had appeared in the past: issue 779627, issue 782054.

Comment 1 by aaboagye@chromium.org, Jun 12 2018 

*Sorry that first bug number was actually  issue 779267 .

Comment 2 by norvez@chromium.org, Jun 12 2018 

I think that error is a red herring, it's coming from issue 782054. The first attempt for VMTests failed in a way that left the image in a bad state, and the 2nd attempt fails for this unrelated reason.

Looking at the 1st VMTests, the error is:

"
15:11:15: INFO: Test to flash the VM device with the latest image.
15:11:15: INFO: RunCommand: cros flash --no-wipe --no-reboot ssh://localhost:39749 latest
15:19:13: ERROR: Failed to flash the VM device.
15:19:13: INFO: 
----------- Start of cros flash log -----------
15:11:17: NOTICE: Preparing to update the remote device localhost
15:11:18: NOTICE: Using image betty/R69-10773.0.0/chromiumos_test_image.bin
15:17:41: NOTICE: Finalizing image.
15:18:40: NOTICE: Update completed.
15:19:10: ERROR: Stateful update failed.
15:19:12: ERROR: Device update failed.
15:19:13: ERROR: cros flash failed before completing.
15:19:13: ERROR: Failed to perform stateful partition update: return code: 1; command: ssh -p 39749 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/cbuildbot-tmpvo9Myy/ssh-tmpMiskNN/testing_rsa root@localhost -- sh /mnt/stateful_partition/cros-flash/tmp.DUPiJZg17a/stateful_update /mnt/stateful_partition/cros-flash/tmp.DUPiJZg17a/stateful.tgz
Warning: Permanently added '[localhost]:39749' (ED25519) to the list of known hosts.
Warning: Permanently added '[localhost]:39749' (ED25519) to the list of known hosts.
Reading local payload /mnt/stateful_partition/cros-flash/tmp.DUPiJZg17a/stateful.tgz
tar: dev_image_new/libexec/tast/bundles/local/cros: Wrote only 9728 of 10240 bytes
tar: dev_image_new/include: Cannot mkdir: No space left on device
tar: dev_image_new/include: Cannot mkdir: No space left on device
"


That looks similar to  issue 770765  and  issue 793958

Comment 3 by norvez@chromium.org, Jun 12 2018

Status: Started (was: Untriaged)
CL https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-betty-private/+/639651 is up but not tested yet. I've kicked off a tryjob, I'll send the CL for review if the tryjob passes.

Comment 4 by norvez@chromium.org, Jun 12 2018 

Tryjob passed, CL is ready for review.
Project Member

Comment 5 by bugdroid1@chromium.org, Jun 13 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-betty-private/+/224f77abbaa33e312af824cca4dc220356d8fe38

commit 224f77abbaa33e312af824cca4dc220356d8fe38
Author: Nicolas Norvez <norvez@google.com>
Date: Wed Jun 13 04:50:46 2018

Comment 6 by aaboagye@chromium.org, Jun 13 2018

Status: Verified (was: Started)
Excellent, the most recent run of betty-release is green!
►

Sign in to add a comment