New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 770765 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

betty-arc64-release fails dev_mode_test in VMTests

Project Member Reported by norvez@chromium.org, Oct 2 2017

Issue description


https://luci-milo.appspot.com/buildbot/chromeos/betty-arc64-release/145

"
2017/10/02 07:13:54 - cros_build_lib.py - DEBUG   : RunCommand: sudo 'CROS_CACHEDIR=/b/c/cbuild/repository/.cache' 'CROS_SUDO_KEEP_ALIVE=unknown' -- rm -rf -- /b/c/cbuild/repository/chroot/tmp/devserver_wrapper_u4vB0
2017/10/02 07:13:54 - cros_build_lib.py - DEBUG   : RunCommand: /b/c/cbuild/repository/src/scripts/bin/cros_stop_vm --kvm_pid /tmp/cbuildbot-tmpJYrHjH/tmpWfmZ9bDevModeTest/kvm.pid
07:14:05: INFO: Refreshing due to a 401 (attempt 1/2)
07:14:05: INFO: Refreshing access_token
07:14:06: INFO: Running VM test cros_vm_test.
07:14:06: INFO: Setting up the VM for testing.
07:14:06: INFO: RunCommand: parted -m /b/c/cbuild/repository/src/build/images/betty-arc64/latest-cbuildbot/chromiumos_qemu_image.bin unit MB print
07:14:44: INFO: VM started at port 37923
07:14:44: INFO: The VM has been successfully set up. Ready to run tests.
07:14:44: INFO: Running test for cros shell.
07:14:44: INFO: Test to use shell command to write a file to the VM device.
07:14:44: INFO: RunCommand: cros shell --no-known-hosts ssh://localhost:37923 -- 'echo "shell command test file" > /tmp/shell-test'
07:14:45: INFO: Test to use shell command to read a file on the VM device.
07:14:45: INFO: RunCommand: cros shell --no-known-hosts ssh://localhost:37923 -- 'cat /tmp/shell-test'
07:14:45: INFO: Test to use shell command to remove a file on the VM device.
07:14:45: INFO: RunCommand: cros shell --no-known-hosts ssh://localhost:37923 -- 'rm /tmp/shell-test'
07:14:46: INFO: Test for cros shell passed.
07:14:46: INFO: Running test for cros debug.
07:14:46: INFO: Test to start and debug a new process on the VM device.
07:14:46: INFO: RunCommand: cros debug --exe /bin/bash ssh://localhost:37923
07:14:54: INFO: Test to attach a running process on the VM device.
07:14:54: INFO: RunCommand: cros debug --pid 1890 ssh://localhost:37923
07:15:02: INFO: Test for cros debug passed.
07:15:02: INFO: Running test for cros flash.
07:15:02: INFO: Test to flash the VM device with the latest image.
07:15:02: INFO: RunCommand: cros flash --no-wipe --no-reboot ssh://localhost:37923 latest
07:17:06: INFO: Refreshing due to a 401 (attempt 1/2)
07:17:06: INFO: Refreshing access_token
07:24:36: ERROR: Failed to flash the VM device.
07:24:36: INFO: 
----------- Start of cros flash log -----------
07:15:03: NOTICE: Preparing to update the remote device localhost
07:15:04: NOTICE: Using image betty-arc64/R63-9995.0.0/chromiumos_test_image.bin
07:23:05: NOTICE: Finalizing image.
07:24:09: NOTICE: Update completed.
07:24:35: ERROR: Stateful update failed.
07:24:36: ERROR: Device update failed.
07:24:36: ERROR: cros flash failed before completing.
07:24:36: ERROR: Failed to perform stateful partition update: return code: 1; command: ssh -p 37923 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/cbuildbot-tmpJYrHjH/ssh-tmpiS6l7N/testing_rsa root@localhost -- sh /mnt/stateful_partition/cros-flash/tmp.BgmpQyJQRC/stateful_update /mnt/stateful_partition/cros-flash/tmp.BgmpQyJQRC/stateful.tgz
Warning: Permanently added '[localhost]:37923' (ED25519) to the list of known hosts.
Warning: Permanently added '[localhost]:37923' (ED25519) to the list of known hosts.
Reading local payload /mnt/stateful_partition/cros-flash/tmp.BgmpQyJQRC/stateful.tgz
tar: dev_image_new/lib64/python2.7/sre_constants.pyc: Wrote only 4096 of 6195 bytes
tar: dev_image_new/lib64/python2.7/rlcompleter.pyo: Cannot write: No space left on device
tar: dev_image_new/lib64/python2.7/copy_reg.py: Cannot write: No space left on device
tar: dev_image_new/lib64/python2.7/ftplib.pyc: Cannot open: No space left on device
tar: dev_image_new/lib64/python2.7/sre.pyo: Cannot open: No space left on device
tar: dev_image_new/lib64/python2.7/webbrowser.pyc: Cannot open: No space left on device
tar: dev_image_new/lib64/python2.7/rfc822.pyc: Cannot open: No space left on device
<snip>
"

Note that the second attempt is a different failure mode (see  Issue 770764 )
 
Cc: teravest@chromium.org bmgordon@chromium.org
I saw this on betty-arc64-release today. Is there anything stopping us from increasing the filesystem size on betty? It's only used for VM testing, right?
bmgordon@ appears to be hitting this in the CQ with a change he's making as well.

 crbug.com/770764  definitely makes this harder to diagnose.

Comment 2 by norvez@chromium.org, Oct 30 2017

I think you're right, it should be OK to just increase the filesystem size for betty-arc64. I'll prepare a CL.

Regarding  crbug.com/770764 , https://chromium-review.googlesource.com/c/chromiumos/platform/crostestutils/+/741684 should fix it.
I've been looking into this.  It looks like the autotests succeed, but then attempting to re-flash the VM fails.  I've reproduced it on my local VM.  cros flash downloads a large update file into the stateful partition, then tries to unpack it somewhere and runs out of space.
seems to be passing https://uberchromegw.corp.google.com/i/chromeos/builders/betty-arc64-paladin?numbuilds=100

Was something done to fix this, or did it fix itself?
It fixed itself.  I'm working on a CL to increase the stateful partition size so that this doesn't come back.
Project Member

Comment 6 by bugdroid1@chromium.org, Nov 18 2017

Comment 7 by norvez@chromium.org, Nov 20 2017

Status: Fixed (was: Assigned)
betty-arc64-release now passes after increasing the size of the stateful partition.

Comment 8 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Comment 9 by dchan@chromium.org, Jan 23 2018

Status: Fixed (was: Archived)

Sign in to add a comment