VMTest failures "TestError: Test for cros flash failed." due to rsync failures |
||||
Issue descriptionI am seeing broken pipe errors for rsync as the main cause of many vmtest failures, for instance for samus-release: https://uberchromegw.corp.google.com/i/chromeos/builders/samus-release The error looks mostly like this: 06:34:11: ERROR: Device update failed. 06:34:11: ERROR: cros flash failed before completing. 06:34:11: ERROR: return code: 12; command: rsync --perms --verbose --times --compress --omit-dir-times --exclude .svn --links --rsync-path 'PATH=/usr/local/bin:/usr/local/sbin:$PATH rsync' --recursive --rsh 'ssh -p 58455 -oConnectionAttempts=4 -oUserKnownHostsFile=/dev/null -oProtocol=2 -oConnectTimeout=30 -oServerAliveCountMax=3 -oStrictHostKeyChecking=no -oServerAliveInterval=10 -oNumberOfPasswordPrompts=0 -i /tmp/cbuildbot-tmpxQRNS6/ssh-tmpX19_q2/testing_rsa' /tmp/cbuildbot-tmpxQRNS6/cros-flashxTe4S2/update.gz '[root@localhost]:/mnt/stateful_partition/cros-flash/tmp.OhIuZdeyqL/static/pregenerated' Warning: Permanently added '[localhost]:58455' (RSA) to the list of known hosts. Warning: Permanently added '[localhost]:58455' (RSA) to the list of known hosts. rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: connection unexpectedly closed (28 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] at least these three builds show the error: https://uberchromegw.corp.google.com/i/chromeos/builders/samus-release/builds/3212 https://uberchromegw.corp.google.com/i/chromeos/builders/samus-release/builds/3209 https://uberchromegw.corp.google.com/i/chromeos/builders/samus-release/builds/3199 Seems like network flakiness to me, but there are a lot of similar failures. I'll keep digging and update this bug as I find more examples.
,
May 20 2016
A good anchor text for this bug is "TestError: Test for cros flash failed." as that is the ultimate failure message.
,
May 20 2016
Also affecting jecht, see for instance https://uberchromegw.corp.google.com/i/chromeos/builders/jecht-release/builds/41
,
May 20 2016
#1 so the rsync through the virtual network is failing? Interesting. Out of space maybe?
,
May 20 2016
+martinroth@ you were looking at this at some point yesterday. Did you find anything? semenzato@ yes that's possible. From my quick reading of the logs at https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/samus-release/R52-8350.0.0/vm_test_results_2 I can't find relevant messages. I suspect the test that failed is not logging sufficiently. Maybe we need to instrument this test code better.
,
May 23 2016
Yes, I dug in some more, and these are also "no space left on device" errors. For instance, look at client.0.DEBUG in this directory: https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/samus-release/R53-8362.0.0/vm_test_results_2/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-platform_FilePerms/debug/ |
||||
►
Sign in to add a comment |
||||
Comment 1 by akes...@chromium.org
, May 20 2016