provision failure due to failure in initial rsync to DUT |
||||||||
Issue descriptionBuild https://luci-milo.appspot.com/buildbot/chromeos/kip-paladin/836 Test http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=103381347 Provision job: http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row2-rack13-host17/521306-provision/ Relevant snippet: 2017/02/25 05:19:02.773 DEBUG| auto_updater:0250| Restore stateful partition is not required. 2017/02/25 05:19:02.773 DEBUG| cros_update:0215| Start CrOS update process... 2017/02/25 05:19:02.774 DEBUG| cros_build_lib:0564| RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmp53_rtF/testing_rsa root@chromeos4-row2-rack13-host17 -- mkdir -p /mnt/stateful_partition/unencrypted/preserve/cros-update/tmp.s6zbFr1ciV/static/pregenerated Warning: Permanently added 'chromeos4-row2-rack13-host17,100.115.193.135' (RSA) to the list of known hosts. 2017/02/25 05:19:02.997 INFO | auto_updater:0501| Copying rootfs payload to device... 2017/02/25 05:19:02.998 DEBUG| cros_build_lib:0564| RunCommand: rsync --perms --verbose --times --compress --omit-dir-times --exclude .svn --links --rsync-path 'PATH=/usr/local/bin:/usr/local/sbin:$PATH rsync' --recursive --rsh 'ssh -p 22 -oConnectionAttempts=4 -oUserKnownHostsFile=/dev/null -oProtocol=2 -oConnectTimeout=30 -oServerAliveCountMax=3 -oStrictHostKeyChecking=no -oServerAliveInterval=10 -oNumberOfPasswordPrompts=0 -oIdentitiesOnly=yes -i /tmp/ssh-tmp53_rtF/testing_rsa' /home/chromeos-test/images/kip-paladin/R58-9316.0.0-rc1/update.gz '[root@chromeos4-row2-rack13-host17]:/mnt/stateful_partition/unencrypted/preserve/cros-update/tmp.s6zbFr1ciV/static/pregenerated' Warning: Permanently added 'chromeos4-row2-rack13-host17,100.115.193.135' (RSA) to the list of known hosts. sending incremental file list update.gz Timeout, server chromeos4-row2-rack13-host17 not responding. rsync: [sender] write error: Broken pipe (32) rsync error: unexplained error (code 255) at io.c(837) [sender=3.1.0] 2017/02/25 05:20:26.419 DEBUG| retry_util:0136| <class 'chromite.lib.cros_build_lib.RunCommandError'>(return code: 255; command: rsync --perms --verbose --times --compress --omit-dir-times --exclude .svn --links --rsync-path 'PATH=/usr/local/bin:/usr/local/sbin:$PATH rsync' --recursive --rsh 'ssh -p 22 -oConnectionAttempts=4 -oUserKnownHostsFile=/dev/null -oProtocol=2 -oConnectTimeout=30 -oServerAliveCountMax=3 -oStrictHostKeyChecking=no -oServerAliveInterval=10 -oNumberOfPasswordPrompts=0 -oIdentitiesOnly=yes -i /tmp/ssh-tmp53_rtF/testing_rsa' /home/chromeos-test/images/kip-paladin/R58-9316.0.0-rc1/update.gz '[root@chromeos4-row2-rack13-host17]:/mnt/stateful_partition/unencrypted/preserve/cros-update/tmp.s6zbFr1ciV/static/pregenerated' Could not copy /home/chromeos-test/images/kip-paladin/R58-9316.0.0-rc1/update.gz to device.) 2017/02/25 05:20:26.419 DEBUG| retry_util:0073| Retrying in 4.347227 seconds... 2017/02/25 05:20:30.770 DEBUG| cros_build_lib:0564| RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmp53_rtF/testing_rsa root@chromeos4-row2-rack13-host17 -- mkdir -p /mnt/stateful_partition/unencrypted/preserve/cros-update/tmp.s6zbFr1ciV/static/pregenerated ssh: connect to host chromeos4-row2-rack13-host17 port 22: Connection timed out 2017/02/25 05:22:33.946 ERROR| remote_access:0856| Error connecting to device chromeos4-row2-rack13-host17 2017/02/25 05:22:33.946 DEBUG| retry_util:0129| ending retries with error: <class 'chromite.lib.remote_access.SSHConnectionError'>(None) 2017/02/25 05:22:33.947 DEBUG| cros_build_lib:0564| RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmp53_rtF/testing_rsa root@chromeos4-row2-rack13-host17 -- rm -rf /mnt/stateful_partition/unencrypted/preserve/cros-update/tmp.s6zbFr1ciV 2017/02/25 05:24:37.130 ERROR| remote_access:0856| Error connecting to device chromeos4-row2-rack13-host17 2017/02/25 05:24:37.131 DEBUG| cros_update:0253| Error happens in CrOS auto-update: SSHConnectionError('ssh: connect to host chromeos4-row2-rack13-host17 port 22: Connection timed out\r\n',)
,
Feb 28 2017
,
Feb 28 2017
2017-02-28 07:41:26 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row1-rack3-host7/2999886-repair/
2017-02-28 07:28:29 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row1-rack3-host7/2999735-provision/
,
Feb 28 2017
,
Mar 7 2017
,
Mar 14 2017
,
Jun 20 2017
Believe it's due to devserver overload/high bandwidth usage. @Don will work on this by a new design doc, so mark this as wontfix.
,
Jul 17 2017
ChromeOS Infra P1 Bugscrub. P1 Bugs in this component should be important enough to get weekly status updates. Is this already fixed? -> Fixed Is this no longer relevant? -> Archived or WontFix Is this not a P1, based on go/chromeos-infra-bug-slo rubric? -> lower priority. Is this a Feature Request rather than a bug? Type -> Feature Is this missing important information or scope needed to decide how to proceed? -> Ask question on bug, possibly reassign. Does this bug have the wrong owner? -> reassign. Bugs that remain in this state next week will be downgraded to P2.
,
Jul 24 2017
ChromeOS Infra P1 Bugscrub. Issue untouched in a week after previous message. Downgrading to P2.
,
Mar 19 2018
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by chingcodes@chromium.org
, Feb 27 2017