SSH failure while auto-updating on [1]. So far just one instance on the PFQ so it could be a flake.


1: https://luci-milo.appspot.com/buildbot/chromeos/peppy-chrome-pfq/4176



Logs:

2017/10/24 04:35:11.730 DEBUG|        retry_util:0203| <class 'chromite.lib.cros_build_lib.RunCommandError'>(return code: 1; command: scp -P 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=60' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' -i /tmp/ssh-tmpURF30w/testing_rsa -r /home/chromeos-test/images/peppy-chrome-pfq/R64-10063.0.0-rc1/stateful.tgz root@100.115.197.171:/mnt/stateful_partition/unencrypted/preserve/cros-update/tmp.E6s1VrOPuG/
Could not copy /home/chromeos-test/images/peppy-chrome-pfq/R64-10063.0.0-rc1/stateful.tgz to device.)
2017/10/24 04:35:11.731 DEBUG|        retry_util:0065| Retrying in 5.277977 (0.000000 + jitter 5.277977) seconds ...
2017/10/24 04:35:17.014 DEBUG|      auto_updater:0581| Checking whether file stateful_update_bin needs to be transferred to device...
2017/10/24 04:35:17.014 WARNI|      auto_updater:0408| Could not find chroot stateful_update script in /home/chromeos-test/chromiumos/chroot/usr/bin/stateful_update, falling back to the client copy.
2017/10/24 04:35:17.014 DEBUG|      auto_updater:0413| Use stateful_update script in devserver path: /home/chromeos-test/chromiumos/src/platform/dev/stateful_update
2017/10/24 04:35:17.014 INFO |      auto_updater:0585| Copying stateful_update_bin to device...
2017/10/24 04:35:17.015 DEBUG|    cros_build_lib:0592| RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmpURF30w/testing_rsa root@100.115.197.171 -- 'PATH=/usr/local/bin:/usr/local/sbin:$PATH rsync' --version
2017/10/24 04:37:20.114 DEBUG|        retry_util:0201| ending retries with error: <class 'chromite.lib.remote_access.SSHConnectionError'>(ssh: connect to host 100.115.197.171 port 22: Connection timed out
)
2017/10/24 04:37:20.115 DEBUG|    cros_build_lib:0592| RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmpURF30w/testing_rsa root@100.115.197.171 -- rm -rf /mnt/stateful_partition/unencrypted/preserve/cros-update/tmp.E6s1VrOPuG
2017/10/24 04:39:23.242 ERROR|     remote_access:0906| Error connecting to device 100.115.197.171
2017/10/24 04:39:23.243 DEBUG|       cros_update:0269| Error happens in CrOS auto-update: SSHConnectionError('ssh: connect to host 100.115.197.171 port 22: Connection timed out\r\n',)