Devserver may fail staging when using SSH |
||||
Issue descriptionSSH can be aborted by devserver, which causes failure in CQ like: Running 'ssh 172.17.40.17 'curl "http://172.17.40.17:8082/is_staged?artifacts=full_payload,stateful,autotest_packages&files=&archive_url=gs://chromeos-image-archive/x86-alex-paladin/R51-8104.0.0-rc2"'' 03/23 22:52:10.740 DEBUG| base_utils:0177| Running 'ssh 172.17.40.17 'curl "http://172.17.40.17:8082/is_staged?artifacts=full_payload,stateful,autotest_packages&files=&archive_url=gs://chromeos-image-archive/x86-alex-paladin/R51-8104.0.0-rc2"'' 03/23 22:52:17.009 DEBUG| base_utils:0177| Running 'ssh 172.17.40.17 'curl "http://172.17.40.17:8082/is_staged?artifacts=full_payload,stateful,autotest_packages&files=&archive_url=gs://chromeos-image-archive/x86-alex-paladin/R51-8104.0.0-rc2"'' 03/23 22:52:17.176 DEBUG| dev_server:0728| Error occurred with exit_code 255 when executing thessh call: ssh_exchange_identification: Connection closed by remote host Current mechanism don't retry ssh connection if we enter an ssh 'CmdError'. That may be an issue.
,
Mar 29 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/476408397bec660680d7d4f347ead1eb7dc21aa4 commit 476408397bec660680d7d4f347ead1eb7dc21aa4 Author: xixuan <xixuan@google.com> Date: Thu Mar 24 17:33:16 2016 Autotest: Make Devserver retry ssh connection, and Add stats for ssh failures. When trying to SSH to a devserver, this SSH connection could be aborted by the devserver itself, which causes staging failure. This CL add 'retry' to all ssh connections to devserver, and add stats for ssh failures: ssh_dev_server_failure, and ssh_tunnel_failure, to estimate the stabality of ssh-based approaches. BUG= chromium:597680 TEST=Ran dev_server_unittest to ensure error.CmdError causes retry. Turn on the enable_ssh switch to test that ssh to devserver & servo still works. Change-Id: I4b9e6092d8989fb2ecb770367c5869042e8e5740 Reviewed-on: https://chromium-review.googlesource.com/334770 Reviewed-by: Xixuan Wu <xixuan@chromium.org> Tested-by: Xixuan Wu <xixuan@chromium.org> Commit-Queue: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/476408397bec660680d7d4f347ead1eb7dc21aa4/server/hosts/rpc_server_tracker.py [modify] https://crrev.com/476408397bec660680d7d4f347ead1eb7dc21aa4/client/common_lib/cros/dev_server.py [modify] https://crrev.com/476408397bec660680d7d4f347ead1eb7dc21aa4/client/common_lib/cros/dev_server_unittest.py
,
Apr 26 2016
,
Jun 9 2016
,
Jun 27 2016
Closing... please feel free to reopen if its not fixed.
,
Jun 27 2016
|
||||
►
Sign in to add a comment |
||||
Comment 1 by fdeng@chromium.org
, Mar 24 2016