link-paladin: provision: FAIL: Unhandled AutoservSSHTimeout |
||
Issue descriptionhttps://luci-milo.appspot.com/buildbot/chromeos/link-paladin/30328 https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/157397275-chromeos-test/chromeos4-row5-rack13-host5/provision_AutoUpdate/debug/ 11/20 09:10:13.926 INFO | ssh_multiplex:0107| Timed out waiting for master-ssh connection to be established. 11/20 09:11:17.235 ERROR| utils:0280| [stderr] ssh: connect to host chromeos4-row5-rack13-host5 port 22: Connection timed out 11/20 09:11:17.242 DEBUG| test:0410| Test failed due to ('ssh timed out', * Command: /usr/bin/ssh -a -x -o ControlPath=/tmp/_autotmp_h1e7gjssh- master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 chromeos4-row5-rack13-host5 "export LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack: :get_chromeos_release_milestone|_get_lsb_release_content|run] -> ssh_run(cat \\\"/etc/lsb-release\\\")\";fi; cat \"/etc/lsb-release\"" Exit status: 255 Duration: 63.1770660877 stderr: ssh: connect to host chromeos4-row5-rack13-host5 port 22: Connection timed out). Exception log follows the after_iteration_hooks. 11/20 09:11:17.243 DEBUG| test:0415| Starting after_iteration_hooks for provision_AutoUpdate 11/20 09:11:17.243 DEBUG| test:0420| after_iteration_hooks completed 11/20 09:11:17.247 WARNI| test:0637| The test failed with the following exception Traceback (most recent call last): File "/usr/local/autotest/client/common_lib/test.py", line 631, in _exec _call_test_function(self.execute, *p_args, **p_dargs) File "/usr/local/autotest/client/common_lib/test.py", line 837, in _call_test_function raise error.UnhandledTestFail(e) UnhandledTestFail: Unhandled AutoservSSHTimeout: ('ssh timed out', * Command: /usr/bin/ssh -a -x -o ControlPath=/tmp/_autotmp_h1e7gjssh- master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 chromeos4-row5-rack13-host5 "export LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack: :get_chromeos_release_milestone|_get_lsb_release_content|run] -> ssh_run(cat \\\"/etc/lsb-release\\\")\";fi; cat \"/etc/lsb-release\"" Exit status: 255 Duration: 63.1770660877 stderr: ssh: connect to host chromeos4-row5-rack13-host5 port 22: Connection timed out) Traceback (most recent call last): File "/usr/local/autotest/client/common_lib/test.py", line 831, in _call_test_function return func(*args, **dargs) File "/usr/local/autotest/client/common_lib/test.py", line 495, in execute dargs) File "/usr/local/autotest/client/common_lib/test.py", line 362, in _call_run_once_with_retry postprocess_profiled_run, args, dargs) File "/usr/local/autotest/client/common_lib/test.py", line 400, in _call_run_once self.run_once(*args, **dargs) File "/usr/local/autotest/server/site_tests/provision_AutoUpdate/provision_AutoUpdate.py", line 121, in run_once with_cheets=with_cheets) File "/usr/local/autotest/server/afe_utils.py", line 124, in machine_install_and_update_labels *args, **dargs) File "/usr/local/autotest/server/hosts/cros_host.py", line 805, in machine_install_by_devserver force_original = self.get_chromeos_release_milestone() is None File "/usr/local/autotest/server/hosts/cros_host.py", line 1398, in get_chromeos_release_milestone lsb_release_content=self._get_lsb_release_content()) File "/usr/local/autotest/server/hosts/cros_host.py", line 1377, in _get_lsb_release_content 'cat "%s"' % client_constants.LSB_RELEASE).stdout.strip() File "/usr/local/autotest/server/hosts/ssh_host.py", line 318, in run return self.run_very_slowly(*args, **kwargs) File "/usr/local/autotest/server/hosts/ssh_host.py", line 307, in run_very_slowly ssh_failure_retry_ok) File "/usr/local/autotest/server/hosts/ssh_host.py", line 249, in _run raise error.AutoservSSHTimeout("ssh timed out", result) AutoservSSHTimeout: ('ssh timed out', * Command: /usr/bin/ssh -a -x -o ControlPath=/tmp/_autotmp_h1e7gjssh- master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 chromeos4-row5-rack13-host5 "export LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack: :get_chromeos_release_milestone|_get_lsb_release_content|run] -> ssh_run(cat \\\"/etc/lsb-release\\\")\";fi; cat \"/etc/lsb-release\"" Exit status: 255 Duration: 63.1770660877 stderr: ssh: connect to host chromeos4-row5-rack13-host5 port 22: Connection timed out)
,
Nov 22 2017
Filed the repair bug at b/69681430
,
Nov 22 2017
swapped chromeos4-row5-rack13-host5 with chromeos4-row10-rack4-host19 |
||
►
Sign in to add a comment |
||
Comment 1 by nxia@chromium.org
, Nov 22 2017chromeos4-row5-rack13-host5 has been failing at provision, locking the dut and filing a repair bug. nxia@nxia:~/chromiumos/src/third_party/autotest/files/site-packages$ dut-status -f chromeos4-row5-rack13-host5 chromeos4-row5-rack13-host5 2017-11-22 12:59:39 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4713120-provision/ 2017-11-22 12:24:57 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712873-verify/ 2017-11-22 11:30:27 NO http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712587-repair/ 2017-11-22 11:24:20 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712579-verify/ 2017-11-22 10:28:01 NO http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712368-repair/ 2017-11-22 10:18:17 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712352-provision/ 2017-11-22 08:54:20 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712293-repair/ 2017-11-22 08:53:12 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712288-verify/ 2017-11-22 08:11:01 NO http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712087-repair/ 2017-11-22 08:01:15 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712052-provision/ 2017-11-22 06:57:31 OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4711984-repair/ 2017-11-22 06:52:10 -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4711974-verify/