New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 787062 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Nov 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

link-paladin: provision: FAIL: Unhandled AutoservSSHTimeout

Project Member Reported by nxia@chromium.org, Nov 20 2017

Issue description


https://luci-milo.appspot.com/buildbot/chromeos/link-paladin/30328


https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/157397275-chromeos-test/chromeos4-row5-rack13-host5/provision_AutoUpdate/debug/


11/20 09:10:13.926 INFO |     ssh_multiplex:0107| Timed out waiting for master-ssh connection to be established.
11/20 09:11:17.235 ERROR|             utils:0280| [stderr] ssh: connect to host chromeos4-row5-rack13-host5 port 22: Connection timed out
11/20 09:11:17.242 DEBUG|              test:0410| Test failed due to ('ssh timed out', * Command: 
    /usr/bin/ssh -a -x    -o ControlPath=/tmp/_autotmp_h1e7gjssh-
    master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
    -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
    ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
    chromeos4-row5-rack13-host5 "export LIBC_FATAL_STDERR_=1; if type
    \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack:
    :get_chromeos_release_milestone|_get_lsb_release_content|run] ->
    ssh_run(cat \\\"/etc/lsb-release\\\")\";fi; cat \"/etc/lsb-release\""
Exit status: 255
Duration: 63.1770660877

stderr:
ssh: connect to host chromeos4-row5-rack13-host5 port 22: Connection timed out). Exception log follows the after_iteration_hooks.
11/20 09:11:17.243 DEBUG|              test:0415| Starting after_iteration_hooks for provision_AutoUpdate
11/20 09:11:17.243 DEBUG|              test:0420| after_iteration_hooks completed
11/20 09:11:17.247 WARNI|              test:0637| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/test.py", line 631, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 837, in _call_test_function
    raise error.UnhandledTestFail(e)
UnhandledTestFail: Unhandled AutoservSSHTimeout: ('ssh timed out', * Command: 
    /usr/bin/ssh -a -x    -o ControlPath=/tmp/_autotmp_h1e7gjssh-
    master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
    -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
    ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
    chromeos4-row5-rack13-host5 "export LIBC_FATAL_STDERR_=1; if type
    \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack:
    :get_chromeos_release_milestone|_get_lsb_release_content|run] ->
    ssh_run(cat \\\"/etc/lsb-release\\\")\";fi; cat \"/etc/lsb-release\""
Exit status: 255
Duration: 63.1770660877

stderr:
ssh: connect to host chromeos4-row5-rack13-host5 port 22: Connection timed out)
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/test.py", line 831, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 495, in execute
    dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 362, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 400, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/server/site_tests/provision_AutoUpdate/provision_AutoUpdate.py", line 121, in run_once
    with_cheets=with_cheets)
  File "/usr/local/autotest/server/afe_utils.py", line 124, in machine_install_and_update_labels
    *args, **dargs)
  File "/usr/local/autotest/server/hosts/cros_host.py", line 805, in machine_install_by_devserver
    force_original = self.get_chromeos_release_milestone() is None
  File "/usr/local/autotest/server/hosts/cros_host.py", line 1398, in get_chromeos_release_milestone
    lsb_release_content=self._get_lsb_release_content())
  File "/usr/local/autotest/server/hosts/cros_host.py", line 1377, in _get_lsb_release_content
    'cat "%s"' % client_constants.LSB_RELEASE).stdout.strip()
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 318, in run
    return self.run_very_slowly(*args, **kwargs)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 307, in run_very_slowly
    ssh_failure_retry_ok)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 249, in _run
    raise error.AutoservSSHTimeout("ssh timed out", result)
AutoservSSHTimeout: ('ssh timed out', * Command: 
    /usr/bin/ssh -a -x    -o ControlPath=/tmp/_autotmp_h1e7gjssh-
    master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
    -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
    ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
    chromeos4-row5-rack13-host5 "export LIBC_FATAL_STDERR_=1; if type
    \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack:
    :get_chromeos_release_milestone|_get_lsb_release_content|run] ->
    ssh_run(cat \\\"/etc/lsb-release\\\")\";fi; cat \"/etc/lsb-release\""
Exit status: 255
Duration: 63.1770660877

stderr:
ssh: connect to host chromeos4-row5-rack13-host5 port 22: Connection timed out)
 

Comment 1 by nxia@chromium.org, Nov 22 2017

chromeos4-row5-rack13-host5 has been failing at provision, locking the dut and filing a repair bug.

nxia@nxia:~/chromiumos/src/third_party/autotest/files/site-packages$ dut-status -f chromeos4-row5-rack13-host5
chromeos4-row5-rack13-host5
    2017-11-22 12:59:39  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4713120-provision/
    2017-11-22 12:24:57  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712873-verify/
    2017-11-22 11:30:27  NO http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712587-repair/
    2017-11-22 11:24:20  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712579-verify/
    2017-11-22 10:28:01  NO http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712368-repair/
    2017-11-22 10:18:17  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712352-provision/
    2017-11-22 08:54:20  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712293-repair/
    2017-11-22 08:53:12  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712288-verify/
    2017-11-22 08:11:01  NO http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712087-repair/
    2017-11-22 08:01:15  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4712052-provision/
    2017-11-22 06:57:31  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4711984-repair/
    2017-11-22 06:52:10  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row5-rack13-host5/4711974-verify/



Comment 2 by nxia@chromium.org, Nov 22 2017

Filed the repair bug at b/69681430

Comment 3 by nxia@chromium.org, Nov 22 2017

Status: Fixed (was: Untriaged)
swapped chromeos4-row5-rack13-host5 with chromeos4-row10-rack4-host19

Sign in to add a comment