New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 610823 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Last visit > 30 days ago
Closed: May 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

FAFT DUTs in the lab failed to repair due to the command "mosys ec info"

Project Member Reported by waihong@chromium.org, May 10 2016

Issue description

Some of the DUTs in the lab failed to repair. As the new-added check of EC existence requires executing the command "mosys ec info" on DUT; however, the DUT may be offline.

For example: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos1-row1-rack10-host4/55151442-repair/debug/

05/10 11:15:29.412 INFO |      abstract_ssh:0735| Master ssh connection to chromeos1-row1-rack10-host4 is down.
05/10 11:15:30.415 INFO |      abstract_ssh:0749| Starting master ssh connection '/usr/bin/ssh -a -x   -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_ECzVIxssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 chromeos1-row1-rack10-host4'
05/10 11:15:35.462 INFO |      abstract_ssh:0764| Timed out waiting for master-ssh connection to be established.
05/10 11:17:38.921 ERROR|            repair:0397| Repair failed: Re-install the stable firmware
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 395, in _repair_host
    self.repair(host)
  File "/usr/local/autotest/server/hosts/cros_repair.py", line 280, in repair
    host.firmware_install()
  File "/usr/local/autotest/server/hosts/cros_host.py", line 892, in firmware_install
    if self.get_ec():
  File "/usr/local/autotest/server/hosts/cros_host.py", line 1909, in get_ec
    ecinfo = self.run(command=cmd, ignore_status=True)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 190, in run
    options, stdin, args, ignore_timeout)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 151, in _run
    raise error.AutoservSSHTimeout("ssh timed out", result)
AutoservSSHTimeout: ('ssh timed out', * Command: 
    /usr/bin/ssh -a -x    -o ControlPath=/tmp/_autotmp_ECzVIxssh-
    master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
    -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
    ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
    chromeos1-row1-rack10-host4 "export LIBC_FATAL_STDERR_=1; if type
    \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\"
    \"server[stack::repair|firmware_install|get_ec] -> ssh_run(mosys ec
    info)\";fi; mosys ec info"
Exit status: 255
Duration: 123.418763161

stderr:
ssh: connect to host chromeos1-row1-rack10-host4 port 22: Connection timed out)
05/10 11:17:38.924 INFO |        server_job:0128| 		FAIL	----	repair.firmware	timestamp=1462904258	localtime=May 10 11:17:38	('ssh timed out', * Command: 
      /usr/bin/ssh -a -x    -o ControlPath=/tmp/_autotmp_ECzVIxssh-
      master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
      -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
      ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
      chromeos1-row1-rack10-host4 "export LIBC_FATAL_STDERR_=1; if type
      \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\"
      \"server[stack::repair|firmware_install|get_ec] -> ssh_run(mosys ec
      info)\";fi; mosys ec info"
  Exit status: 255
  Duration: 123.418763161
  
  stderr:
  ssh: connect to host chromeos1-row1-rack10-host4 port 22: Connection timed out)
05/10 11:17:38.925 INFO |        server_job:0128| 	END FAIL	----	repair.firmware	timestamp=1462904258	localtime=May 10 11:17:38

Should use other way to check the EC existence, like FAFT config.
 
Project Member

Comment 1 by bugdroid1@chromium.org, May 11 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/552fee1a0708a49434a5f90000b7420080626064

commit 552fee1a0708a49434a5f90000b7420080626064
Author: Tom Wai-Hong Tam <waihong@chromium.org>
Date: Tue May 10 22:27:58 2016

Use another way to check EC existence for offlined DUT

The original way requires DUT to execute a command. It may not
work for offlined DUT. Use another way to check.

BUG= chromium:610823 
TEST=Ran the following code to verify:

import common
from autotest_lib.server import hosts

h = hosts.create_host(dut, servo_args={})
h.firmware_install()

Change-Id: I91780e57779958896b0b3fb034ff0545efcce6ed
Reviewed-on: https://chromium-review.googlesource.com/344005
Tested-by: Wai-Hong Tam <waihong@chromium.org>
Reviewed-by: danny chan <dchan@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/552fee1a0708a49434a5f90000b7420080626064/server/hosts/cros_host.py

Status: Fixed (was: Started)
Bulk verified
Status: Verified (was: Fixed)
bulk verified

Sign in to add a comment