New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 768110 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Oct 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

"Failed: Legacy host verification checks" should be much more obvious about what services are not running

Project Member Reported by akes...@chromium.org, Sep 23 2017

Issue description

https://luci-milo.appspot.com/buildbot/chromeos/peppy-paladin/16552

09/22 19:39:24.201 DEBUG|          ssh_host:0296| Running (ssh) 'status system-services | grep start/running' from 'verify|verify_software|upstart_status|run|wrapper|run_very_slowly'
09/22 19:39:24.319 ERROR|            repair:0332| Failed: Legacy host verification checks
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 329, in _verify_host
    self.verify(host)
  File "/usr/local/autotest/server/hosts/repair.py", line 55, in verify
    host.verify_software()
  File "/usr/local/autotest/server/hosts/cros_host.py", line 1587, in verify_software
    if not self.upstart_status('system-services'):
  File "/usr/local/autotest/server/hosts/cros_host.py", line 1551, in upstart_status
    service_name).stdout.strip() != ''
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 316, in run
    return self.run_very_slowly(*args, **kwargs)
  File "/usr/local/autotest/site-packages/chromite/lib/metrics.py", line 482, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 307, in run_very_slowly
    ssh_failure_retry_ok)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 258, in _run
    raise error.AutoservRunError("command execution error", result)
AutoservRunError: command execution error
* Command: 
    /usr/bin/ssh -a -x    -o ControlPath=/tmp/_autotmp_AdXP6Pssh-
    master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
    -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
    ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
    chromeos4-row6-rack12-host3 "export LIBC_FATAL_STDERR_=1; if type
    \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\"
    \"server[stack::upstart_status|run|wrapper] -> ssh_run(status system-
    services | grep start/running)\";fi; status system-services | grep
    start/running"


 
Labels: -Pri-3 Pri-2
Owner: jrbarnette@chromium.org
Cc: jrbarnette@chromium.org
Owner: akes...@chromium.org
Status: Started (was: Untriaged)
https://chromium-review.googlesource.com/#/c/chromiumos/third_party/autotest/+/680076/
Project Member

Comment 3 by bugdroid1@chromium.org, Sep 26 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/f2208f9be7fff6e5eaef7eb30ebe61c42c071483

commit f2208f9be7fff6e5eaef7eb30ebe61c42c071483
Author: Aviv Keshet <akeshet@chromium.org>
Date: Tue Sep 26 20:36:19 2017

cros_host: allow upstart_status to fail in more informative way

BUG= chromium:768110 
TEST=None

Change-Id: Ia0098959818a3b1d8908ad9be48805b120ac7bf0
Reviewed-on: https://chromium-review.googlesource.com/680076
Commit-Ready: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/f2208f9be7fff6e5eaef7eb30ebe61c42c071483/server/hosts/cros_host.py

Project Member

Comment 4 by bugdroid1@chromium.org, Sep 27 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/e204dc5fe5bde85ad91b16ad5a2f577d4ebd84a6

commit e204dc5fe5bde85ad91b16ad5a2f577d4ebd84a6
Author: Richard Barnette <jrbarnette@chromium.org>
Date: Wed Sep 27 01:24:12 2017

[autotest] Make CrosHost.upstart_status() more debuggable.

Change the `run()` invocation in `CrosHost.upstart_status()` so that
stdout of the 'status' command will be logged.  This will make it
easier to see what's wrong in cases where the call returns false
unexpectedly.

BUG= chromium:768110 ,chromium:765686
TEST=manually invoke the function from a Python CLI.

Change-Id: Iaeaf97e2a73802b62bc111ad717e13ed627317bc
Reviewed-on: https://chromium-review.googlesource.com/685798
Commit-Ready: Richard Barnette <jrbarnette@chromium.org>
Tested-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/e204dc5fe5bde85ad91b16ad5a2f577d4ebd84a6/server/hosts/cros_host.py

Status: Fixed (was: Started)
Presumably fixed.

Sign in to add a comment