New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 858821 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Devices in "Repair Failed" state with reason "Python is missing; may be caused by powerwash"

Project Member Reported by dsunk...@chromium.org, Jun 28 2018

Issue description

Chrome Version: R67-10575.58.0 (Also happens on other OS versions)
OS: Chrome OS

Devices going to "Repair Failed" state. Unable to restore them by running "Repari" Job.

Stack trace on failure: 

06/28 13:39:10.857 INFO |            repair:0110| Skipping this operation: All host verification checks pass
06/28 13:39:10.857 DEBUG|            repair:0111| The following dependencies failed:
06/28 13:39:10.857 DEBUG|            repair:0113|     The most recent AU attempt on this DUT succeeded
06/28 13:39:10.857 DEBUG|            repair:0113|     Python on the host is installed and working
06/28 13:39:10.857 ERROR|            repair:0048| Repair failed due to Exception.
Traceback (most recent call last):
  File "/usr/local/autotest/server/control_segments/repair", line 38, in repair
    target.repair()
  File "/usr/local/autotest/server/hosts/cros_host.py", line 944, in repair
    self._repair_strategy.repair(self)
  File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 774, in repair
    self._verify_root._verify_host(host, silent)
  File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 348, in _verify_host
    self._verify_dependencies(host, silent)
  File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 226, in _verify_dependencies
    self._verify_list(host, self._dependency_list, silent)
  File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 216, in _verify_list
    raise AutoservVerifyDependencyError(self, failures)
AutoservVerifyDependencyError: Last AU on this DUT failed
Python is missing; may be caused by powerwash
06/28 13:39:10.858 INFO |        server_job:0216| END FAIL	----	repair	timestamp=1530218350	localtime=Jun 28 13:39:10

Some devices to name in this state:
chromeos15-row2-rack2-host3
chromeos15-row1-rack8-host7
chromeos15-row3-rack9-host1
chromeos15-row3-rack10-host3
chromeos15-row3-rack9-host5

Also, staging is failing:
ImageInstallError: Download and install failed from chromeos15-infra-devserver9.cros.corp.google.com onto chromeos15-row3-rack9-host5: Failed to install device image using payload at http://100.115.126.4:8082/update/chell-release/R68-10718.29.0 on chromeos15-row3-rack9-host5. : command execution error


Dev Servers are not reachable:
dsunkara@dsunkara0:~/chromiumos$ ping chromeos15-infra-devserver6
PING chromeos15-infra-devserver6.cros.corp.google.com (100.115.126.1) 56(84) bytes of data.
^C
--- chromeos15-infra-devserver6.cros.corp.google.com ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2049ms

dsunkara@dsunkara0:~/chromiumos$ ping chromeos15-infra-devserver7
PING chromeos15-infra-devserver7.cros.corp.google.com (100.115.126.2) 56(84) bytes of data.
^C
--- chromeos15-infra-devserver7.cros.corp.google.com ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2029ms

dsunkara@dsunkara0:~/chromiumos$ ping chromeos15-infra-devserver8
PING chromeos15-infra-devserver8.cros.corp.google.com (100.115.126.3) 56(84) bytes of data.
^C
--- chromeos15-infra-devserver8.cros.corp.google.com ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4090ms

dsunkara@dsunkara0:~/chromiumos$ ping chromeos15-infra-devserver9
PING chromeos15-infra-devserver9.cros.corp.google.com (100.115.126.4) 56(84) bytes of data.
^C
--- chromeos15-infra-devserver9.cros.corp.google.com ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3078ms




 

Comment 1 by jkop@chromium.org, Jun 28 2018

Owner: xixuan@chromium.org
Status: Assigned (was: Untriaged)
-> Deputy

Comment 2 by xixuan@chromium.org, Jun 29 2018

Status: Fixed (was: Assigned)
The devservers are powered off by lab stuff at that time. 

They're now powered on, and get updated with the latest devserver push.

I randomly check two of the DUTs: chromeos15-row2-rack2-host3 & chromeos15-row1-rack8-host7, they're back.

Sign in to add a comment