Chrome Version: R67-10575.58.0 (Also happens on other OS versions)
OS: Chrome OS
Devices going to "Repair Failed" state. Unable to restore them by running "Repari" Job.
Stack trace on failure:
06/28 13:39:10.857 INFO | repair:0110| Skipping this operation: All host verification checks pass
06/28 13:39:10.857 DEBUG| repair:0111| The following dependencies failed:
06/28 13:39:10.857 DEBUG| repair:0113| The most recent AU attempt on this DUT succeeded
06/28 13:39:10.857 DEBUG| repair:0113| Python on the host is installed and working
06/28 13:39:10.857 ERROR| repair:0048| Repair failed due to Exception.
Traceback (most recent call last):
File "/usr/local/autotest/server/control_segments/repair", line 38, in repair
target.repair()
File "/usr/local/autotest/server/hosts/cros_host.py", line 944, in repair
self._repair_strategy.repair(self)
File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 774, in repair
self._verify_root._verify_host(host, silent)
File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 348, in _verify_host
self._verify_dependencies(host, silent)
File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 226, in _verify_dependencies
self._verify_list(host, self._dependency_list, silent)
File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 216, in _verify_list
raise AutoservVerifyDependencyError(self, failures)
AutoservVerifyDependencyError: Last AU on this DUT failed
Python is missing; may be caused by powerwash
06/28 13:39:10.858 INFO | server_job:0216| END FAIL ---- repair timestamp=1530218350 localtime=Jun 28 13:39:10
Some devices to name in this state:
chromeos15-row2-rack2-host3
chromeos15-row1-rack8-host7
chromeos15-row3-rack9-host1
chromeos15-row3-rack10-host3
chromeos15-row3-rack9-host5
Also, staging is failing:
ImageInstallError: Download and install failed from chromeos15-infra-devserver9.cros.corp.google.com onto chromeos15-row3-rack9-host5: Failed to install device image using payload at http://100.115.126.4:8082/update/chell-release/R68-10718.29.0 on chromeos15-row3-rack9-host5. : command execution error
Dev Servers are not reachable:
dsunkara@dsunkara0:~/chromiumos$ ping chromeos15-infra-devserver6
PING chromeos15-infra-devserver6.cros.corp.google.com (100.115.126.1) 56(84) bytes of data.
^C
--- chromeos15-infra-devserver6.cros.corp.google.com ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2049ms
dsunkara@dsunkara0:~/chromiumos$ ping chromeos15-infra-devserver7
PING chromeos15-infra-devserver7.cros.corp.google.com (100.115.126.2) 56(84) bytes of data.
^C
--- chromeos15-infra-devserver7.cros.corp.google.com ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2029ms
dsunkara@dsunkara0:~/chromiumos$ ping chromeos15-infra-devserver8
PING chromeos15-infra-devserver8.cros.corp.google.com (100.115.126.3) 56(84) bytes of data.
^C
--- chromeos15-infra-devserver8.cros.corp.google.com ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4090ms
dsunkara@dsunkara0:~/chromiumos$ ping chromeos15-infra-devserver9
PING chromeos15-infra-devserver9.cros.corp.google.com (100.115.126.4) 56(84) bytes of data.
^C
--- chromeos15-infra-devserver9.cros.corp.google.com ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3078ms
Comment 1 by jkop@chromium.org
, Jun 28 2018Status: Assigned (was: Untriaged)