guado_moblab failed at HWTest stage due to this weird issue:
https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/6767
Here is the suite log:
Autotest instance created: cautotest
08-02-2017 [14:00:03] Created suite job: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=132240809
@@@STEP_LINK@Link to suite@http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=132240809@@@
08-02-2017 [14:27:48] Suite job is finished.
08-02-2017 [14:27:48] Start collecting test results and dump them to json.
Suite job [ PASSED ]
provision [ FAILED ]
provision FAIL: completed successfully, list index out of range
provision [ FAILED ]
provision FAIL: completed successfully, list index out of range
Here is the provision test log:
08/02 14:24:49.486 INFO | cros_repair:0099| Cannot determine battery status - skipping check.
08/02 14:24:49.487 INFO | server_job:0214| GOOD ---- verify.power timestamp=1501709089 localtime=Aug 02 14:24:49
08/02 14:24:49.487 INFO | repair:0327| Verifying this condition: The firmware on this DUT is up-to-date
08/02 14:24:49.621 DEBUG| ssh_host:0296| Running (ssh) 'crossystem fwid' from '_verify_host|verify|_get_rw_firmware|run|wrapper|run_very_slowly'
08/02 14:24:49.761 DEBUG| utils:0299| [stdout] Google_Guado.6301.108.4
08/02 14:24:49.763 INFO | server_job:0214| GOOD ---- verify.rwfw timestamp=1501709089 localtime=Aug 02 14:24:49
08/02 14:24:49.763 INFO | repair:0327| Verifying this condition: Python on the host is installed and working
08/02 14:24:49.770 DEBUG| ssh_host:0296| Running (ssh) 'python -c "import cPickle"' from '_verify_list|_verify_host|verify|run|wrapper|run_very_slowly'
08/02 14:24:49.930 INFO | server_job:0214| GOOD ---- verify.python timestamp=1501709089 localtime=Aug 02 14:24:49
08/02 14:24:49.931 INFO | repair:0327| Verifying this condition: Legacy host verification checks
08/02 14:24:49.931 DEBUG| autotest:0119| Using existing host autodir: /usr/local/autodir
08/02 14:24:49.931 INFO | base_classes:0353| Checking for >= 0.7 GB of space under /usr/local/autodir on machine chromeos2-row1-rack8-host7
08/02 14:24:49.939 DEBUG| ssh_host:0296| Running (ssh) 'df -PB 1000000 /usr/local/autodir | tail -1' from 'verify_software|verify_software|check_diskspace|run|wrapper|run_very_slowly'
08/02 14:24:50.083 ERROR| utils:0280| [stderr] df: /usr/local/autodir: No such file or directory
08/02 14:24:50.086 ERROR| repair:0332| Failed: Legacy host verification checks
Traceback (most recent call last):
File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 329, in _verify_host
self.verify(host)
File "/usr/local/autotest/server/hosts/repair.py", line 55, in verify
host.verify_software()
File "/usr/local/autotest/server/hosts/cros_host.py", line 1536, in verify_software
super(CrosHost, self).verify_software()
File "/usr/local/autotest/server/hosts/abstract_ssh.py", line 755, in verify_software
self.AUTOTEST_GB_DISKSPACE_REQUIRED)
File "/usr/local/autotest/client/common_lib/hosts/base_classes.py", line 355, in check_diskspace
free_space_gb = int(df[3]) / mb_per_gb
IndexError: list index out of range
08/02 14:24:50.087 INFO | server_job:0214| FAIL ---- verify.cros timestamp=1501709090 localtime=Aug 02 14:24:50 list index out of range
08/02 14:24:50.087 INFO | repair:0105| Skipping this operation: All host verification checks pass
08/02 14:24:50.087 DEBUG| repair:0106| The following dependencies failed:
08/02 14:24:50.087 DEBUG| repair:0108| Legacy host verification checks
08/02 14:24:50.087 ERROR| control:0077| Provision failed due to Exception.
Traceback (most recent call last):
File "/usr/local/autotest/results/hosts/chromeos2-row1-rack8-host7/638505-provision/20170208141516/control.srv", line 54, in provision_machine
host.verify()
File "/usr/local/autotest/server/hosts/cros_host.py", line 1571, in verify
self._repair_strategy.verify(self)
File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 685, in verify
self._verify_root._verify_host(host, silent)
File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 326, in _verify_host
self._verify_dependencies(host, silent)
File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 199, in _verify_dependencies
self._verify_list(host, self._dependency_list, silent)
File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 188, in _verify_list
raise AutoservVerifyDependencyError(self, failures)
AutoservVerifyDependencyError: list index out of range
08/02 14:24:50.088 INFO | server_job:0214| END FAIL ---- provision timestamp=1501709090 localtime=Aug 02 14:24:50
From the log, it shows that there is no /usr/local/autodir on host chromeos2-row1-rack8-host7.
However, I login into that host and find that the autodir has been there since 5/15
localhost ~ # ls -l /usr/local/ | grep autodir
drwxr-xr-x 12 root root 4096 May 15 23:50 autodir
And the failed disk checking command also ran without any error:
localhost ~ # df -PB 1000000 /usr/local/autodir | tail -1
/dev/sda1 121103 1363 113544 2% /usr/local
I think this is probably a flake. Filed this bug for tracking whether it would happen again.
Comment 1 by shuqianz@chromium.org
, Aug 3 2017Status: Duplicate (was: Untriaged)