New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 751885 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 751895
Owner: ----
Closed: Aug 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

provision completely successfully, but the test failed for host verification due to index out of list

Project Member Reported by shuqianz@chromium.org, Aug 2 2017

Issue description

guado_moblab failed at HWTest stage due to this weird issue:
https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/6767

Here is the suite log:
  Autotest instance created: cautotest
  08-02-2017 [14:00:03] Created suite job: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=132240809
  @@@STEP_LINK@Link to suite@http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=132240809@@@
  08-02-2017 [14:27:48] Suite job is finished.
  08-02-2017 [14:27:48] Start collecting test results and dump them to json.
  Suite job   [ PASSED ]
  provision   [ FAILED ]
  provision     FAIL: completed successfully, list index out of range
  provision   [ FAILED ]
  provision     FAIL: completed successfully, list index out of range

Here is the provision test log:

08/02 14:24:49.486 INFO |       cros_repair:0099| Cannot determine battery status - skipping check.
08/02 14:24:49.487 INFO |        server_job:0214| 	GOOD	----	verify.power	timestamp=1501709089	localtime=Aug 02 14:24:49	
08/02 14:24:49.487 INFO |            repair:0327| Verifying this condition: The firmware on this DUT is up-to-date
08/02 14:24:49.621 DEBUG|          ssh_host:0296| Running (ssh) 'crossystem fwid' from '_verify_host|verify|_get_rw_firmware|run|wrapper|run_very_slowly'
08/02 14:24:49.761 DEBUG|             utils:0299| [stdout] Google_Guado.6301.108.4
08/02 14:24:49.763 INFO |        server_job:0214| 	GOOD	----	verify.rwfw	timestamp=1501709089	localtime=Aug 02 14:24:49	
08/02 14:24:49.763 INFO |            repair:0327| Verifying this condition: Python on the host is installed and working
08/02 14:24:49.770 DEBUG|          ssh_host:0296| Running (ssh) 'python -c "import cPickle"' from '_verify_list|_verify_host|verify|run|wrapper|run_very_slowly'
08/02 14:24:49.930 INFO |        server_job:0214| 	GOOD	----	verify.python	timestamp=1501709089	localtime=Aug 02 14:24:49	
08/02 14:24:49.931 INFO |            repair:0327| Verifying this condition: Legacy host verification checks
08/02 14:24:49.931 DEBUG|          autotest:0119| Using existing host autodir: /usr/local/autodir
08/02 14:24:49.931 INFO |      base_classes:0353| Checking for >= 0.7 GB of space under /usr/local/autodir on machine chromeos2-row1-rack8-host7
08/02 14:24:49.939 DEBUG|          ssh_host:0296| Running (ssh) 'df -PB 1000000 /usr/local/autodir | tail -1' from 'verify_software|verify_software|check_diskspace|run|wrapper|run_very_slowly'
08/02 14:24:50.083 ERROR|             utils:0280| [stderr] df: /usr/local/autodir: No such file or directory
08/02 14:24:50.086 ERROR|            repair:0332| Failed: Legacy host verification checks
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 329, in _verify_host
    self.verify(host)
  File "/usr/local/autotest/server/hosts/repair.py", line 55, in verify
    host.verify_software()
  File "/usr/local/autotest/server/hosts/cros_host.py", line 1536, in verify_software
    super(CrosHost, self).verify_software()
  File "/usr/local/autotest/server/hosts/abstract_ssh.py", line 755, in verify_software
    self.AUTOTEST_GB_DISKSPACE_REQUIRED)
  File "/usr/local/autotest/client/common_lib/hosts/base_classes.py", line 355, in check_diskspace
    free_space_gb = int(df[3]) / mb_per_gb
IndexError: list index out of range
08/02 14:24:50.087 INFO |        server_job:0214| 	FAIL	----	verify.cros	timestamp=1501709090	localtime=Aug 02 14:24:50	list index out of range
08/02 14:24:50.087 INFO |            repair:0105| Skipping this operation: All host verification checks pass
08/02 14:24:50.087 DEBUG|            repair:0106| The following dependencies failed:
08/02 14:24:50.087 DEBUG|            repair:0108|     Legacy host verification checks
08/02 14:24:50.087 ERROR|           control:0077| Provision failed due to Exception.
Traceback (most recent call last):
  File "/usr/local/autotest/results/hosts/chromeos2-row1-rack8-host7/638505-provision/20170208141516/control.srv", line 54, in provision_machine
    host.verify()
  File "/usr/local/autotest/server/hosts/cros_host.py", line 1571, in verify
    self._repair_strategy.verify(self)
  File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 685, in verify
    self._verify_root._verify_host(host, silent)
  File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 326, in _verify_host
    self._verify_dependencies(host, silent)
  File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 199, in _verify_dependencies
    self._verify_list(host, self._dependency_list, silent)
  File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 188, in _verify_list
    raise AutoservVerifyDependencyError(self, failures)
AutoservVerifyDependencyError: list index out of range
08/02 14:24:50.088 INFO |        server_job:0214| END FAIL	----	provision	timestamp=1501709090	localtime=Aug 02 14:24:50	

From the log, it shows that there is no /usr/local/autodir on host chromeos2-row1-rack8-host7.

However, I login into that host and find that the autodir has been there since 5/15
localhost ~ # ls -l /usr/local/ | grep autodir                                                                                                                                                              
drwxr-xr-x 12 root   root    4096 May 15 23:50 autodir

And the failed disk checking command also ran without any error:
localhost ~ # df -PB 1000000 /usr/local/autodir | tail -1                                                                                                                                                   
/dev/sda1              121103  1363    113544       2% /usr/local

I think this is probably a flake. Filed this bug for tracking whether it would happen again.

 
Mergedinto: 751895
Status: Duplicate (was: Untriaged)
It turns out to be caused by a bad CL

Sign in to add a comment