New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 678464 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Last visit > 30 days ago
Closed: Jan 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

Running Platform_InstallTestImage fails with 'ssh_failure_retry_ok' error.

Project Member Reported by aashuto...@chromium.org, Jan 5 2017

Issue description

Any idea what is causing this?

scripts $test_that -b hana --autotest_dir=../third_party/autotest/files/ --args "servo_host=100.96.49.99" 100.96.48.179 platform_InstallTestImage
WARNING:root:Failed to import ts_mon, monitoring is disabled: No module named urllib.parse
INFO:root:Identity added: /tmp/test_that_results_3vJJ2z/testing_rsa (/tmp/test_that_results_3vJJ2z/testing_rsa)
17:35:20 INFO | Began logging to /tmp/test_that_results_3vJJ2z
Adding labels [u'cros-version:ad_hoc_build', u'board:hana'] to host 100.96.48.179
19:35:20 INFO | Fetching suite for job named platform_InstallTestImage...
19:35:23 INFO | Scheduling suite for job named platform_InstallTestImage...
19:35:23 INFO | ... scheduled 1 job(s).
19:35:23 INFO | autoserv| WARNING:root:Failed to import ts_mon, monitoring is disabled: No module named urllib.parse
19:35:24 INFO | autoserv| Results placed in /tmp/test_that_results_3vJJ2z/results-1-platform_InstallTestImage
19:35:24 INFO | autoserv| Logged pid 329 to /tmp/test_that_results_3vJJ2z/results-1-platform_InstallTestImage/.autoserv_execute
19:35:24 INFO | autoserv| I am PID 329
19:35:24 INFO | autoserv| Starting master ssh connection '/usr/bin/ssh -a -x -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_WctTrNssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmpwldDAA -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=300 -l root -p 22 100.96.48.179'
19:35:25 INFO | autoserv| Starting master ssh connection '/usr/bin/ssh -a -x   -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_vNM_lessh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 100.96.48.179'
19:35:26 INFO | autoserv| get_network_stats: at-start RXbytes 4619065 TXbytes 270227
19:35:26 INFO | autoserv| Not checking if job_repo_url contains autotest packages on ['100.96.48.179']
19:35:26 INFO | autoserv| Processing control file
19:35:26 INFO | autoserv| Starting master ssh connection '/usr/bin/ssh -a -x -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_iIL5FTssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmpmJPG6G -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=300 -l root -p 22 100.96.48.179'
19:35:27 INFO | autoserv| Verifying this condition: host is available via ssh
19:35:27 INFO | autoserv| Failed: host is available via ssh
19:35:27 INFO | autoserv| Traceback (most recent call last):
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/client/common_lib/hosts/repair.py", line 329, in _verify_host
19:35:27 INFO | autoserv| self.verify(host)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/repair.py", line 26, in verify
19:35:27 INFO | autoserv| if host.is_up():
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/abstract_ssh.py", line 552, in is_up
19:35:27 INFO | autoserv| base_cmd=base_cmd)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/abstract_ssh.py", line 526, in ssh_ping
19:35:27 INFO | autoserv| ssh_failure_retry_ok=True)
19:35:27 INFO | autoserv| TypeError: run() got an unexpected keyword argument 'ssh_failure_retry_ok'
19:35:27 INFO | autoserv| Attempting this repair action: Power cycle the host with RPM
19:35:27 INFO | autoserv| Repair failed: Power cycle the host with RPM
19:35:27 INFO | autoserv| Traceback (most recent call last):
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/client/common_lib/hosts/repair.py", line 447, in _repair_host
19:35:27 INFO | autoserv| self.repair(host)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/repair.py", line 91, in repair
19:35:27 INFO | autoserv| '%s has no RPM connection.' % host.hostname)
19:35:27 INFO | autoserv| AutoservRepairError: 100.96.49.99 has no RPM connection.
19:35:27 INFO | autoserv| Skipping this operation: Start servod with the proper config settings.
19:35:27 INFO | autoserv| Skipping this operation: Wait for update, then reboot servo host.
19:35:27 INFO | autoserv| Skipping this operation: servo host software is up-to-date
19:35:27 INFO | autoserv| Skipping this operation: servo BOARD setting is correct
19:35:27 INFO | autoserv| Skipping this operation: servo SERIAL setting is correct
19:35:27 INFO | autoserv| Skipping this operation: servod upstart job is running
19:35:27 INFO | autoserv| Skipping this operation: servod service is taking calls
19:35:27 INFO | autoserv| Skipping this operation: pwr_button control is normal
19:35:27 INFO | autoserv| Skipping this operation: servo BOARD setting is correct
19:35:27 INFO | autoserv| Skipping this operation: servo SERIAL setting is correct
19:35:27 INFO | autoserv| Skipping this operation: servod upstart job is running
19:35:27 INFO | autoserv| Skipping this operation: servod service is taking calls
19:35:27 INFO | autoserv| Skipping this operation: lid_open control is normal
19:35:27 INFO | autoserv| Skipping this operation: All host verification checks pass
19:35:27 INFO | autoserv| Exception escaped control file, job aborting:
19:35:27 INFO | autoserv| Traceback (most recent call last):
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/server_job.py", line 780, in run
19:35:27 INFO | autoserv| self._execute_code(server_control_file, namespace)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/server_job.py", line 1280, in _execute_code
19:35:27 INFO | autoserv| execfile(code_file, namespace, namespace)
19:35:27 INFO | autoserv| File "/tmp/test_that_results_3vJJ2z/results-1-platform_InstallTestImage/control.srv", line 71, in <module>
19:35:27 INFO | autoserv| parallel_simple(run, machines)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/subcommand.py", line 93, in parallel_simple
19:35:27 INFO | autoserv| function(arg)
19:35:27 INFO | autoserv| File "/tmp/test_that_results_3vJJ2z/results-1-platform_InstallTestImage/control.srv", line 36, in run
19:35:27 INFO | autoserv| host = hosts.create_host(machine, servo_args=servo_args)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/factory.py", line 188, in create_host
19:35:27 INFO | autoserv| host_instance = host_class(hostname, **args)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/base_classes.py", line 56, in __init__
19:35:27 INFO | autoserv| super(Host, self).__init__(*args, **dargs)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/client/common_lib/hosts/base_classes.py", line 69, in __init__
19:35:27 INFO | autoserv| self._initialize(*args, **dargs)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/cros_host.py", line 305, in _initialize
19:35:27 INFO | autoserv| try_servo_repair=try_servo_repair)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/servo_host.py", line 850, in create_servo_host
19:35:27 INFO | autoserv| newhost.repair(silent=True)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/servo_host.py", line 622, in repair
19:35:27 INFO | autoserv| self._repair_strategy.repair(self, silent)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/client/common_lib/hosts/repair.py", line 703, in repair
19:35:27 INFO | autoserv| self._verify_root._verify_host(host, silent)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/client/common_lib/hosts/repair.py", line 326, in _verify_host
19:35:27 INFO | autoserv| self._verify_dependencies(host, silent)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/client/common_lib/hosts/repair.py", line 199, in _verify_dependencies
19:35:27 INFO | autoserv| self._verify_list(host, self._dependency_list, silent)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/client/common_lib/hosts/repair.py", line 188, in _verify_list
19:35:27 INFO | autoserv| raise AutoservVerifyDependencyError(self, failures)
19:35:27 INFO | autoserv| AutoservVerifyDependencyError: run() got an unexpected keyword argument 'ssh_failure_retry_ok'
19:35:27 INFO | autoserv| INFO	----	----	timestamp=1483580127	job_abort_reason=run() got an unexpected keyword argument 'ssh_failure_retry_ok'	localtime=Jan 04 19:35:27	run() got an unexpected keyword argument 'ssh_failure_retry_ok'
19:35:27 INFO | autoserv| {'drone': 'aashutoshk.mtv.corp.google.com', 'user': 'autotest_system', 'job_started': 1483580124, 'hostname': '100.96.48.179', 'status_version': 1, 'experimental': 'False', 'label': 'ad_hoc_build/ad_hoc_suite/platform_InstallTestImage'}
19:35:27 INFO | autoserv| MACHINE NAME: 100.96.48.179
19:35:27 INFO | autoserv| Unable to parse host keyval for 100.96.48.179
19:35:27 INFO | autoserv| MACHINE GROUP:
19:35:27 INFO | autoserv| parsing partial test ---- SERVER_JOB
19:35:27 INFO | autoserv| 
19:35:27 INFO | autoserv| STATUS: INFO	----	----	timestamp=1483580127	job_abort_reason=run() got an unexpected keyword argument 'ssh_failure_retry_ok'	localtime=Jan 04 19:35:27	run() got an unexpected keyword argument 'ssh_failure_retry_ok'
19:35:27 INFO | autoserv| parsing test ---- SERVER_JOB
19:35:27 INFO | autoserv| Starting master ssh connection '/usr/bin/ssh -a -x -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_C2ibB2ssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmpZSvToT -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=300 -l root -p 22 100.96.48.179'
19:35:29 INFO | autoserv| Starting master ssh connection '/usr/bin/ssh -a -x   -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_autBSmssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 100.96.48.179'
19:35:29 INFO | autoserv| There are no orphaned crashes; deleting /tmp/test_that_results_3vJJ2z/results-1-platform_InstallTestImage/crashinfo.100.96.48.179
19:35:29 INFO | autoserv| Starting master ssh connection '/usr/bin/ssh -a -x -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_sNemrfssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmpwmBHwq -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=300 -l root -p 22 100.96.48.179'
19:35:30 INFO | autoserv| Starting master ssh connection '/usr/bin/ssh -a -x   -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_fZaWJEssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 100.96.48.179'
19:35:31 INFO | autoserv| get_network_stats: at-end RXbytes 4662791 TXbytes 306550
19:35:34 INFO | autoserv| Traceback (most recent call last):
19:35:34 INFO | autoserv| File "../third_party/autotest/files/server/autoserv", line 572, in run_autoserv
19:35:34 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/site_server_job.py", line 65, in run
19:35:34 INFO | autoserv| super(site_server_job, self).run(*args, **dargs)
19:35:34 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/server_job.py", line 780, in run
19:35:34 INFO | autoserv| self._execute_code(server_control_file, namespace)
19:35:34 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/server_job.py", line 1280, in _execute_code
19:35:34 INFO | autoserv| execfile(code_file, namespace, namespace)
19:35:34 INFO | autoserv| File "/tmp/test_that_results_3vJJ2z/results-1-platform_InstallTestImage/control.srv", line 71, in <module>
19:35:34 INFO | autoserv| parallel_simple(run, machines)
19:35:34 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/subcommand.py", line 93, in parallel_simple
19:35:34 INFO | autoserv| function(arg)
19:35:34 INFO | autoserv| File "/tmp/test_that_results_3vJJ2z/results-1-platform_InstallTestImage/control.srv", line 36, in run
19:35:34 INFO | autoserv| host = hosts.create_host(machine, servo_args=servo_args)
19:35:34 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/factory.py", line 188, in create_host
19:35:34 INFO | autoserv| host_instance = host_class(hostname, **args)
19:35:34 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/base_classes.py", line 56, in __init__
19:35:34 INFO | autoserv| super(Host, self).__init__(*args, **dargs)
19:35:34 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/client/common_lib/hosts/base_classes.py", line 69, in __init__
19:35:34 INFO | autoserv| self._initialize(*args, **dargs)
19:35:34 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/cros_host.py", line 305, in _initialize
19:35:34 INFO | autoserv| try_servo_repair=try_servo_repair)
19:35:34 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/servo_host.py", line 850, in create_servo_host
19:35:34 INFO | autoserv| newhost.repair(silent=True)
19:35:34 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/servo_host.py", line 622, in repair
19:35:34 INFO | autoserv| self._repair_strategy.repair(self, silent)
19:35:34 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/client/common_lib/hosts/repair.py", line 703, in repair
19:35:34 INFO | autoserv| self._verify_root._verify_host(host, silent)
19:35:34 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/client/common_lib/hosts/repair.py", line 326, in _verify_host
19:35:34 INFO | autoserv| self._verify_dependencies(host, silent)
19:35:34 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/client/common_lib/hosts/repair.py", line 199, in _verify_dependencies
19:35:34 INFO | autoserv| self._verify_list(host, self._dependency_list, silent)
19:35:34 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/client/common_lib/hosts/repair.py", line 188, in _verify_list
19:35:34 INFO | autoserv| raise AutoservVerifyDependencyError(self, failures)
19:35:34 INFO | autoserv| AutoservVerifyDependencyError: run() got an unexpected keyword argument 'ssh_failure_retry_ok'
19:35:34 INFO | autoserv| 1
19:35:34 INFO | autoserv| Traceback (most recent call last):
19:35:34 INFO | autoserv| File "../third_party/autotest/files/server/autoserv", line 773, in main
19:35:34 INFO | autoserv| File "../third_party/autotest/files/server/autoserv", line 586, in run_autoserv
19:35:34 INFO | autoserv| SystemExit: 1
19:35:34 INFO | autoserv| record_state_duration failed: job_or_task_id=None, hostname=100.96.48.179, status=Running
-----------------------------------------------------------------------------
/tmp/test_that_results_3vJJ2z/results-1-platform_InstallTestImage [  FAILED  ]
-----------------------------------------------------------------------------



 
Cc: nxia@chromium.org

Comment 2 by nxia@chromium.org, Jan 7 2017

19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/repair.py", line 26, in verify
19:35:27 INFO | autoserv| if host.is_up():
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/abstract_ssh.py", line 552, in is_up
19:35:27 INFO | autoserv| base_cmd=base_cmd)
19:35:27 INFO | autoserv| File "/mnt/host/source/src/third_party/autotest/files/server/hosts/abstract_ssh.py", line 526, in ssh_ping
19:35:27 INFO | autoserv| ssh_failure_retry_ok=True)
19:35:27 INFO | autoserv| TypeError: run() got an unexpected keyword argument 'ssh_failure_retry_ok'



You added ssh_failure_retry_ok which isn't supported?

Comment 3 by nxia@chromium.org, Jan 7 2017

The bug has been reverted at https://chromium-review.googlesource.com/#/c/425682/ 

Syncing to the latest code should solve this issue.
Status: Fixed (was: Assigned)
Will check and reopen if I see any issues. 
Status: Verified (was: Fixed)

Sign in to add a comment