https://chromegw.corp.google.com/i/chromeos/builders/veyron_mighty-paladin/builds/5549 failed at cheets_CTS_N.7.1_r6.arm.CtsDramTestCases of HWTest phase:
Error messages:
cheets_CTS_N.7.1_r6.arm.CtsDramTestCases             [ FAILED ]
cheets_CTS_N.7.1_r6.arm.CtsDramTestCases               ABORT: Timed out, did not run.


https://chromegw.corp.google.com/i/chromeos/builders/veyron_mighty-paladin/builds/5550 failed at cheets_CTS_N.7.1_r6.arm.CtsDramTestCases of HWTest phase:
Error messages:
cheets_CTS_N.7.1_r6.arm.CtsDramTestCases             [ FAILED ]
cheets_CTS_N.7.1_r6.arm.CtsDramTestCases               ABORT: Timed out, did not run.


https://chromegw.corp.google.com/i/chromeos/builders/veyron_mighty-paladin/builds/5551 failed at cheets_GTS.4.1_r2.GtsPlacementTestCases of HWTest phase:
Error messages:
cheets_GTS.4.1_r2.GtsPlacementTestCases              [ FAILED ]
cheets_GTS.4.1_r2.GtsPlacementTestCases                ABORT: Timed out, did not run.


https://chromegw.corp.google.com/i/chromeos/builders/veyron_mighty-paladin/builds/5552 failed at Suite job of HWTest phase:
Error messages:
06/07 06:24:02.072 DEBUG|        retry_util:0133| ending retries with error: <class 'autotest_lib.client.common_lib.error.NoEligibleHostException'>(NoEligibleHostException: No hosts within board:veyron_mighty satisfy arc, cts_abi_x86, cros-version:veyron_mighty-paladin/R61-9626.0.0-rc2, board:veyron_mighty, pool:cq.
Traceback (most recent call last):
  File "/usr/local/autotest/frontend/afe/json_rpc/serviceHandler.py", line 120, in dispatchRequest
    results['result'] = self.invokeServiceEndpoint(meth, args)
  File "/usr/local/autotest/frontend/afe/json_rpc/serviceHandler.py", line 160, in invokeServiceEndpoint
    return meth(*args)
  File "/usr/local/autotest/frontend/afe/rpc_handler.py", line 270, in new_fn
    return f(*args, **keyword_args)
  File "/usr/local/autotest/frontend/afe/rpc_utils.py", line 1167, in replacement
    return func(**kwargs)
  File "/usr/local/autotest/frontend/afe/rpc_interface.py", line 905, in create_job
    require_ssp=require_ssp)
  File "/usr/local/autotest/frontend/afe/rpc_utils.py", line 825, in create_job_common
    metahost_objects=metahost_objects)
  File "/usr/local/autotest/frontend/afe/rpc_utils.py", line 509, in create_new_job
    check_job_metahost_dependencies(metahost_objects, dependencies)
  File "/usr/local/autotest/frontend/afe/rpc_utils.py", line 349, in check_job_metahost_dependencies
    % (metahost.name, ', '.join(job_dependencies)))
NoEligibleHostException: No hosts within board:veyron_mighty satisfy arc, cts_abi_x86, cros-version:veyron_mighty-paladin/R61-9626.0.0-rc2, board:veyron_mighty, pool:cq.
)

I found a series of aborted jobs on the DUT chromeos4-row6-rack10-host5 for reasons I don't yet understand. I locked the DUT and rebalanced the CQ pool.

http://chromeos-server14.mtv.corp.google.com/afe/#tab_id=view_host&object_id=1770

Every aborted job I could find had been assigned to that DUT.

xixuan traced through one of the aborted jobs, and we found that this command just goes away without doing anything.

06/07 10:23:57.295 INFO |     drone_manager:0759| command = ['nice', '-n', '10', '/usr/local/autotest/server/autoserv', '-p', '-r', u'/usr/local/autotest/results/122035835-chromeos-test/chromeos4-row6-rack10-host5', '-m', u'chromeos4-row6-rack10-host5', '-u', u'chromeos-test', '-l', u'veyron_mighty-paladin/R61-9626.0.0-rc3/bvt-inline/provision_AutoUpdate.double', '-s', '--lab', 'True', '-P', u'122035835-chromeos-test/chromeos4-row6-rack10-host5', '-n', '/usr/local/autotest/results/drone_tmp/attach.16637', '--require-ssp', '--verify_job_repo_url']
06/07 10:23:57.295 INFO |     drone_manager:0760| log file = localhost:/usr/local/autotest/results/drone_tmp/execute.16638
06/07 10:23:57.296 INFO |     drone_manager:0785| monitoring pidfile /usr/local/autotest/results/122035835-chromeos-test/chromeos4-row6-rack10-host5/.autoserv_execute

traceback for manually kicking off the cmd on shard:

chromeos-test@chromeos-server14:~$ nice -n 10 /usr/local/autotest/server/autoserv -p -r /tmp/122035835-chromeos-test/chromeos4-row6-rack10-host5 -m chromeos4-r
ow6-rack10-host5 -u chromeos-test -l veyron_mighty-paladin/R61-9626.0.0-rc3/bvt-inline/provision_AutoUpdate.double -s --lab True -P 122035835-chromeos-test/chromeos4-row6-rack10-host5 -n /usr/local/autotest/results/drone_tmp/attach.16637 --require-ssp --verify_job_repo_url
DEBUG:root:Refreshing HostInfo using store <autotest_lib.server.hosts.afe_store.AfeStore object at 0x7f68ef57a4d0>
DEBUG:root:Old host_info: None
DEBUG:root:New host_info: HostInfo [Labels: ['bluetooth', 'ec:cros', 'veyron_mighty', 'gpu_family:mali', 'board_freq_mem:veyron_mighty_1.8GHz_2GB', 'graphics:gles', 'hw_video_acc_vp8', 'hw_video_acc_enc_vp8', 'audio_loopback_dongle', 'cts_abi_arm', 'internal_display', 'os:cros', 'power:battery', 'hw_video_acc_h264', 'storage:mmc', 'servo', 'board:veyron_mighty', 'arc', 'webcam', 'sku:mighty_rk3288_2Gb', 'variant:mighty', 'phase:PVT2', 'touchpad', 'cros-version:veyron_mighty-paladin/R61-9626.0.0-rc3', 'pool:suites'], Attributes: {'powerunit_hostname': 'chromeos4-row5_6-rack10-rpm1', 'powerunit_outlet': '.A5'}]
DEBUG:root:Running (ssh) 'test ! -e /var/log/messages || cp -f /var/log/messages /var/tmp/messages.autotest_start'
INFO:root:Starting master ssh connection '/usr/bin/ssh -a -x   -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_z8D9j2ssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 chromeos4-row6-rack10-host5'
DEBUG:root:Running '/usr/bin/ssh -a -x   -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_z8D9j2ssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 chromeos4-row6-rack10-host5'
INFO:root:Getting devservers for host: None
Traceback (most recent call last):
  File "/usr/local/autotest/server/autoserv", line 782, in <module>
    main()
  File "/usr/local/autotest/server/autoserv", line 628, in main
    ssp_url, ssp_error_msg = _stage_ssp(parser)
  File "/usr/local/autotest/server/autoserv", line 134, in _stage_ssp
    execfile(STAGE_SERVER_SIDE_PACKAGE_CONTROL_FILE, namespace, script_locals)
  File "/usr/local/autotest/server/control_segments/stage_server_side_package", line 21, in <module>
    ssp_url = host.stage_server_side_package(image)
  File "/usr/local/autotest/server/hosts/cros_host.py", line 463, in stage_server_side_package
    match = re.match('.*/R\d+-(\d+)\.', image_name)
UnboundLocalError: local variable 'image_name' referenced before assignment

This is a fallout of  issue 730272  and the fact that stage_server_side_package is yucky: https://chromium-review.googlesource.com/c/527600/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/0c1dd4d190ca810b8888225480f30b81f3ac7d3a

commit 0c1dd4d190ca810b8888225480f30b81f3ac7d3a
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Tue Jun 13 01:20:07 2017

[autotest] Hotfix stage_server_side_package undefined variable

This function needs to be cleaned, but I'm not going to do it right now.
This CL will make stage_server_side_package not fail in the purportedly
supported case of a DUT that has a build assigned but no job_repo_url.
This may still fail later in the flow somewhere.

The fact that nobody noticed this probably means we don't really support
this case.

BUG= chromium:730431 
TEST=None.

Change-Id: I984caf4d70bd5b58f1d39678c8e62a0e7c19f397
Reviewed-on: https://chromium-review.googlesource.com/527600
Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Xixuan Wu <xixuan@chromium.org>

[modify] https://crrev.com/0c1dd4d190ca810b8888225480f30b81f3ac7d3a/server/hosts/cros_host.py