Example job here: http://cautotest/afe/#tab_id=view_job&object_id=68097700
Logs from that job here: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/68097700-chromeos-test/chromeos3-row1-rack2-host16/ssp_logs/debug/

Same thing on last night's run, only this occurred on a different devserver.

http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=68143730

06/30 05:49:46.235 DEBUG|          ssh_host:0180| Running (ssh) 'wget --connect-timeout=15 --retry-connrefused --wait=5 -nv http://172.22.39.162:8082/static/celes-release/R53-8458.0.0/autotest/packages/client-autotest.tar.bz2 -O /tmp/sysinfo/autoserv-UvA7NM/packages/client-autotest.tar.bz2'
06/30 05:49:46.554 ERROR|        base_utils:0278| [stderr] http://172.22.39.162:8082/static/celes-release/R53-8458.0.0/autotest/packages/client-autotest.tar.bz2:
06/30 05:49:46.555 ERROR|        base_utils:0278| [stderr] 2016-06-30 05:49:47 ERROR 404: Not Found.
06/30 05:49:46.556 DEBUG|     base_packages:0464| command execution error
* Command: 
    /usr/bin/ssh -a -x    -o ControlPath=/tmp/_autotmp_tYqQMKssh-
    master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
    -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
    ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
    chromeos3-row1-rack1-host9 "export LIBC_FATAL_STDERR_=1; if type
    \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\"
    \"server[stack::fetch_pkg|fetch_pkg_file|_run_command] -> ssh_run(wget
    --connect-timeout=15 --retry-connrefused --wait=5 -nv
    http://172.22.39.162:8082/static/celes-
    release/R53-8458.0.0/autotest/packages/client-autotest.tar.bz2 -O
    /tmp/sysinfo/autoserv-UvA7NM/packages/client-autotest.tar.bz2)\";fi; wget
    --connect-timeout=15 --retry-connrefused --wait=5 -nv
    http://172.22.39.162:8082/static/celes-
    release/R53-8458.0.0/autotest/packages/client-autotest.tar.bz2 -O
    /tmp/sysinfo/autoserv-UvA7NM/packages/client-autotest.tar.bz2"
Exit status: 8
Duration: 0.26985001564

stderr:
http://172.22.39.162:8082/static/celes-release/R53-8458.0.0/autotest/packages/client-autotest.tar.bz2:

ran the verify dev server commands on system with IP 172.22.39.164
completed with Success

chromeos-test@chromeos3-infra-devserver3:~$ curl "http://localhost:8082/stage?artifacts=full_payload,stateful,autotest_packages&files=&archive_url=gs://chromeos-image-archive/daisy-release/R53-8480.0.0"
Successchromeos-test@chromeos3-infra-devserver3:~$ curl "http://localhost:8082/sild_id=2457013&files=&target=shamu-userdebug&artifacts=bootloader_image,zip_images,test_zip&branch=git_mnc-release&os_type=android"
Successchromeos-test@chromeos3-infra-devserver3:~$ 

ran the verify dev server commands on system with IP 172.22.39.162
completed with Success

chromeos-test@chromeos3-infra-devserver1:~$ curl "http://localhost:8082/stage?artifacts=full_payload,stateful,autotest_packages&files=&archive_url=gs://chromeos-image-archive/daisy-release/R53-8480.0.0"
Successchromeos-test@chromeos3-infra-devserver1:~$ curl "http://localhost:8082/sild_id=2457013&files=&target=shamu-userdebug&artifacts=bootloader_image,zip_images,test_zip&branch=git_mnc-release&os_type=android"
Successchromeos-test@chromeos3-infra-devserver1:~$ 

Same occurred on last Friday's run, again on a different devserver. It seems this problem is devserver-wide.

http://cautotest/afe/#tab_id=view_job&object_id=68265534

07/01 22:39:43.981 DEBUG|        base_utils:0185| Running 'sudo wget --timeout=300 -nv http://172.22.39.161:8082/static/nyan_big-release/R53-8529.0.0/autotest_server_package.tar.bz2 -O /usr/local/autotest/containers/test_68265534_1467437982_18594/delta0/usr/local/autotest_server_package.tar.bz2'
07/01 22:39:44.166 WARNI|             retry:0181| <class 'autotest_lib.client.common_lib.error.CmdError'>(Command <sudo wget --timeout=300 -nv http://172.22.39.161:8082/static/nyan_big-release/R53-8529.0.0/autotest_server_package.tar.bz2 -O /usr/local/autotest/containers/test_68265534_1467437982_18594/delta0/usr/local/autotest_server_package.tar.bz2> failed, rc=4, Command returned non-zero exit status
* Command: 
    sudo wget --timeout=300 -nv http://172.22.39.161:8082/static/nyan_big-
    release/R53-8529.0.0/autotest_server_package.tar.bz2 -O /usr/local/autotes
    t/containers/test_68265534_1467437982_18594/delta0/usr/local/autotest_serv
    er_package.tar.bz2
Exit status: 4
Duration: 0.168177127838
)

The drone failed to use http protocol to wget server-side package file from the devserver in restricted network.

Code change is needed for SSP to work again. autoserv needs to resolve a new devserver not in restricted network and wget the autotest_server_package.tar.bz2 file from that devserver.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/d37736b67febeebd991d0209dd030635f7ff5cfd

commit d37736b67febeebd991d0209dd030635f7ff5cfd
Author: Dan Shi <dshi@google.com>
Date: Wed Jul 06 22:10:29 2016

[autotest] Update stage_server_side_package to support restricted subnet.

If a dut is in restricted subnet, server-side packaging uses the devserver in
the same subnet of the dut. This leads to test failure as drone, which is
outside the restricted subnet, doesn't have http access to the devserver.

This change update method stage_server_side_package so it resolves a new
devserver outside restricted subnet if needed.

BUG= chromium:624554 
TEST=local run test

Change-Id: I2c431fe7b0b8d4453978917e8a5e96fd310805f6
Reviewed-on: https://chromium-review.googlesource.com/358610
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Simran Basi <sbasi@chromium.org>

[modify] https://crrev.com/d37736b67febeebd991d0209dd030635f7ff5cfd/server/hosts/cros_host.py
[modify] https://crrev.com/d37736b67febeebd991d0209dd030635f7ff5cfd/server/site_utils.py
[modify] https://crrev.com/d37736b67febeebd991d0209dd030635f7ff5cfd/server/hosts/adb_host.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/321cd4e5c52d7be24791fe0b3265fc1a7c78c593

commit 321cd4e5c52d7be24791fe0b3265fc1a7c78c593
Author: Dan Shi <dshi@google.com>
Date: Thu Jul 07 20:16:42 2016

[autotest] Fix a bug in stage_server_side_package

version prefix in server/cros/dynamic_suite/constants.py is different from
the one in server/cros/provision.py, which doesn't have tailing `:`

Related change:
https://chromium-review.googlesource.com/#/c/358610/2

BUG= chromium:624554 
TEST=local run test

Change-Id: I3cd38792ca50c5856367ee3390e878ede635b1c3
Reviewed-on: https://chromium-review.googlesource.com/358814
Reviewed-by: Dan Shi <dshi@google.com>
Commit-Queue: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/321cd4e5c52d7be24791fe0b3265fc1a7c78c593/server/hosts/cros_host.py

Test with ssp is working now in ACL enabled lab:
http://cautotest/afe/#tab_id=view_job&object_id=68806892

Closing. please reopen if its not fixed.