At then end of `bin/run_server_task DevserverProvisionTask`, the
command performs two staging calls to the devserver service on
the target system (presumably as a sanity check).  Sometimes, that
check fails.  Retrying the command tends to work.

The failure looks like this:
[chromeos15-infra-devserver12.cros.corp.google.com] sudo: service devserver status
[chromeos15-infra-devserver12.cros.corp.google.com] out: sudo password:
[chromeos15-infra-devserver12.cros.corp.google.com] out: devserver start/running, process 1092
[chromeos15-infra-devserver12.cros.corp.google.com] out: 

[chromeos15-infra-devserver12.cros.corp.google.com] run: gsutil ls gs://chromeos-image-archive/daisy-release/ | tail -2 | head -1
[chromeos15-infra-devserver12.cros.corp.google.com] out: gs://chromeos-image-archive/daisy-release/R70-10934.0.0/
[chromeos15-infra-devserver12.cros.corp.google.com] out: 

[chromeos15-infra-devserver12.cros.corp.google.com] run: curl "http://localhost:8082/stage?artifacts=full_payload,stateful,autotest_packages&files=&archive_url=gs://chromeos-image-archive/daisy-release/R70-10934.0.0/"
[chromeos15-infra-devserver12.cros.corp.google.com] out: curl: (7) Failed to connect to localhost port 8082: Connection refused

At first blush, this looks like it might be a race with devserver
startup (the command is run immediately after a reboot).  Morever, at
one point I went to a server that had failed this way, and manually ran
the same 'curl' command; it passed without incident.

Reinforcing the "race condition" hypothesis:  When I saw this fail, it
was during a sequence of deployments on three different servers:
  * The first server failed, and succeeded on retry.
  * The second server failed twice, and succeeded on the third try.
  * The third server never saw this failure.

Attached is the log from the server with the single failure.

 

Deleted: deploy-12.log 37.5 KB

deploy-12.log 37.5 KB View Download