Staging jobs failing to proceed past Starting.
See e.g. http://cautotest-staging/afe/#tab_id=view_job&object_id=14472
Examining scheduler logs, it appears job didn't proceed past Starting. Likely related to recent Lucifer rollout.
chromeos-test@chromeos-staging-master2:/usr/local/autotest/logs$ grep 14472 scheduler.log.2018-03-29-*
scheduler.log.2018-03-29-02.06.10:03/29 02:07:52.166 DEBUG| ssh_host:0301| Running (ssh) 'python /usr/local/autotest/scheduler/drone_utility.py --call_time 1522314472.16' from 'worker|execute_queued_calls|_execute_calls|_execute_calls_impl|run|run_very_slowly'
scheduler.log.2018-03-29-02.06.10:03/29 02:07:52.666 DEBUG| ssh_host:0301| Running (ssh) 'python /usr/local/autotest/scheduler/drone_utility.py --call_time 1522314472.66' from 'worker|execute_queued_calls|_execute_calls|_execute_calls_impl|run|run_very_slowly'
scheduler.log.2018-03-29-17.05.43:03/29 17:10:51.387 INFO | scheduler_models:0641| HQE: 15309, for job: 14472 and host: no host has status:Queued -> Starting
scheduler.log.2018-03-29-17.05.43:03/29 17:10:51.727 DEBUG| ssh_host:0301| Running (ssh) '/usr/bin/env /usr/local/autotest/bin/job_reporter --jobdir /usr/local/autotest/leases --run-job-path /opt/infra-tools/usr/bin/lucifer_run_job --watcher-path /opt/infra-tools/usr/bin/lucifer_watcher --lucifer-level STARTING --job-id 14472 --results-dir /usr/local/autotest/results/14472-chromeos-test/hostless --execution-tag 14472-chromeos-test/hostless </dev/null >>/usr/local/autotest/results/14472-chromeos-test/hostless/lucifer/job_reporter_output.log 2>&1 &' from '_send_to_lucifer|_send_starting_to_lucifer|spawn_starting_job_handler|spawn|run|run_very_slowly'
Comment 1 by akes...@chromium.org
, Mar 30 2018