New issue
Advanced search Search tips

Issue 827447 link

Starred by 1 user

Issue metadata

Status: Duplicate
Owner:
Closed: Mar 2018
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Staging jobs failing to proceed past Starting

Project Member Reported by akes...@chromium.org, Mar 30 2018

Issue description

Staging jobs failing to proceed past Starting.

See e.g. http://cautotest-staging/afe/#tab_id=view_job&object_id=14472

Examining scheduler logs, it appears job didn't proceed past Starting. Likely related to recent Lucifer rollout.

chromeos-test@chromeos-staging-master2:/usr/local/autotest/logs$ grep 14472 scheduler.log.2018-03-29-*
scheduler.log.2018-03-29-02.06.10:03/29 02:07:52.166 DEBUG|          ssh_host:0301| Running (ssh) 'python /usr/local/autotest/scheduler/drone_utility.py --call_time 1522314472.16' from 'worker|execute_queued_calls|_execute_calls|_execute_calls_impl|run|run_very_slowly'
scheduler.log.2018-03-29-02.06.10:03/29 02:07:52.666 DEBUG|          ssh_host:0301| Running (ssh) 'python /usr/local/autotest/scheduler/drone_utility.py --call_time 1522314472.66' from 'worker|execute_queued_calls|_execute_calls|_execute_calls_impl|run|run_very_slowly'
scheduler.log.2018-03-29-17.05.43:03/29 17:10:51.387 INFO |  scheduler_models:0641| HQE: 15309, for job: 14472 and host: no host has status:Queued -> Starting
scheduler.log.2018-03-29-17.05.43:03/29 17:10:51.727 DEBUG|          ssh_host:0301| Running (ssh) '/usr/bin/env /usr/local/autotest/bin/job_reporter --jobdir /usr/local/autotest/leases --run-job-path /opt/infra-tools/usr/bin/lucifer_run_job --watcher-path /opt/infra-tools/usr/bin/lucifer_watcher --lucifer-level STARTING --job-id 14472 --results-dir /usr/local/autotest/results/14472-chromeos-test/hostless --execution-tag 14472-chromeos-test/hostless </dev/null >>/usr/local/autotest/results/14472-chromeos-test/hostless/lucifer/job_reporter_output.log 2>&1 &' from '_send_to_lucifer|_send_starting_to_lucifer|spawn_starting_job_handler|spawn|run|run_very_slowly'

 
I'm pretty sure there are more detailed lucifer logs to find somewhere, but I don't know where.
Mergedinto: 827423
Status: Duplicate (was: Assigned)

Sign in to add a comment