New issue
Advanced search Search tips

Issue 615591 link

Starred by 1 user

Issue metadata

Status: Verified
Owner: ----
Closed: Mar 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

power_daily suite doesn't produce child jobs

Project Member Reported by tbroch@chromium.org, May 28 2016

Issue description

power_daily suite doesn't produce child jobs.

Also not able to glean what failure mode is from available logs ...

http://cautotest/afe/#tab_id=view_job&object_id=64549928

05/25 04:15:27.896 INFO |          autoserv:0681| Results placed in /usr/local/autotest/results/64549928-chromeos-test/hostless
05/25 04:15:27.896 DEBUG|          autoserv:0689| autoserv is running in drone chromeos-server6.mtv.corp.google.com.
05/25 04:15:27.897 DEBUG|          autoserv:0690| autoserv command was: /usr/local/autotest/server/autoserv -p -r /usr/local/autotest/results/64549928-chromeos-test/hostless -u chromeos-test -l falco-release/R53-8369.0.0-test_suites/control.power_daily -s --lab True -P 64549928-chromeos-test/hostless -n /usr/local/autotest/results/drone_tmp/attach.89148
05/25 04:15:27.897 INFO |           pidfile:0016| Logged pid 1256 to /usr/local/autotest/results/64549928-chromeos-test/hostless/.autoserv_execute
05/25 04:15:27.899 DEBUG|          autoserv:0406| faulthandler registered on SIGTERM.
05/25 04:15:27.901 DEBUG|          base_job:0348| Persistent state global_properties.test_retry now set to 0
05/25 04:15:27.902 DEBUG|          base_job:0348| Persistent state global_properties.tag now set to '64549928-chromeos-test/hostless'
05/25 04:15:27.902 DEBUG|          base_job:0348| Persistent state global_properties.last_boot_tag now set to None
05/25 04:15:28.015 INFO |        server_job:0604| I am PID 1256
05/25 04:15:28.018 WARNI|        server_job:0659| Not checking if job_repo_url contains autotest packages on []
05/25 04:15:28.019 INFO |        server_job:0682| Processing control file
05/25 04:15:28.082 DEBUG|     dynamic_suite:0482| Determined own job id: 64549928
05/25 04:15:28.083 INFO |        dev_server:0931| Staging artifacts on devserver http://172.17.40.19:8082: build=falco-release/R53-8369.0.0, artifacts=['control_files', 'test_suites'], files=, archive_url=gs://chromeos-image-archive/falco-release/R53-8369.0.0
05/25 04:15:28.086 DEBUG|        base_utils:0176| Running 'ssh 172.17.40.19 'curl "http://172.17.40.19:8082/stage?artifacts=control_files,test_suites&files=&async=True&archive_url=gs://chromeos-image-archive/falco-release/R53-8369.0.0"''
05/25 04:15:29.311 DEBUG|        base_utils:0176| Running 'ssh 172.17.40.19 'curl "http://172.17.40.19:8082/is_staged?artifacts=control_files,test_suites&files=&archive_url=gs://chromeos-image-archive/falco-release/R53-8369.0.0"''
05/25 04:15:30.452 INFO |        dev_server:0953| Finished staging artifacts: build=falco-release/R53-8369.0.0, artifacts=['control_files', 'test_suites'], files=, archive_url=gs://chromeos-image-archive/falco-release/R53-8369.0.0
05/25 04:15:30.454 DEBUG|             suite:1145| Getting control file list for suite: power_daily
05/25 04:15:30.454 DEBUG|        base_utils:0176| Running 'ssh 172.17.40.19 'curl "http://172.17.40.19:8082/list_suite_controls?suite_name=power_daily&build=falco-release/R53-8369.0.0"''
05/25 04:15:31.648 DEBUG|             suite:1156| Parsing control files ...
05/25 04:15:31.666 DEBUG|             suite:1221| Parsed 1 control files.
05/25 04:15:31.667 DEBUG|     dynamic_suite:0545| delay_minutes is set. Sleeping 1440 minutes before creating test jobs.
05/26 04:15:31.767 DEBUG|     dynamic_suite:0548| Finished waiting for 1440 minutes before creating test jobs.
05/26 04:15:31.767 DEBUG|             suite:0907| Discovered 1 stable tests.
05/26 04:15:31.768 DEBUG|             suite:0909| Discovered 0 unstable tests.
05/26 04:15:31.770 INFO |        server_job:0128| INFO	----	Start power_daily	timestamp=1464261331	localtime=May 26 04:15:31	
05/26 04:15:31.770 DEBUG|             suite:0825| Scheduling Power daily tests
05/26 04:15:32.166 DEBUG|             suite:1079| Adding job keyval for Power daily tests=64677967-chromeos-test
05/26 04:15:32.167 DEBUG|     dynamic_suite:0554| Waiting on suite.
05/26 04:15:32.411 INFO |         discovery:0190| URL being requested: https://monorail-prod.appspot.com/_ah/api/discovery/v1/apis/monorail/v1/rest
05/26 04:19:37.047 DEBUG|     dynamic_suite:0556| Finished waiting on suite. Returning from _perform_reimage_and_run.
05/26 04:19:37.047 DEBUG|     dynamic_suite:0495| Returning from dynamic_suite.reimage_and_run.
05/26 04:19:37.048 INFO |        server_job:0685| Finished processing control file
05/26 04:19:37.193 DEBUG|   logging_manager:0627| Logging subprocess finished
05/26 04:19:37.193 DEBUG|   logging_manager:0627| Logging subprocess finished

This spawns suite ... but there's nothing beyond that ...

http://cautotest/afe/#tab_id=view_job&object_id=64677967

05/26 04:18:12.532 INFO |          autoserv:0681| Results placed in /usr/local/autotest/results/64677967-chromeos-test
05/26 04:18:12.533 DEBUG|          autoserv:0689| autoserv is running in drone test_64677967_1464261463_5954.
05/26 04:18:12.533 DEBUG|          autoserv:0690| autoserv command was: /usr/local/autotest/server/autoserv -p -r /usr/local/autotest/results/64677967-chromeos-test -m chromeos2-row24-rack6-host9 -u chromeos-test -l falco-release/R53-8369.0.0/power_daily/Power daily tests -s --lab True -P /usr/local/autotest/results/64677967-chromeos-test -n /usr/local/autotest/drone_tmp/attach.18840 --verify_job_repo_url --use-existing-results --pidfile-label container_autoserv
05/26 04:18:12.533 INFO |           pidfile:0016| Logged pid 1594 to /usr/local/autotest/results/64677967-chromeos-test/.container_autoserv_execute
05/26 04:18:12.545 DEBUG|          autoserv:0406| faulthandler registered on SIGTERM.
05/26 04:18:12.548 DEBUG|          base_job:0348| Persistent state global_properties.test_retry now set to 0
05/26 04:18:12.548 DEBUG|          base_job:0348| Persistent state global_properties.tag now set to '/usr/local/autotest/results/64677967-chromeos-test'
05/26 04:18:12.548 DEBUG|          base_job:0348| Persistent state global_properties.last_boot_tag now set to None
05/26 04:18:12.690 INFO |        server_job:0604| I am PID 1594
05/26 04:18:12.725 INFO |verify_job_repo_ur:0009| Verifying job repo url for machine {'host_attributes': {'job_repo_url': 'http://172.17.40.19:8082/static/falco-release/R53-8369.0.0/autotest/packages'}, 'hostname': 'chromeos2-row24-rack6-host9'}
05/26 04:18:12.746 DEBUG|          ssh_host:0180| Running (ssh) 'grep -q CHROMEOS /etc/lsb-release && ! test -f /mnt/stateful_partition/.android_tester && ! grep -q moblab /etc/lsb-release'
05/26 04:18:13.653 DEBUG|          ssh_host:0180| Running (ssh) 'test ! -e /var/log/messages || cp -f /var/log/messages /var/tmp/messages.autotest_start'
05/26 04:18:14.422 INFO |         cros_host:0417| Verifying job repo url http://172.17.40.19:8082/static/falco-release/R53-8369.0.0/autotest/packages
05/26 04:18:14.424 INFO |         cros_host:0424| Staging autotest artifacts for falco-release/R53-8369.0.0 on devserver http://172.17.40.19:8082
05/26 04:18:14.425 INFO |        dev_server:0931| Staging artifacts on devserver http://172.17.40.19:8082: build=falco-release/R53-8369.0.0, artifacts=['autotest_packages'], files=, archive_url=gs://chromeos-image-archive/falco-release/R53-8369.0.0
05/26 04:18:14.427 DEBUG|        base_utils:0176| Running 'ssh 172.17.40.19 'curl "http://172.17.40.19:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/falco-release/R53-8369.0.0"''
05/26 04:18:15.841 DEBUG|        base_utils:0176| Running 'ssh 172.17.40.19 'curl "http://172.17.40.19:8082/is_staged?artifacts=autotest_packages&files=&archive_url=gs://chromeos-image-archive/falco-release/R53-8369.0.0"''
05/26 04:18:17.241 INFO |        dev_server:0953| Finished staging artifacts: build=falco-release/R53-8369.0.0, artifacts=['autotest_packages'], files=, archive_url=gs://chromeos-image-archive/falco-release/R53-8369.0.0
05/26 04:18:17.245 INFO |        server_job:0682| Processing control file
05/26 04:18:17.249 DEBUG|          ssh_host:0180| Running (ssh) 'grep -q CHROMEOS /etc/lsb-release && ! test -f /mnt/stateful_partition/.android_tester && ! grep -q moblab /etc/lsb-release'
05/26 04:18:18.055 INFO |        server_job:0685| Finished processing control file
05/26 04:18:18.127 DEBUG|          ssh_host:0180| Running (ssh) 'grep -q CHROMEOS /etc/lsb-release && ! test -f /mnt/stateful_partition/.android_tester && ! grep -q moblab /etc/lsb-release'
05/26 04:18:18.907 DEBUG|          ssh_host:0180| Running (ssh) 'python -c 'import cPickle, glob, sys;cPickle.dump(glob.glob(sys.argv[1]), sys.stdout, 0)''
05/26 04:18:19.560 INFO | site_crashcollect:0191| There are no orphaned crashes; deleting /usr/local/autotest/results/64677967-chromeos-test/crashinfo.chromeos2-row24-rack6-host9
05/26 04:18:19.694 DEBUG|   logging_manager:0627| Logging subprocess finished
05/26 04:18:19.694 DEBUG|   logging_manager:0627| Logging subprocess finished


 

Comment 1 by tbroch@chromium.org, Jul 15 2016

Still seeing same failure mechanism,

http://cautotest/afe/#tab_id=view_job&object_id=69708218
http://chromeos-server3.hot.corp.google.com/results/69708218-chromeos-test/chromeos4-row11-rack7-host10/

status.log empty

If I goto the host,
http://cautotest/afe/#tab_id=view_host&object_id=4485

and look around the time of suite completion ...
07/15 13:00:26.631 DEBUG|   logging_manager:0627| Logging subprocess finished

There's no additional jobs on that dut host.  Looks like the following duts have image on it ( lars-release/R54-8588.0.0 ) so they could presumably land power_daily jobs although I'd suspect there should be some evidence in the suite job if that were the case:

chromeos4-row11-rack8-host2
chromeos1-row2-rack10-host6
chromeos4-row11-rack8-host5
chromeos4-row11-rack8-host4
chromeos4-row11-rack7-host20
chromeos4-row11-rack7-host21
chromeos4-row11-rack7-host13
chromeos4-row11-rack7-host19
chromeos4-row11-rack7-host12
chromeos4-row11-rack7-host9
chromeos4-row11-rack7-host11
chromeos1-row2-rack3-host2

Going to try just issuing the suite myself as digging around hosts is pretty time consuming.

Cc: -mshe...@chromium.org
Status: Archived (was: Untriaged)

Comment 4 by ketakid@google.com, Mar 18 2017

Labels: Pri-3
Status: Available (was: Archived)
Activating. Please assign to the right owner and the appropriate priority.

Comment 5 by tbroch@chromium.org, Mar 24 2017

Status: Fixed (was: Available)
got fixed somewhere along the way

Comment 6 by dchan@google.com, May 30 2017

Labels: VerifyIn-60

Comment 7 by dchan@chromium.org, Aug 1 2017

Labels: VerifyIn-61

Sign in to add a comment