peach_pit, wizpig, wolf devices not running jobs |
||||
Issue descriptionSimilar to grunt issue recently resolved. http://crbug.com/891758 Build: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8933051048707852288 From the logs (https://luci-logdog.appspot.com/logs/chromeos/bb/chromeos/wolf-paladin/18919/+/recipes/steps/HWTest__provision_/0/stdout): Autotest instance created: cautotest-prod 10-10-2018 [07:18:21] Created suite job: http://cautotest-prod/afe/#tab_id=view_job&object_id=247168034 @@@STEP_LINK@Link to suite@http://cautotest-prod/afe/#tab_id=view_job&object_id=247168034@@@ The suite job has another 0:59:49.682533 till timeout. 10-10-2018 [08:03:30] printing summary of incomplete jobs (9): dummy_Pass: http://cautotest-prod/afe/#tab_id=view_job&object_id=247169757 dummy_Pass: http://cautotest-prod/afe/#tab_id=view_job&object_id=247169762 dummy_Pass: http://cautotest-prod/afe/#tab_id=view_job&object_id=247169766 dummy_Pass: http://cautotest-prod/afe/#tab_id=view_job&object_id=247169772 dummy_Pass: http://cautotest-prod/afe/#tab_id=view_job&object_id=247169778 dummy_Pass: http://cautotest-prod/afe/#tab_id=view_job&object_id=247169786 dummy_Pass: http://cautotest-prod/afe/#tab_id=view_job&object_id=247169793 dummy_Pass: http://cautotest-prod/afe/#tab_id=view_job&object_id=247169799 dummy_Pass: http://cautotest-prod/afe/#tab_id=view_job&object_id=247169807 The suite job has another 0:29:43.384902 till timeout. The suite job has another -1 day, 23:59:33.512447 till timeout. 10-10-2018 [08:49:28] Suite job is finished. Suite timed out. Started on 10-10-2018 [07:18:21], timed out on 10-10-2018 [08:49:28] 10-10-2018 [08:49:28] Start collecting test results and dump them to json. Suite job [ FAILED ] Suite job ABORT:
,
Oct 10
,
Oct 10
,
Oct 10
,
Oct 10
,
Oct 10
,
Oct 10
peach_pit and wizpig are different shards, but also possibly same issue. handing off to Alex to investigate. A good starting point is https://viceroy.corp.google.com/chromeos/dut_utilization for those boards.
,
Oct 10
Attempting Aviv's fix of issue 891758 : remove and re-add each board.
,
Oct 10
wolf: cros-full-0008 wizpig: cros-full-0019 peach_pit: cros-full-0011
,
Oct 10
Removed wolf. Waiting for all jobs to be deleted.
,
Oct 10
Re-added wolf.
,
Oct 11
wolf looked better for a couple of hours and now it is back to the original state with a lot of idle duts: http://shortn/_8QMiAwYslN The underlying issue is probably different from the one in issue 891758 where there was an obvious change in dut metrics after the re-addition: http://shortn/_tJcGYggBkP Will revisit this tomorrow.
,
Oct 11
wolf looks a lot better starting this morning: http://shortn/_XnN6S7sQYb The re-adding seems to have fixed whatever the underlying issue was.
,
Oct 11
Will attempt removing and re-adding wizpig. Will leave peach_pit alone because it looks like it recovered.
,
Oct 12
wizpig-paladin looks healthy. |
||||
►
Sign in to add a comment |
||||
Comment 1 by saklein@chromium.org
, Oct 10