orco-release builder failing paygen and hwtest due to suite timeout |
||||||||||
Issue descriptionIt looks like the tests aren't even running, they get scheduled but nothing else happens after that. The orco duts are all idle so they haven't been running anything either. https://uberchromegw.corp.google.com/i/chromeos/builders/orco-release/builds/735 There are a ton of queued jobs right now and nothing is running on the shard (chromeos-server46.hot). There are 3 aborted jobs that show up in the running status: http://chromeos-server46.hot.corp.google.com/afe/#tab_id=job_list&state_filter=running&type_filter=all
,
Jan 6 2017
Nothing fishy in the PaygenTest stages. The tests were indeed scheduled. Autotest claims that they were queued, but then aborted. canary channel suite job: http://shortn/_v7nI1039vG dev channel suite job: http://shortn/_LmRMplaZGc
,
Jan 6 2017
It seems like the 3 jobs are just stuck in provisioning. Is there anyway to just clobber those jobs? One of the provisioning jobs just seems stuck... http://shortn/_tDlHKPuGOV
,
Jan 7 2017
Mmm. Do you see any processes getting stuck at or near exit? We're looking into a strange unittest failure in issue 678643 . It seems to coincide with a glibc change in the chroot. If you can catch a stuck process in the act, sudo cat /proc/<pid>/stack and see if it's also stuck in a futex call.
,
Jan 9 2017
assigning to current deputy to resolve, builder is still failing.
,
Jan 9 2017
The glibc change mentioned in #4 appears to be a red herring.
,
Jan 10 2017
upping priority and adding Charlene in case it's an issue with the shard. Charlene, jobs seemed to be stalled on this shard, do you know what might be going on?
,
Jan 10 2017
It looks like every Orco DUT in the BVT pool is marked as dead. Investigating further.
,
Jan 10 2017
That shard was rebooted on Jan 4th, but appears to be up and running to me. I'm not sure if it's healthy, but it's up and running shard like processes.
,
Jan 10 2017
I just forced another restart, which seems to have aborted the stuck jobs. I'm now forcing a reverify for all BVT pool DUTs.
,
Jan 10 2017
They seem to have all passed!
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Oct 14 2017
|
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by kevcheng@chromium.org
, Jan 6 2017