Frequent suite job timeouts during build |
||||||||||||||||||||
Issue descriptionE.g. https://uberchromegw.corp.google.com/i/chromeos/builders/jecht-release-group/builds/1356 There have been a lot of similar bvt_inline timeouts recently. After the last test, the suite times out - even though the test seems to have passed. Example on guado: https://uberchromegw.corp.google.com/i/chromeos/builders/jecht-release-group/builds/1356/steps/HWTest%20%5Bguado%5D%20%5Bbvt-inline%5D/logs/stdio login_GuestAndActualSession: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=58960906 The suite job has another 0:59:14.691822 till timeout. The suite job has another 0:29:04.965304 till timeout. The suite job has another -1 day, 23:58:58.914383 till timeout. Suite timed out. Started on 04-04-2016 [07:37:09], timed out on 04-04-2016 [09:41:04] Suite job [ FAILED ] Suite job ABORT: No reference platform in particular. For example, on M51-8150.0.0 alone, this same failure showed up on 23 boards: gandof, panther, monroe, stout, guado, rikku, sentry, big, pi, mario, mickey, banjo, sumo, glimmer, quawks, heli, squawks, winky, parrot, falco_li, celes, ultima, and cyan. Reks and terra had the same failure with a different ending test. Also happens for paygen tests. Example on a M50 buddy build. Tests pass but suite is aborted: https://uberchromegw.corp.google.com/i/chromeos_release/builders/auron-b-release-group%20release-R50-7978.B/builds/38
,
Apr 4 2016
Issue 600506 has been merged into this issue.
,
Apr 5 2016
CQ also has some suite timeout issue: https://bugs.chromium.org/p/chromium/issues/detail?id=600022 Not sure whether it's related.
,
Apr 5 2016
The canaries are still running into this issue. There do seem to be underlying issues that others are looking into (provision failures, SSH reporting a generic error.) But, that should be different from the suite timing out, right?
,
Apr 5 2016
canaries has larger probabilities to be suite timeout, since there exist no enough shards for it, which leads to slow processing. Infra team is adding more shards.
,
Apr 5 2016
Re#5, is there a bug tracking that effort?
,
Apr 5 2016
no actually, it's in our OKR in this quarter.
,
Apr 11 2016
related to canary timeouts?
,
Apr 11 2016
Re #8, yes, the bvt-inline suites are related to canary hwtest.
,
Apr 26 2016
,
Apr 27 2016
,
May 19 2016
This is still happening: bvt-inline failures cause build timeout.
,
May 19 2016
Could problems today be related to the number of release builds running as tryjobs?
,
Aug 4 2016
Dan, is this still an issue? If so, please assign and prioritize.
,
Aug 4 2016
haven't seen for a while. Since the bug filed we have added many shards, now total at 43. The performance should be much better now. Close the bug for now.
,
Aug 29 2016
,
Aug 30 2016
Seeing this issue again in last two builds of veyron_mighty-paladin HWTest [bvt-inline]: https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_mighty-paladin/builds/2988/steps/HWTest%20%5Bbvt-inline%5D/logs/stdio https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_mighty-paladin/builds/2989/steps/HWTest%20%5Bbvt-inline%5D/logs/stdio The suite job has another 0:29:44.595083 till timeout. The suite job has another -1 day, 23:59:42.196116 till timeout. Suite timed out. Started on 08-29-2016 [21:46:44], timed out on 08-29-2016 [23:26:04] Suite job [ FAILED ] Suite job ABORT:
,
Aug 30 2016
Please open new bugs for this failure. From the job: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=75012916 FAIL security_NetworkListeners security_NetworkListeners timestamp=1472521827 localtime=Aug 29 18:50:27 Android did not boot! The test failed already, the timeout is caused by crash dump collection/symbolication.
,
Oct 7 2016
,
Oct 10 2016
,
Nov 19 2016
,
Jan 21 2017
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Oct 14 2017
|
||||||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||||||
Comment 1 by bhthompson@chromium.org
, Apr 4 2016Labels: OS-Chrome