New issue
Advanced search Search tips

Issue 893211 link

Starred by 1 user

Issue metadata

Status: Untriaged
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Mac perf builders occasionally timed out

Project Member Reported by nedngu...@google.com, Oct 8

Issue description

This also happen to win-10-perf config today

https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win-10-perf/862
The builds are hitting their 7 hour timeout. And they're timing out because they're waiting for a performance_test_suite shard that doesn't finish until it hits its 7 hour timeout. eg:

https://chrome-swarming.appspot.com/tasklist?c=name&c=state&c=created_ts&c=duration&c=pending_time&c=pool&c=bot&et=1539193680000&f=buildername-tag%3Amac-10_13_laptop_high_end-perf&f=name-tag%3Aperformance_test_suite&f=buildnumber-tag%3A1512&l=50&n=true&s=created_ts%3Adesc&st=1538416080000
and
https://chrome-swarming.appspot.com/tasklist?c=name&c=state&c=created_ts&c=duration&c=pending_time&c=pool&c=bot&et=1539193680000&f=buildername-tag%3Amac-10_13_laptop_high_end-perf&f=name-tag%3Aperformance_test_suite&f=buildnumber-tag%3A1518&l=50&n=true&s=created_ts%3Adesc&st=1538416080000

So:
- Ensure builder timeout is sufficiently larger than individual shard timeout. (Definitely shouldn't be the same.)
and/or
- Find out why a couple performance_test_suite shards are hitting their timeout.
THanks for the analysis, Ben! I think we can totally adjust the timeout to 3hrs max for these builders which have lots of sharding machines already

Sign in to add a comment