This just happened recently, not sure what's going on: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/mac-10_12_laptop_low_end-perf/1723 https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/mac-10_13_laptop_high_end-perf/1512 https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/mac-10_13_laptop_high_end-perf/1504
This also happen to win-10-perf config today https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win-10-perf/862
The builds are hitting their 7 hour timeout. And they're timing out because they're waiting for a performance_test_suite shard that doesn't finish until it hits its 7 hour timeout. eg: https://chrome-swarming.appspot.com/tasklist?c=name&c=state&c=created_ts&c=duration&c=pending_time&c=pool&c=bot&et=1539193680000&f=buildername-tag%3Amac-10_13_laptop_high_end-perf&f=name-tag%3Aperformance_test_suite&f=buildnumber-tag%3A1512&l=50&n=true&s=created_ts%3Adesc&st=1538416080000 and https://chrome-swarming.appspot.com/tasklist?c=name&c=state&c=created_ts&c=duration&c=pending_time&c=pool&c=bot&et=1539193680000&f=buildername-tag%3Amac-10_13_laptop_high_end-perf&f=name-tag%3Aperformance_test_suite&f=buildnumber-tag%3A1518&l=50&n=true&s=created_ts%3Adesc&st=1538416080000 So: - Ensure builder timeout is sufficiently larger than individual shard timeout. (Definitely shouldn't be the same.) and/or - Find out why a couple performance_test_suite shards are hitting their timeout.
THanks for the analysis, Ben! I think we can totally adjust the timeout to 3hrs max for these builders which have lots of sharding machines already
Comment 1 by nedngu...@google.com
, Oct 9