+martinis: is this an expired task?

Yes, this is expired task: https://chromium-swarm.appspot.com/task?id=33a77726f535b810&refresh=10&show_raw=1

It targets build33-b1 bot specifically (id:build33-b1 dimension). Here's the history of tasks that target this specific bot: https://chromium-swarm.appspot.com/tasklist?c=name&c=state&c=created_ts&c=user&c=bot&f=id%3Abuild33-b1&l=500&s=created_ts%3Adesc

The list occasionally turns purple (streaks of expired tasks).  

It looks to me there's too much work to handle by this one bot. Occasionally the pending queue grows over 4 hours (the task expiration timeout) and oldest tasks expire.

(I don't know the specifics of how perf uses swarming and can't give any recommendations how to fix this... Usually we just add more bots, but you probably target single specific bot for a reason).

Stephen, can you take a look?

Looking. Probably some tests are taking longer than usual... this bot was green last week I think.

It looks like smoothness.tough_animation_cases almost doubled in running time, from about 34 minutes to over an hour.

Before:  https://chromium-swarm.appspot.com/task?id=339bed7227d60d10&refresh=10&show_raw=1
After: https://chromium-swarm.appspot.com/task?id=33abb3c1563d7010&refresh=10

Not sure why. But I think that could cause the timeouts we saw.

They're running the same amount of tests; 66.

I did some log grepping, and came up with the test runtimes for the two tasks above:

25682 -> 55421 css_transitions_staggered_chaining_by_updating_class.html?N=0316
28236 -> 41480 css_value_type_shadow.html?api=web_animations&N=0316
31491 -> 43966 balls_css_transition_40_properties.html
26468 -> 58083 css_value_type_length_complex.html?api=css_animations&N=0316
21180 -> 37614 transform_transitions.html
58418 -> 77330 css_animations_staggered_chaining_by_inserting_new_element.html?N=0316
26235 -> 96763 css_transitions_simultaneous_by_inserting_new_element.html?N=0316
25983 -> 34195 balls_javascript_canvas.html
22853 -> 48499 css_animations_staggered_infinite_iterations.html?N=0316
27553 -> 74490 css_animations_staggered_chaining_by_updating_inline_style.html?N=0316
24962 -> 30960 balls_svg_animations.html
26989 -> 41386 css_value_type_shadow.html?api=css_animations&N=0316
26243 -> 42061 css_animations_simultaneous_by_inserting_new_element.html?N=0316
39776 -> 69398 compositor_heavy_animation.html?N=0200
30878 -> 51559 css_transitions_staggered_triggering_by_inserting_new_element.html?N=0316
26897 -> 64552 css_value_type_length_simple.html?api=css_animations&N=0316
26005 -> 42933 css_transitions_staggered_chaining_by_inserting_style_element.html?N=0316
24041 -> 46206 web_animations_set_current_time_in_raf.html?N=0316
25569 -> 38281 css_transitions_simultaneous_by_updating_inline_style.html?N=0316
22244 -> 49445 keyframed_animations.html
29033 -> 52496 css_animations_staggered_chaining_by_inserting_style_element.html?N=0316
28700 -> 44937 css_value_type_path.html?api=css_animations&N=0316
28879 -> 38024 balls_css_keyframe_animations_composited_transform.html
26491 -> 57602 css_value_type_transform_simple.html?api=css_animations&N=0316
27237 -> 47348 web_animations_staggered_chaining.html?N=0316
25066 -> 32221 balls_javascript_css.html
24508 -> 46367 css_animations_simultaneous_by_updating_inline_style.html?N=0316
23905 -> 38242 css_animations_many_keyframes.html?N=0316
24772 -> 66685 css_transitions_simultaneous_by_updating_class.html?N=0316
26307 -> 43075 mix_blend_mode_animation_hue.html
24291 -> 42280 css_value_type_length_3d.html?api=web_animations&N=0316
24960 -> 41089 mix_blend_mode_animation_screen.html
99617 -> 155247 css_animations_staggered_triggering_by_inserting_new_element.html?N=0316
30411 -> 63237 css_transitions_staggered_triggering_by_inserting_style_element.html?N=0316
27605 -> 48476 css_animations_staggered_chaining_by_updating_class.html?N=0316
32385 -> 50882 css_animations_staggered_triggering_by_inserting_style_element.html?N=0316
31021 -> 88414 css_transitions_staggered_triggering_by_updating_class.html?N=0316
59132 -> 102310 overlay_background_color_css_transitions.html
26996 -> 46867 css_transitions_staggered_chaining_by_inserting_new_element.html?N=0316
26068 -> 55352 css_value_type_length_complex.html?api=web_animations&N=0316
24981 -> 41530 css_value_type_transform_complex.html?api=web_animations&N=0316
25797 -> 69548 css_transitions_staggered_chaining_by_updating_inline_style.html?N=0316
25934 -> 68488 mix_blend_mode_propagating_isolation.html
23233 -> 31683 css_value_type_color.html?api=css_animations&N=0316
33469 -> 82948 css_animations_staggered_triggering_by_updating_inline_style.html?N=0316
23484 -> 27823 transform_transition_js_block.html
30927 -> 55423 css_transitions_staggered_triggering_by_updating_inline_style.html?N=0316
25024 -> 42554 web_animations_many_keyframes.html?N=0316
32516 -> 54099 css_animations_staggered_triggering_by_updating_class.html?N=0316
22753 -> 34738 web_animations_staggered_infinite_iterations.html?N=0316
25207 -> 42766 web_animations_simultaneous.html?N=0316
24409 -> 38792 css_animations_simultaneous_by_updating_class.html?N=0316
25813 -> 41962 mix_blend_mode_animation_difference.html
34021 -> 43318 balls_css_transition_all_properties.html
27353 -> 76316 web_animations_staggered_triggering.html?N=0316
26408 -> 37958 balls_css_keyframe_animations.html
24469 -> 38499 css_animations_simultaneous_by_inserting_style_element.html?N=0316
25611 -> 40887 css_value_type_transform_complex.html?api=css_animations&N=0316
31490 -> 42026 balls_css_transition_2_properties.html
28595 -> 42990 css_value_type_filter.html?api=css_animations&N=0316
24652 -> 42771 css_value_type_length_3d.html?api=css_animations&N=0316
25317 -> 73144 css_transitions_simultaneous_by_inserting_style_element.html?N=0316
24288 -> 43622 css_value_type_transform_simple.html?api=web_animations&N=0316
22887 -> 35323 css_value_type_color.html?api=web_animations&N=0316
28971 -> 43785 css_value_type_path.html?api=web_animations&N=0316
26571 -> 63779 css_value_type_length_simple.html?api=web_animations&N=0316

all times are in milliseconds. It looks like everything got a lot slower.

Suspect builds with possibly bad regression ranges are https://luci-milo.appspot.com/buildbot/chromium.perf/Win%20Zenbook%20Perf/164 and https://luci-milo.appspot.com/buildbot/chromium.perf/Win%20Zenbook%20Perf/167

Looks like that test has only regressed on Zenbook, the other graphs look normal as far as I can tell.

Zenbook is slower than other bots, so it makes sense it regressed. I'm not exactly sure what regressed, but I'm suspecting the CLs in https://luci-milo.appspot.com/buildbot/chromium.perf/Win%20Zenbook%20Perf/169. https://chromium-swarm.appspot.com/task?id=33a2f70196c63b10&refresh=10&show_raw=1 is from build 168, and it runs in 45 minutes, and https://chromium-swarm.appspot.com/task?id=33a45f1902499310&refresh=10&show_raw=1 is from 169, and runs in over an hour. 

To be clear, the reason this could be causing issues is because the tasks take too long, which means the current timeouts on the tests aren't long enough for the bots to eventually get to them. Any time increase is significant as well, because the time delay is doubled, because of reference builds.

It looks like system_health.common_desktop is also timing out after 2 hours, which is also causing a lot of failures. Is there a bug on file for that?

Ok, looks like there are two issues. One is that the smoothness test listed here regressed in cycle time, which is timing out a couple tests that use the same bot. The other problem is that system_health.memory_desktop is timing out at two hours, which also causes tasks after it to time out.

oysteiene@, you should investigate the smoothness test regression somehow. Maybe start a bisect?

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8990631064903838832

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/652
Failure reason: the build has failed due to infrastructure failure.

crbug.com/674945 for output not explaining 24hr timeout

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/549d1501a66ed2ec0283149f65c71d64569183aa

commit 549d1501a66ed2ec0283149f65c71d64569183aa
Author: martiniss <martiniss@chromium.org>
Date: Mon Feb 13 23:56:46 2017

chromium.perf.json: Special case some desktop bots

This should help alleviate capacity issues we've been having for a bit.
Longer-term solutions should be coming for this, but these particular
bots are a bit overloaded right now.

PERF SHERIFFS: This CL will cause changes to the following benchmarks
* page_cycler_v2.intl_hi_ru on Mac 10.12 Perf
* blink_perf.parser on Mac 10.12 Perf

It also affects two other benchmarks, but those are already not sending
data, because of missing capacity, so those won't cause any jumps in
timings.

BUG= 687425 , 680360 , 685275 

Review-Url: https://codereview.chromium.org/2692133002
Cr-Commit-Position: refs/heads/master@{#450159}

[modify] https://crrev.com/549d1501a66ed2ec0283149f65c71d64569183aa/testing/buildbot/chromium.perf.json

This is passing now that we increased timeouts.