The failures are about timeout issue:

"shard #0 timed out, took too much time to complete."

From the swarming log, I do not see any obvious failure.

Ned, can you take a look at why it starts to timeout?

Stephen can you take a look?

So, trivially, it looks like these tests are taking longer than 2 hours to run. There's a timeout on each task in swarming, which is 2 hours I think. They didn't used to take that long, IIRC.

Does anyone know of any changes that could have increased the cycle time?

It could be related to the change Hector did recently to make memory benchmarks more reproducible?

Comparing between before & now, there is clearly a timing explosion here:

Before:
https://chromium-swarm.appspot.com/task?id=33833729e3513f10&refresh=10&show_raw=1
[ OK ] load:media:dailymotion (14963 ms)

Now:
https://chromium-swarm.appspot.com/task?id=33f013a32a4e1510&refresh=10&show_raw=1
[ OK ] load:media:dailymotion (81316 ms)

I am really suspecting Hector's change (https://codereview.chromium.org/2631283002) is creating this time explosion because system_health.common_dekstop doesn't have the same timing explosion (taking 34m in https://build.chromium.org/p/chromium.perf/builders/Win%2010%20Perf/builds/334).

The only differences between the two suite are:
1) system_health.memory_desktop use memory trace whereas system_health.common_desktop uses other trace
2) system_health.memory_desktop clear system cache & browser profile for every single user story whereas system_health.common_desktop ignores it.

It's extremely unlikely that that 50+ seconds regression in #7 is due to memory tracing regression, so I think (2) is likely the root cause.

Assign to Hector for further investigation.

Running try jobs on win-all, mac-all with the change reverted from memory_desktop
to compare. See https://codereview.chromium.org/2658583005.

If I'm looking at the right step "Running WITH patch.Performance Test (With/Without Patch)" https://luci-milo.appspot.com/buildbot/tryserver.chromium.perf/mac_air_perf_bisect/6
then is does seem to be slower.

Shall we try committing https://codereview.chromium.org/2631283002 and see if they recover?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ce9c223faabd70c29759bc6f5d8844f44b40cede

commit ce9c223faabd70c29759bc6f5d8844f44b40cede
Author: hjd <hjd@chromium.org>
Date: Thu Jan 26 19:38:00 2017

Only flush system cache on mobile. Flushing the cache on
desktop increases runtime significantly.

BUG= 685240 

Review-Url: https://codereview.chromium.org/2658583005
Cr-Commit-Position: refs/heads/master@{#446396}

[modify] https://crrev.com/ce9c223faabd70c29759bc6f5d8844f44b40cede/tools/perf/benchmarks/system_health.py