Sorry I wasn't more involved in this earlier, but could we come up with a non-arbitrary rule about how much time budget we can give per user story? For example, if we stick with 1 hour (I like this as it's consistent with the CQ), the other two variables are:
- # of devices
- time it takes a device to run a test (overhead)
- test timeout
- # of tests

When we control the # of devices (which we currently do), we become prescriptive about the number of tests that can possibly run to get us under and hour. If we filtered all of our tests with this in mind: first pass == no tests that take longer than x minutes to run (enforcement of timeout). second pass == no tests that are duplicated by other tests (mutual exclusion). third pass == prioritized tests and draw a line.

Once we have a line, we can then see what more hardware will buy in terms of tests AND buffer for future tests.

Ben: that's a fine idea & have been used in many test framework. I think we should consider, though the challenge is integragration perf test' cycle time can be vastly diverse.
Taking loading for example: loading a site can take any where from 0.5s to 10s of seconds. And folks do want to test all spectrum for coverage completeness.

That's a good point. But even with these examples, we could potentially limit what tests we're willing to cover in the lab. There's a trade-off between coverage and...well...this cost/time-to-bisect/test-run-time thing we keep bringing up.

 Issue 691582  has been merged into this issue.

Moving over to ned.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b7e98079d9dc9a2230d62c587144097171bff499

commit b7e98079d9dc9a2230d62c587144097171bff499
Author: Ned Nguyen <nednguyen@google.com>
Date: Fri Jul 06 02:05:47 2018

Reduce the swarming timeout of perf tests

NOTRY=true
TBR=eyaich@chromium.org

Bug: 713345
Cq-Include-Trybots: master.tryserver.chromium.perf:obbs_fyi
Change-Id: Ibc1954a26040e8899f9ff13fee0756ca0a9b1481
Reviewed-on: https://chromium-review.googlesource.com/1127480
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#572882}
[modify] https://crrev.com/b7e98079d9dc9a2230d62c587144097171bff499/testing/buildbot/chromium.perf.fyi.json
[modify] https://crrev.com/b7e98079d9dc9a2230d62c587144097171bff499/testing/buildbot/chromium.perf.json
[modify] https://crrev.com/b7e98079d9dc9a2230d62c587144097171bff499/tools/perf/core/perf_data_generator.py

So I am not sure we are actively working on this goal anymore.  We were able to reduce to ~2-3 hours and our current goal is to at least maintain this.

Adding Caleb as an FYI of the goal and closing.

Since we aren't quite at 1 hour yet, I will take this bug and see if there is anything we can do with it. This bug is hopefully because we can link to it as a reason to increase hardware and keep test runs short.