Increasing the timeout seems reasonable. I think they're deprecating this config. So probably adding more capacity isn't a good idea. I proposed removing the expensive test suite from this config. Caleb wasn't a fan of that idea.

We can't add more capacity because we can't buy any more of these devices because they aren't on the market any more.

We can't turn off the expensive test suite (performance_test_suite) since a ton of teams depend on it. It's OBBS (One Build Bot Step) test suite that runs *all* the telemetry benchmarks, including V8 team's and GPU team's, etc, etc. A lot more people would be upset about that then if I just disabled all the gtest_perf_tests.

Currently, the expensive step (performance_test_suite) takes 8 hours. Increasing the allocation timeout for the gtest_perf_tests to high enough that we won't hit this again (maybe 16 hours to be safe?) seems like it might be a problem. I wonder what the maximum timeout is.

We could also have the performance_test_suite always use one fewer devices than are available such that the gtest_perf_tests can use the other one. The negative of this is that the gtest_perf_test device would be idle for much of its life and the performance_test_suite would take much longer because it would have one fewer device to shard against.

Essentially we cannot safely make this change until we integrate the gtest_perf_tests into the soft affinity sharding system (OBBS) of the Telemetry tests.

Dirk was one of the main proponents for the OBBS system, and he also reviewed your CL. I wonder if he has advice for what we should do here.

issue 865538 is the bug to replace this configuration with a new one that will have many more bots attached to it so that the runtimes will be much shorter.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/b9208888e944e48a138f342f6269ad27b0fea2dc

commit b9208888e944e48a138f342f6269ad27b0fea2dc
Author: Takuto Ikuta <tikuta@chromium.org>
Date: Wed Jan 16 22:38:01 2019

[chromium_tests, test_utils] change order of test_pre_run

This is re-land of
https://chromium-review.googlesource.com/c/chromium/tools/build/+/1390088
But only enables re-ordering in CQ for  crbug.com/921039  .

When the CL needs to run many tests, test_pre_run steps takes several minutes.
Currently, test_pre_run does not consider the duration of test, but it is better to trigger slow test earlier.
This CL trigger tests having large number of shards earlier so that we can remove several minutes from waiting time of swarming test completion.

e.g.
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win7_chromium_rel_ng/158265
If webkit_layout_tests was triggered first, the tests could start 4 minutes faster in swarming.
That could remove 4 minutes from the builds.

This is similar to
https://chromium-review.googlesource.com/c/chromium/tools/build/+/1096723/
but focus on triggering step.

Bug: 921906
Change-Id: I19c871e3e25244cfdeaedf6bc13084420d0f183f
Reviewed-on: https://chromium-review.googlesource.com/c/1411093
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>

[modify] https://crrev.com/b9208888e944e48a138f342f6269ad27b0fea2dc/scripts/slave/README.recipes.md
[modify] https://crrev.com/b9208888e944e48a138f342f6269ad27b0fea2dc/scripts/slave/recipe_modules/test_utils/api.py
[modify] https://crrev.com/b9208888e944e48a138f342f6269ad27b0fea2dc/scripts/slave/recipe_modules/chromium_tests/steps.py
[modify] https://crrev.com/b9208888e944e48a138f342f6269ad27b0fea2dc/scripts/slave/recipe_modules/chromium_tests/api.py
[modify] https://crrev.com/b9208888e944e48a138f342f6269ad27b0fea2dc/scripts/slave/recipes/chromium_trybot.py
[modify] https://crrev.com/b9208888e944e48a138f342f6269ad27b0fea2dc/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.py

#3, OK.
I'll re-visit this bug to remove conditional variable from recipe after the issue is fixed.