That seems like a good idea. Any idea how long these tests take to run?

cc_perftests: 11m 24s (Android)
gpu_perftest: 1m 16s (Android)
load_library_perf_tests: 1s (Windows)
angle_perftests: 27m 56s (Windows)
tracing_perftests: 37s (Android)

So the total is about 40 minutes machine time. If we shard on 10 bots, it would take 4 minutes.

That is more expensive than I would like. In particular, angle_perftests is quite a bit more expensive :(

Is it possible that any of the test suites could be faster, e.g., by running fewer iterations of a test or some such thing? 

Jamie, Adrienne: is it possible to set the number of iteration used in angle_perftests & cc_perftests through a commandline flag so that we can run fewer iteration on CQ?

I'm not sure what the right answer here is for cc_perftests.  cc_perftests don't fail, in that they don't locally have success or failure conditions.  They just run and spit out a number.  It is possible for them to time out if the bots are slow, but if that's the perf bot health you're trying to fix, then reducing the iterations on the CQ doesn't seem like it would address timing out on other bots.

I think the idea is that we'd rather catch a crash or a timeout on the main waterfall than on the perf waterfall.

To be clear: catch a timeout that the code introduced, rather than a bot-specific issue.

Ah, gotcha.  I don't know that we've ever had a timeout that the code introduced, ever.  Crashes on the other hand, probably a few.  So, I'd almost suggest some sort of smoke test command line flag that just ran like 10 iterations or something and stopped.  That'd probably be pretty fast.

There's no command line flag at the moment that does this (every test runs for 2 seconds, and that number is hardcoded per test), but it would be pretty straightforward to add one.

enne@: are you suggesting that for every slow test, we would do s.t like:

if (command_line->HasSwitch(switches::kRunSmokeTest)) {
    RunConstructTest("10", 10);
} else {
    RunConstructTest("100", 100);
    RunConstructTest("1000", 1000);
    RunConstructTest("10000", 10000);
    RunConstructTest("100000", 100000);
} 

(From https://cs.chromium.org/chromium/src/cc/base/rtree_perftest.cc?type=cs&q=rtree_perftest&l=89)

Yes, I think anything like that would be good.

Most of these tests use LapTimer.  I think you could make a PerfTestLapTimer that just quit after N runs and swap that in without changing too much.  That could be an alternative that doesn't change too much.

Thanks for the pointer, enne@! I will try to prototype something general here. Filed issue 725908 for tracking that work.

nednguyen, yes it should be pretty straight-forward for angle_perftests to have an execution mode that is correctness-only. We'd just have to pass a flag through the harness to the binary.

The following revision refers to this bug:
  https://chromium.googlesource.com/angle/angle/+/981f0f8f6a34b6144e573692c6fcab1625a5f67c

commit 981f0f8f6a34b6144e573692c6fcab1625a5f67c
Author: Jamie Madill <jmadill@chromium.org>
Date: Thu Sep 21 15:14:14 2017

Add flag to do a fast pass through perf tests.

This flag will only render the first frame of each perf test,
regardless of their preferences for how many seconds to run.
It will be useful for speeding up the run time of the perf tests
on testing infrastructure that only cares about correctness.

BUG=chromium:725308
BUG= chromium:765321 

Change-Id: I926f488c42f27ef23ef06a0159902613cff04080
Reviewed-on: https://chromium-review.googlesource.com/677306
Reviewed-by: Geoff Lang <geofflang@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>

[modify] https://crrev.com/981f0f8f6a34b6144e573692c6fcab1625a5f67c/src/tests/angle_perftests_main.cpp
[modify] https://crrev.com/981f0f8f6a34b6144e573692c6fcab1625a5f67c/src/tests/perf_tests/ANGLEPerfTest.cpp
[modify] https://crrev.com/981f0f8f6a34b6144e573692c6fcab1625a5f67c/src/tests/perf_tests/ANGLEPerfTest.h

The following revision refers to this bug:
  https://skia.googlesource.com/skia/+/f50f4a4c55ed4dea457d48010690bbe3f3c69530

commit f50f4a4c55ed4dea457d48010690bbe3f3c69530
Author: angle-deps-roller@chromium.org <angle-deps-roller@chromium.org>
Date: Thu Sep 21 17:07:20 2017

Roll skia/third_party/externals/angle2/ 1b7ed0ef2..981f0f8f6 (4 commits)

https://chromium.googlesource.com/angle/angle.git/+log/1b7ed0ef2d23..981f0f8f6a34

$ git log 1b7ed0ef2..981f0f8f6 --date=short --no-merges --format='%ad %ae %s'
2017-09-21 jmadill Add flag to do a fast pass through perf tests.
2017-09-20 jmadill D3D11: Minor optimizations to vertex attribute application.
2017-09-20 jmadill Minor cleanup to setIndexBuffer.
2017-09-20 jmadill Improve the speed of MarkAttachmentsDirty.

Created with:
  roll-dep skia/third_party/externals/angle2
BUG=725308


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=skia.primary:Perf-Win10-MSVC-AlphaR2-GPU-RadeonR9M470X-x86_64-Debug-ANGLE,Perf-Win10-MSVC-Golo-GPU-QuadroP400-x86_64-Debug-ANGLE,Perf-Win10-MSVC-NUC5i7RYH-GPU-IntelIris6100-x86_64-Debug-ANGLE,Perf-Win10-MSVC-NUC6i5SYK-GPU-IntelIris540-x86_64-Debug-ANGLE,Perf-Win10-MSVC-NUCD34010WYKH-GPU-IntelHD4400-x86_64-Debug-ANGLE,Perf-Win10-MSVC-ShuttleC-GPU-GTX960-x86_64-Debug-ANGLE,Test-Win10-MSVC-AlphaR2-GPU-RadeonR9M470X-x86_64-Debug-ANGLE,Test-Win10-MSVC-Golo-GPU-QuadroP400-x86_64-Debug-ANGLE,Test-Win10-MSVC-NUC5i7RYH-GPU-IntelIris6100-x86_64-Debug-ANGLE,Test-Win10-MSVC-NUC6i5SYK-GPU-IntelIris540-x86_64-Debug-ANGLE,Test-Win10-MSVC-NUCD34010WYKH-GPU-IntelHD4400-x86_64-Debug-ANGLE,Test-Win10-MSVC-ShuttleC-GPU-GTX960-x86_64-Debug-ANGLE
TBR=fmalita@google.com

Change-Id: I69a0ded10bc43bfd7aaaa01cd9b03cc12c31bfa7
Reviewed-on: https://skia-review.googlesource.com/49820
Reviewed-by: angle-deps-roller . <angle-deps-roller@chromium.org>
Commit-Queue: angle-deps-roller . <angle-deps-roller@chromium.org>

[modify] https://crrev.com/f50f4a4c55ed4dea457d48010690bbe3f3c69530/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cb2ec8306dd6854d928eb3bf88840af0a5fdb983

commit cb2ec8306dd6854d928eb3bf88840af0a5fdb983
Author: Frank Henigman <fjhenigman@chromium.org>
Date: Thu Sep 21 19:19:25 2017

Roll ANGLE 47bf2dc..1f9d684

https://chromium.googlesource.com/angle/angle.git/+log/47bf2dc..1f9d684

BUG= chromium:767279 ,chromium:725308, chromium:765321 , chromium:655534 

TEST=bots

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel

Change-Id: I794ad67e2a1ef672855c65eef94959800cba9477
Reviewed-on: https://chromium-review.googlesource.com/677428
Reviewed-by: Geoff Lang <geofflang@chromium.org>
Commit-Queue: Frank Henigman <fjhenigman@chromium.org>
Cr-Commit-Position: refs/heads/master@{#503529}
[modify] https://crrev.com/cb2ec8306dd6854d928eb3bf88840af0a5fdb983/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/283d67f6419adc69678e707766c6c45ec6b4c84f

commit 283d67f6419adc69678e707766c6c45ec6b4c84f
Author: skia-deps-roller@chromium.org <skia-deps-roller@chromium.org>
Date: Thu Sep 21 20:43:22 2017

Roll src/third_party/skia/ 5fdbb8a8a..2fbf1bc8c (9 commits)

https://skia.googlesource.com/skia.git/+log/5fdbb8a8a76e..2fbf1bc8c96f

$ git log 5fdbb8a8a..2fbf1bc8c --date=short --no-merges --format='%ad %ae %s'
2017-09-20 djsollen Add SK_API to APIs used by the android framework.
2017-09-21 bsalomon Fix wrap mode asserts in GrVkSampler::GenerateKey
2017-09-21 liyuqian Run the first bench for 1000ms to warm up the nanobench if FLAGS_ms < 1000.
2017-09-20 mtklein Teach TSAN directly about semaphore_t.
2017-09-21 angle-deps-roller Roll skia/third_party/externals/angle2/ 1b7ed0ef2..981f0f8f6 (4 commits)
2017-09-21 caryclark add missing param names to SkBitmap.h
2017-09-21 caryclark feedback
2017-09-21 caryclark regenerate includes
2017-09-21 robertphillips Remove pre-attachment of stencil buffers

Created with:
  roll-dep src/third_party/skia
BUG=725308


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_trusty_blink_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=fmalita@chromium.org

Change-Id: Ib8a043fe4d18794fc1b2b89dcba734df555aeaa6
Reviewed-on: https://chromium-review.googlesource.com/677494
Reviewed-by: Skia Deps Roller <skia-deps-roller@chromium.org>
Commit-Queue: Skia Deps Roller <skia-deps-roller@chromium.org>
Cr-Commit-Position: refs/heads/master@{#503552}
[modify] https://crrev.com/283d67f6419adc69678e707766c6c45ec6b4c84f/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9cddc994b8e58a467d36e2fe7233afc351f3ee01

commit 9cddc994b8e58a467d36e2fe7233afc351f3ee01
Author: Jamie Madill <jmadill@chromium.org>
Date: Tue Sep 26 03:19:21 2017

Add angle_perftests fast run to bots.

The command line flag "--one-frame-only" will force the ANGLE perf
tests to stop execution after one iteration. This will help test
correctness while giving a much faster run time. Currently they take
about ten minutes on the bots, this should hopefully give a 10x
improvement for correctness-only testing.

Requires http://crrev.com/c/677306 to land first.

BUG=725308, 765321 

Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I9bbbe3a2c3100fd4d43759382cd62ec2f45ab33b
Reviewed-on: https://chromium-review.googlesource.com/677307
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Cr-Commit-Position: refs/heads/master@{#504271}
[modify] https://crrev.com/9cddc994b8e58a467d36e2fe7233afc351f3ee01/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/9cddc994b8e58a467d36e2fe7233afc351f3ee01/gpu/angle_perftests_main.cc
[modify] https://crrev.com/9cddc994b8e58a467d36e2fe7233afc351f3ee01/testing/buildbot/chromium.gpu.fyi.json

nednguyen@google.com, angle_perftests is now running on the ANGLE CQ and takes about 15 seconds of shard time. I think the work there is done. I also don't think it's necessary to run on the Chromium CQ unless there's no coverage of a non-telemetry perf test suite at all (I think the others are media and cc perf tests).

+miu: if you work on adding performance_browser_test to CQ, please also reference this bug