If the benchmark fails completely and consistently, we should either replace the reference build, or just stop running the reference build at all on that bot.

On system health I'm only aware of two bots where this is an issue: go/catabug/3399

We decided not to do anything about this. The main reason is that we don't really care about the cycle time of reference build as it is not a lot more than TOT build. 

The strategy proposed in #1 can save some times, but won't be as much when we shard benchmark by stories, so not much win with more code complexity.

Long term: we will carefully track the cycle time of all benchmarks with perf dashboard & will look carefully when a ref benchmarks' cycle time blow up.

This is actually on the table again since with Stephen's sharding algorithm, most benchmark will not be sharded (only ones that are cut in the middle).

Also we found out in code review (https://codereview.chromium.org/2998773002/) today that we can make this happens using 1 line CL to add "--max-failures=5" to all the ref build benchmark.

This makes so much sense to me.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/10450252daa708cd59929774aba0c27fff671fc0

commit 10450252daa708cd59929774aba0c27fff671fc0
Author: Ned Nguyen <nednguyen@google.com>
Date: Mon Aug 14 20:37:26 2017

Set maximum number of failures for reference build benchmarks to be 5

Bug:  707236 
Change-Id: Id129869ac38eed55eb0ce0f5b9daf34e04f5210c
Reviewed-on: https://chromium-review.googlesource.com/611314
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#494170}
[modify] https://crrev.com/10450252daa708cd59929774aba0c27fff671fc0/testing/buildbot/chromium.perf.fyi.json
[modify] https://crrev.com/10450252daa708cd59929774aba0c27fff671fc0/testing/buildbot/chromium.perf.json
[modify] https://crrev.com/10450252daa708cd59929774aba0c27fff671fc0/tools/perf/core/perf_data_generator.py
[modify] https://crrev.com/10450252daa708cd59929774aba0c27fff671fc0/tools/perf/core/perf_data_generator_unittest.py

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

With #9, I think this is fixed