Just thought I'd mention this behaviour somewhere.

In  http://crbug.com/607657  there's a failure in page_cycler.basic_oopif, test is crashing on Android One. Looking at Android One's history, https://build.chromium.org/p/chromium.perf/builders/Android%20One%20Perf%20%281%29?numbuilds=200 this went from always passing to always failing (logs seem to be gone and I can't find the cache link now so I can't look at whether this was retried there). The bisect can't proceed largely because the test was secretly flaky before it started crashing.


Looking at another bot, on Android Nexus 7v2, this is green on every run, although if you look at the logs

https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus7v2%20Perf%20%281%29/builds/2308
https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus7v2%20Perf%20%281%29/builds/2308/steps/page_cycler.basic_oopif/logs/stdio

It's actually failing occasionally with the same error. Should we be retrying like this, or just failing on the first run to make it clear the test is flaky?