stgao: this is an interesting case I wonder if it would be easier to solve with FindIt. There is a test failing, but it's so flaky that bisect got these results when we tried to bisect the failure:

Revision             Exit Code      N
chromium@488897      0 +- N/A       3      good
chromium@488908      0 +- N/A       3      good
chromium@488913      0 +- N/A       3      good
chromium@488914      1 +- N/A       3      bad       <--
chromium@488915      1 +- N/A       3      bad
chromium@488916      1 +- N/A       3      bad
chromium@488918      1 +- N/A       3      bad
chromium@488938      1 +- N/A       3      bad
chromium@488978      1 +- N/A       3      bad

Could we have gotten a better idea from FindIt which CL caused the test to go from flaky to hard failing?

Findit currently only supports analyzing the transition from stable (pass rate is <2% or >98%) to flaky (pass rate is between 2% to 98%).

So Findit doesn't support analyzing the transition from flaky to hard failing yet, although we also observed such cases with C++ gtests.
We could potentially support that with a change to the algorithm to identify the regression range, but since such cases are rare, it is not a priority task at the moment.

I think we could revisit this once Findit supports analyzing flaky Telemetry benchmark tests.
WDYT?

That makes sense to me.

We can says hard failing is 100%? :-) 

Seem like the general logic here could be is FindIt can detect transition of flaky rate from X to Y.

how do you define flaky rate?

For pass rate, our definition is: if M out of N executions of the test failed, the pass rate is M/N.

For hard failing, my understanding is pass-rate = 0%.

Err, I mean "pass rate" in #4 (and it's 0% as you said), sorry for confusing terminology.