Somewhere around 20%-60% of "no-repro" bisects are because the perf dashboard alert was misplaced. Expanding the bisect range caused the alert to reproduce. Often the alert was placed at a reasonable location, but there were multiple possibilities and it just happened to be a wrong one. [1]

The limitation of the alerting algorithm is that it doesn't have a sense of uncertainty; alerts are always placed at a specific data point, rather than having a range of possibilities.

We should experiment with using e-divisive with medians, aka the Twitter BreakoutDetection algorithm. [2] This is a more powerful statistical algorithm which produces a probability distribution for the alert location.


[1] 4 examples:
https://chromeperf.appspot.com/group_report?bug_id=888878
https://chromeperf.appspot.com/report?sid=ebc59faccdf15a4f93d42f1c70af8d643484f41afefc4fe5e993de161f4e52be&rev=592893
https://chromeperf.appspot.com/group_report?bug_id=889112
https://chromeperf.appspot.com/report?sid=77b96a52249bc76cc1a6c2107ab0c7ea803832b8c578e7d4a6a5b9ccc38c2889&rev=621550

[2] https://github.com/twitter/BreakoutDetection
 

Somewhere around 20%-60% of "no-repro" bisects are because the perf dashboard alert was misplaced. Expanding the bisect range caused the alert to reproduce.

Often the alert was placed at a reasonable location, but there were multiple possibilities and it just happened to be a wrong one. [1] The limitation of the alerting algorithm is that it doesn't have a sense of uncertainty; alerts are always placed at a specific data point, rather than having a range of possibilities.

We should experiment with using e-divisive with medians, aka the Twitter BreakoutDetection algorithm. [2] This is a more powerful statistical algorithm which produces a probability distribution for the alert location.


[1] 4 examples:
https://chromeperf.appspot.com/group_report?bug_id=888878
https://chromeperf.appspot.com/report?sid=ebc59faccdf15a4f93d42f1c70af8d643484f41afefc4fe5e993de161f4e52be&rev=592893
https://chromeperf.appspot.com/group_report?bug_id=889112
https://chromeperf.appspot.com/report?sid=77b96a52249bc76cc1a6c2107ab0c7ea803832b8c578e7d4a6a5b9ccc38c2889&rev=621550

[2] https://github.com/twitter/BreakoutDetection