+dtu

There is a small but clear regression. KS or Anderson would've detected the regression after 20-25 runs, but this bisect appeared to stop after 5.

I think there are tradeoffs with every approach. Taking the average would make it harder to detect some kinds of regressions that don't affect the mean much. Also having different algorithms for dashboard and bisect means that they're going to be inconsistent in the types of regressions each can detect.

We have different algorithms for dashboard and bisect already. I agree we should work towards moving them closer, but in the short term, I think we still want to dig more deeply into which algorithms work best for bisect?

The most pragmatic solution I can think of is to add a flag to 'Bisect on the mean of each run' and preset it for specific metrics, allowing the sheriff to override it.

This could be piped through to compare_samples which would in turn average each list of scalars in the sample instead of concatenating all.

The same approach could be used to bisect on stddev of each run.

https://bugs.chromium.org/p/chromium/issues/detail?id=664967#c14

Note for later: A page was disabled, hence 2x as many samples for good as bad.

Simon,

How do you get the raw data values from a bisect after it is done? I'm looking at the std deviation numbers for these bisects, and they seem impossibly large. I'm wondering if there's a bug in their calculation.

re: #c25

Pick any "Bisecting Revision" step, scroll and find the "Compare Samples", there should be 2 in each "Bisecting Revision" and 1 after the initial "Gathering Reference Values"

If you look in there, you'll find all the raw data.

So I did some quickie tests here, added an mean to list_of_scalars akin to what the dashboard does, the results seemed a lot better. Went through the last few bugs blocked against this one, pulled out all the json files and re-ran compare_samples manually.


679559

Original result:
  "result": {
    "U": 2572678.5, 
    "p": 0.08912829748779405, 
    "significance": "FAIL_TO_REJECT"
  }, 

New Result:
tracing/bin/compare_samples "/usr/local/google/home/simonhatch/tmp/bisect_json/good1.json,/usr/local/google/home/simonhatch/tmp/bisect_json/good2.json,/usr/local/google/home/simonhatch/tmp/bisect_json/good3.json,/usr/local/google/home/simonhatch/tmp/bisect_json/good4.json,/usr/local/google/home/simonhatch/tmp/bisect_json/good5.json,/usr/local/google/home/simonhatch/tmp/bisect_json/good6.json" "/usr/local/google/home/simonhatch/tmp/bisect_json/bad1.json,/usr/local/google/home/simonhatch/tmp/bisect_json/bad2.json,/usr/local/google/home/simonhatch/tmp/bisect_json/bad3.json,/usr/local/google/home/simonhatch/tmp/bisect_json/bad4.json,/usr/local/google/home/simonhatch/tmp/bisect_json/bad5.json,/usr/local/google/home/simonhatch/tmp/bisect_json/bad6.json" frame_times/http___techcrunch.com --chartjson
{"sampleA":[16.427904639201067,16.458113989681777,16.447558139568766,16.425819587615347,16.496716494796818,16.48418041241845],"sampleB":[16.778348285006974,16.710595800494897,16.734767810021353,16.785488126696258,16.71902368421617,16.713241469828787],"result":{"U":0,"p":0.005074868097940222,"significance":"REJECT"}}




679503

Original result:
  "result": {
    "U": 352, 
    "p": 0.14944870623046325, 
    "significance": "FAIL_TO_REJECT"
  }, 

New Result:
tracing/bin/compare_samples "/usr/local/google/home/simonhatch/tmp/bisect_json/679503/g1.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679503/g2.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679503/g3.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679503/g4.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679503/g5.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679503/g6.json" "/usr/local/google/home/simonhatch/tmp/bisect_json/679503/b1.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679503/b2.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679503/b3.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679503/b4.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679503/b5.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679503/b6.json" memory:chrome:all_processes:reported_by_chrome:v8:effective_size_max/memory:chrome:all_processes:reported_by_chrome:v8:effective_size_max --chartjson
{"sampleA":[65618580.8,65973312,62538356.8,62881980.8,62823992,65665654.4],"sampleB":[78500612.8,86430537.6,86990617.6,82670548.8,78776180.8,76795252.8],"result":{"U":0,"p":0.005074868097940222,"significance":"REJECT"}}



679509

Original result:
  "result": {
    "U": 13060, 
    "p": 0.2375743650159785, 
    "significance": "FAIL_TO_REJECT"
  }, 

New Result:
tracing/bin/compare_samples "/usr/local/google/home/simonhatch/tmp/bisect_json/679509/g1.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679509/g2.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679509/g3.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679509/g4.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679509/g5.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679509/g6.json" "/usr/local/google/home/simonhatch/tmp/bisect_json/679509/b1.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679509/b2.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679509/b3.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679509/b4.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679509/b5.json,/usr/local/google/home/simonhatch/tmp/bisect_json/679509/b6.json" mean_input_event_latency/mean_input_event_latency --chartjson
{"sampleA":[12.868892857142853,12.883678571428572,14.063392857142858,13.938214285714283,11.953214285714287,12.62953571428571],"sampleB":[14.016071428571426,13.82682142857143,15.281678571428568,15.008535714285713,15.695357142857139,15.607178571428571],"result":{"U":3,"p":0.02024057057707751,"significance":"NEED_MORE_DATA"}}

Seems to have made a fairly massive difference, most of those old bisects were able to run through to culprits (or narrowed ranges due to build failures).

\o/

simonhatch@,

Mind linking to the CLs that fixed this?

Sorry, that just get's a little delayed since they're in catapult and it'll update once catapult roller lands in chromium.

https://codereview.chromium.org/2620413003/

Ah shoot. Didn't realize they weren't in Chromium yet. I reran ~6 bisects for videostack team. 

The bug only updates once it makes its way to chromium, but the fix is live since the code is in infra/catapult.

Keep in mind this may or may not fix your individual problems, this will help a lot with bisects where it could clearly reproduce the issue, but wasn't proceeding past gathering the reference values.

Often looked something like this:

Regression: 10%
Values on dashbaord: 10 -> 11


Bisect results:

chromium@000001  10 += 9238503928029
chromium@000010  11 += 3295823095820 (note the crazily large std dev).