All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=797416

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=67fcb13d7f07aa0270bab53d408af1b7fd045d267639a98d030b6a20338c778a


Bot(s) for this bug's original alert(s):

chromium-rel-win10

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/12a426b1040000

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/12a426b1040000

[Telemetry] Update system_health_smoke_test to use expectations file.
By rnephew@chromium.org · Mon Dec 18 20:19:30 2017
chromium @ ae3f936d5d6e9c26c9d0ad76d4a61ce028cd8897

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/14f1f329040000

It seems highly unlikely that that change would cause a regression. It just changes how we decide if a test is disable. The expected failure mode of this would be it running tests it shouldn't, not running tests it should, or failing disastrously. Nothing in this CL changes how tests are actually run, how measurements are taken, or any changes to the browser itself.

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/14f1f329040000

VideoFrame::BitsPerChannel() doesn't need the parameter
By mcasas@chromium.org · Mon Dec 18 20:28:23 2017
chromium @ 134e466415cf125f27905d1ca8646bdaf265496a

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

I'm not 100% sure that the CL in c6 is the guilty one although
it's true that it touches video, so it might affect the parameters
in the chromeperf, but if so it'd be way more widespread than just
on Win10 and for the cnn homesite -- this CL just removes a parameter
in a method call.

brianderson@ could you retrigger the pinpoint job or paste here a link
to the CLs in range? Thx

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/1291675d040000

Range according to the dashboard:
https://chromium.googlesource.com/chromium/src/+log/bb65e11e10c704249b39c1060405887245cf3d68%5E..00710b48b6d06931a868dc6982b0daafd8875560?pretty=fuller&n=1000

📍 Couldn't reproduce a difference.
https://pinpoint-dot-chromeperf.appspot.com/job/1291675d040000

tdresser: This metric seems super noisy, pinpoint is reproducing regressions at different CLs on different runs.

There's noise in the number of navigations we're seeing. 
I think we should switch to alerting on the max value, instead of the average.

Deep, does that make sense to you?

I hope I'm understanding this correctly: It looks like this story actually navigates multiple times [1] - it goes to the home page and then clicks two more links. Here[2] is a link to the trace. Taking the max here makes me uncomfortable because the intention of the test was probably to monitor average loading metrics, and if we only take the max we will almost certainly only capture the first cold navigation to the home page and not have any visibility into the rest of the browsing. 

I looked around in some other traces in pinpoint, and the number of navigation reaching first paint is consistently five.  


[1] https://cs.chromium.org/chromium/src/tools/perf/page_sets/system_health/browsing_stories.py?l=112&rcl=43487fc89193d2cd284c17d355787727482763bd

[2] https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/browse_news_cnn_2018-01-02_18-01-55_66952.html

Deep and I chatted about this. Sounds like avg is probably more correct here (though full histogram diffing will make more sense).

If we configure pinpoint correctly, the failure mode should be failing to reproduce regressions, not identifying different CLs on different runs.

Do we need to bump up the threshold for regression detection?

Naively, the graphs themselves appear relatively stable.

Bisect can't repro.