All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=800014

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=7dc36a947da5aa5af4c6673562253a11880ec721efad0face4ef36874af25ccc


Bot(s) for this bug's original alert(s):

chromium-rel-win7-gpu-ati
chromium-rel-win7-x64-dual

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/12e091db040000

📍 Couldn't reproduce a difference.
https://pinpoint-dot-chromeperf.appspot.com/job/12e091db040000

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/14fbc467040000

📍 Couldn't reproduce a difference.
https://pinpoint-dot-chromeperf.appspot.com/job/14fbc467040000

tdresser, kouhei: Any interest in digging into this regression?
The dashboard charts look pretty clear to me, but it seems that cpuTimeToFirstMeaningfulPaint and timeToFirstContentfulPaint might be too noisy for Pinpoint to bisect. If you open the pinpoint link in #5 and click the dots in the graph, you'll see the histograms look bimodal.

Why don't we see the histograms for the pinpoint run in #3?

Just making sure I understand what's going on here: pinpoint is reproducing the regression, but believes it isn't statistically significant due to the spread in the variance in the metric?

I see bimodal histograms when I click the points in the charts in both #2-3 and #4-5.

I'm not sure that pinpoint is reproducing the regression. The dashboard timeseries for warm/Kakaku jumped from 250ms to 286ms, but pinpoint reports averages from 251 to 254. The dashboard timeseries for cold/Taobao jumped from 20 to 51, but pinpoint reports a decline from 22.5 to 20ms.
The dashboard charts look clear enough to me that I don't expect broadening the revision range to help pinpoint find the regression.

+dtu: Do you suspect another test environment change here?

I'd interpret this as "Pinpoint did not reproduce the regression." Looks like this metric (comment 3) just is ~20 on some devices and ~52 on some other devices, and for an unknown reason the device running on the perf waterfall switched from the former group to to the latter group.

I've seen a few alerts that follow this pattern. If you are able to see anything in the traces that help us figure out what's different between the two groups, I'd be interested in that.

~20:
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/Taobao_2018-01-08_11-18-22_49775.html
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/Taobao_2018-01-08_11-18-46_7694.html
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/Taobao_2018-01-08_11-17-04_13210.html
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/Taobao_2018-01-08_11-17-30_4200.html

~52:
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/Taobao_2018-01-08_11-25-14_69417.html
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/Taobao_2018-01-08_11-25-50_92564.html
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/Taobao_2018-01-08_11-25-10_89933.html
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/Taobao_2018-01-08_11-24-44_8118.html

In the ~20 case, we're reporting FMP for a single url, but in the ~52 case, we're reporting for 2 urls.

We then proceed to take the average.

The page loads look completely different in the two cases. Is the page load just high non-deterministic?
From the traces, it looks like we actually navigated one additional time in one trace than the other.

https://chromeperf.appspot.com/report?sid=873977ae27369ae4efd5785721aef15a16987b24e8b539b0aa208c4fb9a3d92b

Is there any way in the tooling that we can yell if the number of histogram entries changes?

Enable the count statistic in the metric and alert on *_count?

That sounds good to me, assuming we can communicate what's going on adequately clearly to sheriffs.

We would probably want to apply this to all loading metrics.

It isn't clear to me if it's worth the time investment - have others seen examples of this happening before?

Otherwise we should maybe just keep this in mind as something to do if this becomes more common.

I'm wontfix-ing this bug since it's not a performance regression, but looks similar to https://github.com/catapult-project/catapult/issues/4197