All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=595039

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgwMbvvwoM


Bot(s) for this bug's original alert(s):

android-nexus6

Kicked off bisect for this since my variations change was in range.

My change actually didn't land in this revision, so removing myself as owner.

I removed two metrics that have recovered: webrtc.datachannel / vm_private_dirty_final_browser

Kick off a new bisect as the last two have failed.

Drive-by.  # My change is contained in the range so I'm a little bit concerned :)


In the (one month!) regression range there're no datapoints for
  "smoothness.tough_scrolling_cases / mean_pixels_approximated / text_50000_pixels_per_second",
but exactly during the same range there's
  "smoothness.tough_scrolling_cases / mean_pixels_approximated /  Gesture_ScrollAction / text_50000_pixels_per_second"
recorded. Are they referring to the same metric?

If so it looks helpful to narrow down the range:
https://chromeperf.appspot.com/report?sid=49fe4b9613776405920605f3eb5a6c357819bc27b2e5ac0846a08f0fc95e0ea0&start_rev=376332&end_rev=380207

+toyoshim the perf sheriff this week

Kicked a biset between 377511:377520, https://chromeperf.appspot.com/buildbucket_job_status/9016232364322787280

For some reason the bisection is continuously failing, but to me the point of regression
looks to be somewhere in
http://test-results.appspot.com/revision_range?start=377422&end=377510

377383 3.789
377422 1.709 < never scored below this value (with one exception) after this point
377441 4.226
377466 4.408
377485 2.663
377510 6.075 < never reached this high value before
377520 8.016

ymalik@, your change on smooth scrolling is in this range: http://crrev.com/377476
Do you have any thoughts?

It looks like the metric that's failing is text_50000_pixels_per_second which does a gesture scroll. Gesture scrolls don't go through the code path in my CL so its unlikely that my change may have caused this.

Something suspect in these alert is that they have occurred at the same time as re-enabling the ref build metric, and the ref build metric values are in the same range.

So perhaps it is just a side-effect of enabling the reference build?

I agree that it's odd that it's around re-enabling the reference build. I wondered if it changed the board temperature and introduced throttling, but apparently we don't have data that far back?

Kari, Annie, have you seen any issues with ref build changes?

Unfortunately, there is almost a month's worth of data between the previous data point (on February 19) and the data point with the regression (on March 14). So I think the only reason this coincides with re-enabling the ref build is the large range :(

This is also going to be very hard to bisect since the test was failing for so much of the time that we want to get data for.

This only occurred on a single device (Nexus 6) and although we can't verify since the logs are now gone, it's quite likely that their was a device swap in the month we lost data for. WontFix-ing for this reason.

sullivan@, ptal at my comment #6 and #10.

The data in the regression range is not lost, it's just recorded under wrong name.
(specifically due to 376238 http://crrev.com/1709053002 - 380246 http://crrev.com/1777073002)

We still have the datapoints with normal steps enough to narrow down the range from a month to a day:
https://chromeperf.appspot.com/report?sid=49fe4b9613776405920605f3eb5a6c357819bc27b2e5ac0846a08f0fc95e0ea0&start_rev=376332&end_rev=380207



I don't strongly object closing as WontFix if you still think it's approriate, but I just want to point out the fact before the decision.

Looking at the range suggested in #10 one commit jumps out to me:

crrev.com/377490 "content: Improve thread priority for raster threads." by revman@

AFAICT, the commit changes rater worker priority on Android. In particular in lowers the priority on background raster tasks while improving the priority of foreground ones. This supposed to work well on big.LITTLE devices which nexus 6 isn't. Incidentally, I see some improvement on the same metric on galaxy s5 which is big.LITTLE.

This feel like something that can be related. Assigning to reveman@ to take a closer look. 

The graph showing slight improvement in Galaxy S5 vs Nexus 6: https://chromeperf.appspot.com/report?sid=06fcf2732c6809addf54c6d579b0ab3f245fad72ec5786a52169a56268002012&start_rev=375629&end_rev=381721

Here is a screenshot: https://screenshot.googleplex.com/p9P7EvQcOg2.png

reveman@ - any thoughts?

While the % change might look large, the actual values are not really significant. I don't think this is something that is noticeable to the user. Maybe if some of these values would have jumped up to 25% or 50% then it'd be worth investigating more but this all looks like ~5% at most.