All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=651547

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgpdu1qAkM


Bot(s) for this bug's original alert(s):

linux-release

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9000157274219118832

=== Auto-CCing suspected CL author robliao@chromium.org ===

Hi robliao@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Update fieldtrial_util To Handle Combined Fieldtrial Config Format
Author  : robliao
Commit description:
  
BUG=649363, 650705

Review-Url: https://codereview.chromium.org/2373843002
Cr-Commit-Position: refs/heads/master@{#421670}
Commit  : 8da8200a279fcafc4a7c68f2e6ef95f4d721bb3a
Date    : Wed Sep 28 22:51:09 2016


===== TESTED REVISIONS =====
Revision         Mean     Std Dev   N  Good?
chromium@421667  7.61496  0.898269  5  good
chromium@421669  7.76002  0.413934  5  good
chromium@421670  22.3252  0.952023  5  bad    <--
chromium@421672  21.3529  0.677023  5  bad
chromium@421677  22.7862  0.854803  5  bad
chromium@421686  20.8573  1.20169   5  bad
chromium@421705  22.4349  0.914063  5  bad

Bisect job ran on: linux_perf_bisect
Bug ID: 651547

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests smoothness.scrolling_tough_ad_cases
Test Metric: first_gesture_scroll_update_latency/http___www.latimes.com
Relative Change: 194.62%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6734
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9000157274219118832


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5853479975780352

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

We had about week loss of coverage due to http://crbug.com/649363 (9/21 - 9/28).

tommycli, could https://codereview.chromium.org/2360083003 potentially have caused this?

It's... possible. Were there Flash ads on that site? And does the bot even support Flash?

A quick visit to latimes.com suggests that Flash is indeed on the site.

So... with that field trial configuration change, I'd expect Flash to be turned off. (You can test by going to chrome://plugins and turning off Flash)

Does that negatively impact the measured variable? (Scroll latency?)

That I do not know. Do you want to retry reverting your change to see if the perf bots recover?

robliao: Actually, Yes, since it's a small CL, I don't mind if you speculatively revert to see if it fixes it.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/03f4f7abbeee8660189dc9e1a3504790b2d74adc

commit 03f4f7abbeee8660189dc9e1a3504790b2d74adc
Author: robliao <robliao@chromium.org>
Date: Sat Oct 01 02:16:40 2016

Revert of [HBD] Add Field Trial testing config for PreferHtmlOverPlugins (patchset #5 id:80001 of https://codereview.chromium.org/2360083003/ )

Reason for revert:
Speculative revert to see if smoothness.scrolling_tough_ad_cases improves

Original issue's description:
> [HBD] Add Field Trial testing config for PreferHtmlOverPlugins
>
> BUG= 626728 
>
> Committed: https://crrev.com/41667aa90a95103ac486a938b1b513bf33a72613
> Cr-Commit-Position: refs/heads/master@{#420907}

TBR=engedy@chromium.org,rkaplow@chromium.org,tommycli@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 626728 ,  651547 

Review-Url: https://codereview.chromium.org/2375153008
Cr-Commit-Position: refs/heads/master@{#422288}

[modify] https://crrev.com/03f4f7abbeee8660189dc9e1a3504790b2d74adc/testing/variations/fieldtrial_testing_config.json

The hunt continues.

So the field trial CL was innocent?

It appears so :-/

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8999730733564093296

===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev    N   Good?
chromium@421507  2.87364  0.0139674  18  good
chromium@421538  2.87398  0.013028   18  bad

Bisect job ran on: android_nexus5_perf_bisect
Bug ID: 651547

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests thread_times.key_idle_power_cases
Test Metric: tasks_per_second_total_all/set-timeout.html (Long Idle)
Relative Change: 0.16%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/4174
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8999730733564093296


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5240953281970176

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8999636111345654848

===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev    N   Good?
chromium@421507  2.8742   0.0151867  18  good
chromium@421538  2.87852  0.0202512  18  bad

Bisect job ran on: android_nexus5_perf_bisect
Bug ID: 651547

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests thread_times.key_idle_power_cases
Test Metric: tasks_per_second_total_all/set-timeout.html (Long Idle)
Relative Change: 0.23%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/4185
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8999636111345654848


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6457130784129024

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Is there a way to bisect with a patch applied to historical builds?

There is not, sorry.

 Issue 652303  has been merged into this issue.

I suspect this will need to be split up to the component owners due to the temporary loss of coverage from field trials in the perfbots during this regression.

Many bots have recovered (and I've cleared those from the alerts), but others remain in their regression state.

Spliting the remaining alerts that haven't cleared by now.

These regressions happened before M56 branch. M56 is now in stable. These regressions made it to the stable channel. Marking wontfix.