All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=639756

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg7sSFswoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg7p7vtQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg7vvAtwkM


Bot(s) for this bug's original alert(s):

win-zenbook

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9003644144865459712

===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@413313  490.379  5.72985  12  good
chromium@413315  492.632  3.67374  5   good
chromium@413316  515.92   25.1332  8   bad
chromium@413318  511.426  4.46716  5   bad
chromium@413323  505.386  17.7657  18  bad

Bisect job ran on: winx64_zen_perf_bisect
Bug ID: 639756

Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --also-run-disabled-tests speedometer
Test Metric: BackboneJS-TodoMVC/BackboneJS-TodoMVC
Relative Change: 1.28%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/407
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9003644144865459712


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5301170822709248

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9003627261700887808

===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@413313  490.379  5.72985  12  good
chromium@413315  492.632  3.67374  5   good
chromium@413316  515.92   25.1332  8   bad
chromium@413318  511.426  4.46716  5   bad
chromium@413323  505.386  17.7657  18  bad

Bisect job ran on: winx64_zen_perf_bisect
Bug ID: 639756

Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --also-run-disabled-tests speedometer
Test Metric: BackboneJS-TodoMVC/BackboneJS-TodoMVC
Relative Change: 1.28%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/407
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9003644144865459712


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5301170822709248

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9003626038675499088

=== Auto-CCing suspected CL author abakalov@chromium.org ===

Hi abakalov@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Updating the GitHub commit hash for the language identification model (CLD3). The update adds support for 19 more languages.
Author  : abakalov
Commit description:
  
BUG=624904

Review-Url: https://codereview.chromium.org/2260973002
Cr-Commit-Position: refs/heads/master@{#413318}
Commit  : d93872ee3b832e489a26ac4527ad95f176ef1b60
Date    : Sat Aug 20 04:05:22 2016


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@413313  1423.24  20.7927  27  good
chromium@413316  1424.48  34.5183  27  good
chromium@413317  1407.89  10.621   5   good
chromium@413318  1448.19  46.8895  27  bad    <--
chromium@413323  1463.93  68.575   27  bad

Bisect job ran on: winx64_zen_perf_bisect
Bug ID: 639756

Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --also-run-disabled-tests speedometer
Test Metric: EmberJS-TodoMVC/EmberJS-TodoMVC
Relative Change: 2.38%
Score: 95.0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/410
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9003626038675499088


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5874314075701248

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

I don't think our CL is responsible for the regression because:
- the regression is just for Windows but the model is in Canary on all platforms
- looking at the plots, the latency drops after our CL. Besides, the variance seems quite high around the time our CL was submitted.
- based on the comments in the speedometer code, the tests are about user actions involving todos. However, our model should be independent of these actions. It should be just asynchronously processing the text once the page is loaded.

Should we restart the bisect job?

The fact that the bisect failed twice beforehand is worrisome. +prasad who is on bisect triage.

Also, worrisome: why did the mean jump so much higher in the last bisect? (And note that chromium@413316 switched from bad to good.)

Graph looks noisy, I'm not sure whether bisect could yield any better results.

+dtu@ and robertocn@

Thank you all for taking a look! Adding dtu@ because I think prasadv@ meant to cc him.

Let's close this one, clearly flaky tests.

+flaky hardware

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9003532769511243136

===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : cc : Add OnOpacityAnimated to effect tree
Author  : jaydasika
Commit description:
  
And use it to update opacity animations directly instead of
routing the updates through layers

BUG= 638297 
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel

Review-Url: https://codereview.chromium.org/2253233003
Cr-Commit-Position: refs/heads/master@{#413319}
Commit  : 898b2f178e201a89384927f992735924bdc50ead
Date    : Sat Aug 20 04:23:43 2016


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N  Good?
chromium@413313  570.849  53.2507  5  good
chromium@413318  595.307  15.5098  8  good
chromium@413319  620.667  5.67477  5  bad    <--
chromium@413320  642.766  56.873   5  bad
chromium@413321  632.166  43.4604  8  bad
chromium@413323  633.928  25.0219  5  bad

Bisect job ran on: winx64_zen_perf_bisect
Bug ID: 639756

Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --also-run-disabled-tests speedometer
Test Metric: BackboneJS-TodoMVC/BackboneJS-TodoMVC
Relative Change: 11.05%
Score: 80.0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/409
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9003644144865459712


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5301170822709248

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!