Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8993287998055748752

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8993287566065133776

===== BISECT JOB RESULTS =====
Status: started


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N   Good?
chromium@437909  0          N/A      20  good
chromium@438086  0          N/A      20  bad

Bisect job ran on: linux_perf_bisect
Bug ID: 674146

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http...www.ebay.com v8.top_25_smooth
Test Metric: v8_gc_total/http___www.ebay.com
Relative Change: 0.00%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6939
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8993287998055748752


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6409333506572288

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

===== BISECT JOB RESULTS =====
Status: started


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N   Good?
chromium@437986  0          N/A      20  good
chromium@438090  0          N/A      20  bad

Bisect job ran on: linux_perf_bisect
Bug ID: 674146

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http...www.ebay.com v8.top_25_smooth
Test Metric: v8_gc_total/http___www.ebay.com
Relative Change: 0.00%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6940
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8993287566065133776


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5823946643996672

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Ok so you can see something clearly happened between build 113 and 114.  Revision range: 437986-438090

If you evaluate this link: https://chromium-swarm.appspot.com/tasklist?c=started_ts&c=state&c=created_ts&c=duration&c=abandoned_ts&f=cpu%3Ax86-64&f=gpu%3A102b%3A0534&f=id%3Abuild150-m1&l=500&s=name%3Aasc

and sort by task name you will see a very stark jump in duration from build 113 to 114, some times 2-4 times the run time consistently.  It is not gradual at all, it is a very clear change at build 114.  I have detailed out the jumps below, interestingly it mostly happens on the non-reference build and the reference build remains the same low number, but there are a few reference jumps as well.  

So I evaluated from build 113 to 114 tests that jumped in time: 

blink_perf.css went from consistently 3m11s to consistently 6m30s, but the ref build didn't jump

blink_perf.dom went from 3m to 7m but reference build remained low

blink_perf.svg_slimmingpaintinvalidation went from 3m to 8m but not on reference build

blink_style.top_25 wwent from 7m to 14m on both reference and non-reference build

media.tough_video_cases_extra went from 6m to 10m, but not on reference

oilpan_gc_times.tough_animation_cases went from 22m to 41m but not on the reference build

page_cycler_v2.basic_oopif went from 8m to 14m but not on reference

page_cycler_v2.intl_ar_fa_he went from 16m to 16m

page_cycler_v2.typical_25 went from 37m to 1h6m

rasterize_and_record_micro.key_mobile_sties from 14m to 26m and reference went up

smoothness.tough_webgl_cases from 1m to 4m

No other bots on the waterfall seem to be reflecting this change, so it seems to be something specific to this hardware.  

https://codereview.chromium.org/2568033003 was the culprit