All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=591406

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn_3TsgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn-ubogoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn6HqowoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA38bHuAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn7mxogoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn8-pqAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn6GzugsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn6-QtQsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn8XnrwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA3_KVoQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn-XivQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn7GjtAkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA3-b7oAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA38GkvAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA3-zZsQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn6ONpgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn6PrtwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA37iPrgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn9u1pQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn_nurgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn9urtAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn_nurgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA39KIvgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA3-bYugoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA3-aCpwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA3-aWtgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn4PmtgsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn4Pm9ggM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn5GFvQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn4Pm9goM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn7bpuwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn7uSowoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn-_7sAsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn-_78AgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn9bhpAkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn4P85wgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA38y3rQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn_vMrwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA37KSvQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn_uwtgsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn9Ll5wsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn_uVsQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn_3FowkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA35b3vQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA38GZqwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn_uLqAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA3-yuoAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA38GDugoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn8XSqwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA39jWpAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA35b3vQkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn_vvuQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn_uVsQkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn4f4pAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn-ubogkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn8XOugoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn_vvuQkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA38HnpwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn4e2rAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn9_NtgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA39yEuwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA37jFpgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn7K4vgsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn_mUpgsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA39ySpgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA3_LvqQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA3-HHswoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA35HQqgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn7K4_ggM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA3-HHswkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA39z2twoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn6WhqAkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn7HXrQsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA3-HHswsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA3-GpvQsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn7HX7QgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA34WxpwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA34r4uAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA3-Gp_QgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn6XsoAkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA39j7vQsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA37jgogsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn5HKtQkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA36G6vgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn9-brwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn7-AvgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn4fJvQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn_eZtAkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn6XLpAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn4fJvQkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA34qipQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA36G6vgsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn9-brwkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn5HKtQsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA35GMuwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn7-AvgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA35GapgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDA35H-twoM


Bot(s) for this bug's original alert(s):

android-nexus5
android-nexus6
android-nexus7v2
android-one
chromium-rel-mac-hdd
chromium-rel-mac-retina
chromium-rel-mac10
chromium-rel-mac11
linux-release

=== Auto-CCing suspected CL author dgozman@chromium.org ===

Hi dgozman@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : [DevTools] Postpone more agents instantiation until attached.
Author  : dgozman
Commit description:
  
To support multiple attached devtools, we have to instantiate
individual agents on demand, rather than on creation.
This patch moves most of agents, with only worker and inspector to go.

BUG= 590878 

Review URL: https://codereview.chromium.org/1746393003

Cr-Commit-Position: refs/heads/master@{#378509}
Commit  : b547858514737d59af3c55b2b06b4e511624c304
Date    : Tue Mar 01 19:17:51 2016


===== TESTED REVISIONS =====
Revision                Mean Value  Std. Dev.   Num Values  Good?
chromium@378507         18.76       1.470714    5           good
chromium@378508         19.5        0.863134    5           good
chromium@378509         98.4        3.040559    5           bad
chromium@378510         96.64       1.74442     5           bad
chromium@378513         98.32       1.834939    5           bad
chromium@378519         99.78       4.563113    5           bad

Bisect job ran on: linux_perf_bisect
Bug ID: 591406

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --also-run-disabled-tests blink_perf.pywebsocket
Test Metric: XHR-send-blob-worker-sync-verify/XHR-send-blob-worker-sync-verify
Relative Change: 431.88%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6266
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9019284174577641936


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with label Cr-Tests-AutoBisect.  Thank you!

dgozman@, would your CL have affected blink_perf.pywebsocket perf test results?

The bisect appears to have reproduced a ~400% increase in *-verify time (~20 ms to 90 ms) across all platforms.

https://chromeperf.appspot.com/group_report?bug_id=591406

pfeldman@, dgozman@, do you know why CL might have affected some Telemetry performance test results?

I will look into this today.

I don't see how this may affect performance, but let's revert and see whether it helps.

Thanks! After the revert, if that CL did affect performance test results on the chromium.perf waterfall, we should be able to see a change within a few hours on most/all of the graphs (https://chromeperf.appspot.com/group_report?bug_id=591406).

This CL likely leaks memory, see:

https://bugs.chromium.org/p/chromium/issues/detail?id=591314

I clicked through random graphs, they all recovered. Marking as fixed. Please reopen if some graphs are still in a bad shape.