All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=599945

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgiN2DrwkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgiIOBoQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgiOvPugsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgiPWapQoM


Bot(s) for this bug's original alert(s):

chromium-rel-mac-hdd
chromium-rel-win7-gpu-nvidia

===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision                Mean Value  Std. Dev.   Num Values  Good?
chromium@383977         5724490.33333424791.917099           good
chromium@384008         5821229.90909518821.58267411          bad

Bisect job ran on: mac_hdd_perf_bisect
Bug ID: 599945

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --also-run-disabled-tests tracing.tracing_with_debug_overhead
Test Metric: Total trace size in bytes/Total trace size in bytes
Relative Change: 0.19%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_hdd_perf_bisect/builds/468
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9016034892016261824


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=599945

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

There is a very clear regression on bing.com (https://screenshot.googleplex.com/O5FmsjnS9Ts), so I kick off a tryjob on bing.com page:
https://chromeperf.appspot.com/buildbucket_job_status/9016018867141786352

===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision                Mean Value  Std. Dev.   Num Values  Good?
chromium@383970         1418712.88889692326.7666318          good
chromium@384008         1568501.25  599970.24537712          bad

Bisect job ran on: mac_hdd_perf_bisect
Bug ID: 599945

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --also-run-disabled-tests tracing.tracing_with_debug_overhead
Test Metric: Total trace size in bytes/http___www.bing.com_
Relative Change: 14.51%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_hdd_perf_bisect/builds/470
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9016018867141786352


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=599945

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Dave, Prasad: why does the bisect thinks 1418712.88 is not that different from 1568501.25?

The means are misleading, because the data is bimodal or trimodal. The medians are very close, and the distributions differ mostly at the 75th percentile. Here's a graph of the results.

Deleted: test_plot.png 60.9 KB

	test_plot.png 60.9 KB View Download

Kick off another bisect on total size instead (still bing.com page): https://chromeperf.appspot.com/buildbucket_job_status/9014643744681494176

Prasad: this is a strange case which bisect results is so different from the waterfall. can you take a look?

Trace before & after the regression:
Before:
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_7-2016-03-29_15-03-04-58171.html

After:
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_7-2016-03-30_17-06-52-61187.html

===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision                Mean Value  Std. Dev.   Num Values  Good?
chromium@383970         1281784.81818418139.98183711          good
chromium@384008         1219184.25  388241.2959712          bad

Bisect job ran on: mac_hdd_perf_bisect
Bug ID: 599945

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests tracing.tracing_with_debug_overhead
Test Metric: Total trace size in bytes/http___www.bing.com_
Relative Change: 8.82%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_hdd_perf_bisect/builds/494
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9014643744681494176


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=599945

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Re-initiated new bisect as previously ran bisect did not provide a possible suspect.
Please track the bisect progress here,
https://chromeperf.appspot.com/buildbucket_job_status/9014060396217538064

Thank You.

===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Fix flash local path.
Author  : aiolos
Commit description:
  
BUG= 568599 

Review URL: https://codereview.chromium.org/1845653002

Cr-Commit-Position: refs/heads/master@{#384072}
Commit  : 4187881d1cf074854b65d1d86492a09a838b0f8c
Date    : Wed Mar 30 20:15:52 2016


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N  Good?
chromium@383877  5533101  46132.1  5  good
chromium@383993  5593911  75482.2  7  good
chromium@384054  5582280  95254.0  5  good
chromium@384067  5561446  74879.5  8  good
chromium@384070  5541951  72491.7  5  good
chromium@384071  5598808  65795.0  5  good
chromium@384072  5867200  124139   5  bad    <--
chromium@384073  5900667  144485   8  bad
chromium@384080  5905217  136720   5  bad
chromium@384108  5889057  100864   6  bad

Bisect job ran on: mac_hdd_perf_bisect
Bug ID: 599945

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests tracing.tracing_with_debug_overhead
Test Metric: Total trace size in bytes/Total trace size in bytes
Relative Change: 6.59%
Score: 99.5

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_hdd_perf_bisect/builds/507
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9014060396217538064


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5796493173194752

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!