All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=765769

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=2147c6d1ac260b70870bc23948c014408312d0289b1b498882f2ad2571cac956


Bot(s) for this bug's original alert(s):

chromium-rel-win10
chromium-rel-win7-gpu-ati

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8968360434615361760

=== BISECT JOB RESULTS ===
NO Perf regression found

Bisect Details
  Configuration: winx64_10_perf_bisect
  Benchmark    : media.tough_video_cases_tbmv2
  Metric       : memory:chrome:all_processes:reported_by_chrome:v8:effective_size_avg/video.html?src_garden2_10s.mp4

Revision             Result                  N
chromium@501884      5869763 +- 1199932      21      good
chromium@501921      5794865 +- 1166748      21      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=video.html.src.garden2.10s.mp4 media.tough_video_cases_tbmv2

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8968360434615361760


For feedback, file a bug with component Speed>Bisection

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8968352357323663616

#4 is retry, same range

=== BISECT JOB RESULTS ===
NO Perf regression found

Bisect Details
  Configuration: winx64ati_perf_bisect
  Benchmark    : media.tough_video_cases_tbmv2
  Metric       : memory:chrome:all_processes:reported_by_chrome:v8:effective_size_avg/video.html?src_crowd1080.mp4

Revision             Result                 N
chromium@501846      5595136 +- 0.0         21      good
chromium@501931      5620102 +- 511653      21      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=video.html.src.crowd1080.mp4 media.tough_video_cases_tbmv2

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8968352357323663616


For feedback, file a bug with component Speed>Bisection

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/11e6258bb80000

Regression is there, but the bisection sees too much noise on windows recipe at least. #7 is a retry on mac (which has less noise in original graph; also mac bisection is using pinpoint).

📍 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/11e6258bb80000

Roll src/third_party/catapult/ e45810de0..743ca01bf (3 commits)
By catapult-deps-roller@chromium.org · Thu Sep 14 08:12:24 2017
chromium@d814f79c599c6421b5e280a6d0c4b778c748ec5a

📍 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/11e6258bb80000

Roll src/third_party/catapult/ e45810de0..743ca01bf (3 commits)
By catapult-deps-roller@chromium.org · Thu Sep 14 08:12:24 2017
chromium@d814f79c599c6421b5e280a6d0c4b778c748ec5a

📍 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/11e6258bb80000

Roll src/third_party/catapult/ e45810de0..743ca01bf (3 commits)
By catapult-deps-roller@chromium.org · Thu Sep 14 08:12:24 2017
chromium@d814f79c599c6421b5e280a6d0c4b778c748ec5a

📍 Pinpoint job stopped with an error.
https://pinpoint.chromeperf.appspot.com/job/11e6258bb80000

Roll src/third_party/catapult/ e45810de0..743ca01bf (3 commits)
By catapult-deps-roller@chromium.org · Thu Sep 14 08:12:24 2017
chromium@d814f79c599c6421b5e280a6d0c4b778c748ec5a

📍 Pinpoint job stopped with an error.
https://pinpoint.chromeperf.appspot.com/job/11e6258bb80000

Roll src/third_party/catapult/ e45810de0..743ca01bf (3 commits)
By catapult-deps-roller@chromium.org · Thu Sep 14 08:12:24 2017
chromium@d814f79c599c6421b5e280a6d0c4b778c748ec5a

📍 Pinpoint job stopped with an error.
https://pinpoint.chromeperf.appspot.com/job/11e6258bb80000

Roll src/third_party/catapult/ e45810de0..743ca01bf (3 commits)
By catapult-deps-roller@chromium.org · Thu Sep 14 08:12:24 2017
chromium@d814f79c599c6421b5e280a6d0c4b778c748ec5a

📍 Pinpoint job stopped with an error.
https://pinpoint.chromeperf.appspot.com/job/11e6258bb80000

Roll src/third_party/catapult/ e45810de0..743ca01bf (3 commits)
By catapult-deps-roller@chromium.org · Thu Sep 14 08:12:24 2017
chromium@d814f79c599c6421b5e280a6d0c4b778c748ec5a

📍 Pinpoint job stopped with an error.
https://pinpoint.chromeperf.appspot.com/job/11e6258bb80000

Roll src/third_party/catapult/ e45810de0..743ca01bf (3 commits)
By catapult-deps-roller@chromium.org · Thu Sep 14 08:12:24 2017
chromium@d814f79c599c6421b5e280a6d0c4b778c748ec5a

📍 Pinpoint job stopped with an error.
https://pinpoint.chromeperf.appspot.com/job/11e6258bb80000

Roll src/third_party/catapult/ e45810de0..743ca01bf (3 commits)
By catapult-deps-roller@chromium.org · Thu Sep 14 08:12:24 2017
chromium@d814f79c599c6421b5e280a6d0c4b778c748ec5a

📍 Pinpoint job stopped with an error.
https://pinpoint.chromeperf.appspot.com/job/11e6258bb80000

Reland Add tab.IsServiceWorkerReadyOrNotRegist() which tells whether serviceworker registration is finished ( https://chromiumcodereview.appspot.com/3012573002 )
By yukiy@google.com · Thu Sep 14 04:28:17 2017
catapult@b8895fa6e92daaaf6d4d2c13ae1d358bc056aea5

📍 Pinpoint job stopped with an error.
https://pinpoint.chromeperf.appspot.com/job/11e6258bb80000

Reland Add tab.IsServiceWorkerReadyOrNotRegist() which tells whether serviceworker registration is finished ( https://chromiumcodereview.appspot.com/3012573002 )
By yukiy@google.com · Thu Sep 14 04:28:17 2017
catapult@b8895fa6e92daaaf6d4d2c13ae1d358bc056aea5

📍 Pinpoint job stopped with an error.
https://pinpoint.chromeperf.appspot.com/job/11e6258bb80000

Reland Add tab.IsServiceWorkerReadyOrNotRegist() which tells whether serviceworker registration is finished ( https://chromiumcodereview.appspot.com/3012573002 )
By yukiy@google.com · Thu Sep 14 04:28:17 2017
catapult@b8895fa6e92daaaf6d4d2c13ae1d358bc056aea5

yukiy@, if I understand pinpoint correctly, it identified https://crrev.com/b8895fa6e92daaaf6d4d2c13ae1d358bc056aea5 as the cause of this performance regression.

cc: crouleau@,dtu@: Caleb/Dave - several comments about strange pinpoint behavior versus the bisect recipe system:
1) pinpoint didn't automatically update/reassign this crbug. Is that a known issue?
2) the pinpoint URL in #20 gives me a cert error (pinpoint.chromeperf... versus pointpoint-dot-chromeperf...; link in #7 works)
3) Why are there so many updates from pinpoint, and what does "Pinpoint job stopped with an error" mean? 

Yes, that's right.

1) Yes, that's not implemented. I'll discuss with the team on Wednesday what we want to do for it. Because Pinpoint supports multiple regressions, there may not be a clear owner, so I'm leaning towards just CCing the CL authors and not assigning.

2) Sorry, that's my bad. I used the dev console to mess with some of the jobs' internals on Friday, and the ones I modified have incorrect links.

3a) Filed a bug for the duplicate posts. https://github.com/catapult-project/catapult/issues/3872

3b) It looks like this job failed at the very end, so it still produced the correct culprit despite erroring out. Filed a bug for that, too. https://github.com/catapult-project/catapult/issues/3875

Hi wolenetz@, Thanks for reporting!
I added some methods in https://crrev.com/b8895fa6e92daaaf6d4d2c13ae1d358bc056aea5 , but they are not used yet, so only the loading time of wait_for_serviceworker_registration.js as script_to_evaluate_on_commit affects now.
I think this won't be fixed, but is this regression so much big? If not, I want to close this bug.

@#22 thanks for follow-up

@#23, I'm less certain - are your changes test-only? Perhaps check with area owner(s) to be more certain about what performance impacts are expected/allowed. Tentatively cc'ing shimazu@ for assistance answering your question, based on scanning the crbug linked from https://crrev.com/b8895fa6e92daaaf6d4d2c13ae1d358bc056aea5.

Sorry for my slow reply.
The change made by yukiy@ is just for testing, so there is no real regression here.
I'm also not familiar with the metrics, but the difference looks in the order of flakiness.

If crouleau@ doesn't have any concerns on it, I think we can just close this issue.

I'm not sure what "just for testing" means. If it impacts Chrome's memory usage, then it doesn't matter why you wrote the code.

I don't know as much about Pinpoint, and it looks like the website is down, but for the predecessor of Pinpoint (named "Bisect"), if it decided that your change was the culprit, then that was because it ran a statistical test and got pretty sure of it. For Bisect, it even ran the test a bunch of times and computed the the standard deviation of the results and showed that to you. So if Pinpoint is like Bisect, then your change *did* cause this regression. 

You should check with dtu@ that he is confident in the statistical tests that Pinpoint is doing (maybe he can share the data with us?). After that, you can run the test yourself to see:

src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=video.html.src.crowd1080.mp4 media.tough_video_cases_tbmv2

Sorry for lack of explanation!
"just for testing" means that the change (crrev.com/3010183002) was on telemetry itself and no change in chromium. It injects small amount of code to track if the testing sites installed service workers, and wait for the installation if it's installed.
That doesn't affect the speed metrics, but small amount of difference on memory usage before/after the patch is expected since the following script is injected through devtools protocol.
https://codereview.chromium.org/3010183002/diff/340001/telemetry/telemetry/internal/browser/wait_for_serviceworker_registration.js

Ah great! Thank you. Don't bother looking into this then :)

#26: yep, it's similar to Bisect in that it runs many iterations and performs statistical tests. Pinpoint's test is stricter than Bisect's test, and I'm very sure of these results.
Sorry, the links were incorrect on Friday. You can find the full data at https://pinpoint-dot-chromeperf.appspot.com/job/11e6258bb80000.