All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=756481

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=e615702b550dd583fb8b063a61464518b6b23cc34bd10815cf0de58bbba320ea


Bot(s) for this bug's original alert(s):

chromium-rel-mac11-pro

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8970998702555932032

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8970998652728142352

=== Auto-CCing suspected CL author ksakamoto@chromium.org ===

Hi ksakamoto@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : Kunihiko Sakamoto
  Commit : ca2e6be7bac44462efdd864ef346311b9a6033de
  Date   : Tue Aug 15 04:08:53 2017
  Subject: Convert loading.desktop page set to WprGo

Bisect Details
  Configuration: mac_pro_perf_bisect
  Benchmark    : loading.desktop
  Metric       : timeToFirstMeaningfulPaint_avg/pcv1-cold/Rambler
  Change       : 30.80% | 374.954999999 -> 490.438

Revision             Result                  N
chromium@494236      374.955 +- 20.3577      6      good
chromium@494298      367.162 +- 27.2692      6      good
chromium@494314      372.333 +- 23.1552      6      good
chromium@494322      358.322 +- 17.934       6      good
chromium@494323      495.599 +- 87.9145      6      bad       <--
chromium@494324      427.247 +- 97.592       6      bad
chromium@494326      473.185 +- 167.825      6      bad
chromium@494329      507.562 +- 229.275      6      bad
chromium@494360      502.878 +- 106.881      6      bad
chromium@494484      490.438 +- 158.854      6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=Rambler loading.desktop

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8970998652728142352


For feedback, file a bug with component Speed>Bisection

Trace before WprGo:
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_41-2017-08-14_11-56-50-15270.html

Trace after WprGo:
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_41-2017-08-16_07-42-11-79937.html


In the trace after WprGo, there are some long CertVerifierJob tasks in WorkerPools, posted here:
https://cs.chromium.org/chromium/src/net/cert/multi_threaded_cert_verifier.cc?sq=package:chromium&dr&l=230

I think this is the reason of the FCP/FMP regression.

xunjieli@, any ideas why cert verification takes longer w/ WprGo?

Thanks ksakamoto@. I am not familiar with the certs setup before WprGo. 

Right now because we are not installing a trusted local root anchor on the test machine, we use "--ignore-certificate-errors" Chrome startup flag to bypass certificate errors. This is not good for performance testing because //net opens two SSL connections for every https request. This --ignore-certificate-errors is not a usual setup, so I was told by folks who work on this that there can be unoptimized paths.  Disk cache is bypassed as well.

Our goal is to install a local CA properly on test device (e.g.  Issue 753948 ) through Telemetry and get rid of --ignore-certificate-errors dependency.

Since the metrics are pretty consistent post migration, I'd suggest to WontFix this bug. 

I kicked off a run on perf trybot and got some NetLog dumps. Attached in:
https://build.chromium.org/p/tryserver.chromium.perf/builders/mac_pro_perf_bisect/builds/490

davidben@ helped me with the NetLog dumps. What happened is that the the old WPR serves exactly one certificate for all domains, so CertVerifierJobs cache the result and proceed quickly.
The WprGo uses different certs for different hosts. This is a more realistic behavior which also means that CertVeriferJobs are not cached as much.

One other thing which caused the CertVerifierJob to take long is that WprGo didn't strip "Authority Information Access" extension and CertVerifierJob starts to fetch OCSP information. I filed https://github.com/catapult-project/catapult/issues/3785 to address this.

Small clarification: I suspect it's not bothering to fetch OCSP, since the chain isn't valid, and we only do it for EV certs (OCSP is a revocation check). But it probably is checking the CA issuers list for the missing intermediate certificates (which it won't find, but it doesn't know that). My guess is that would explain why the certificate verification takes longer on the new WPR.

Ok. I clarified with David offline. Once Telemetry installs a local trust anchor correctly ( Issue 753948  and  Issue 756989 ), we do not need worry about AIA extension. Chrome cert verification will pass because WprGo certs are chained to the local trust anchor, and fetching additional intermediates is not needed.

Thanks xunjieli@ and davidben@ for investigation.
Closing this as we now understand the cause.

 Issue 751983  has been merged into this issue.

 Issue 758193  has been merged into this issue.