All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=736279

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgzsHj9QoM


Bot(s) for this bug's original alert(s):

android-webview-nexus6

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8976004943934626864

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8976004816345056208

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8975822805290651184

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8975822801400507488

=== Auto-CCing suspected CL author davidben@chromium.org ===

Hi davidben@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : David Benjamin
  Commit : 11464f93e041d5b6ae47e2f8dfb2765ab727d7e8
  Date   : Thu Jun 22 22:14:37 2017
  Subject: Revert "Don't take a mutex around certificate verifications on Android."

Bisect Details
  Configuration: android_webview_nexus6_aosp_perf_bisect
  Benchmark    : memory.top_10_mobile
  Metric       : memory:webview:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size_avg/foreground/http_yandex_ru_touchsearch_text_science
  Change       : 4.94% | 3885056.0 -> 4077056.0

Revision             Result                  N
chromium@481655      3885056 +- 161234       6      good
chromium@481672      3940523 +- 106677       6      good
chromium@481681      3930965 +- 65574.7      6      good
chromium@481685      3914581 +- 59187.1      6      good
chromium@481687      3949397 +- 62596.8      6      good
chromium@481688      3931648 +- 57229.6      6      good
chromium@481689      4077056 +- 29242.3      6      bad       <--

Please refer to the following doc on diagnosing memory regressions:
  https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests memory.top_10_mobile

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8975822805290651184

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5035439821422592


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Seriously? This was reverted because of  issue #735659  which was a report of a regression by the same system! Bouncing this back so you all can figure out what's going on with these alerts.

Ah yeah and I see from the graph that there is a corresponding drop in that metric when my change went in.

Unfortunately, like I said on the other bug, there is nothing I can do here. The tooling for WebView did not reproduce locally and the alerts themselves do not have sufficient tracing to diagnose this. I would need to see //net-level tracing and the ability to schedule my own runs so I can add some traces of the certificate verify calls.

> Ah yeah and I see from the graph that there is a corresponding drop in that metric when my change went in.

Sorry, there is a drop when the original change went in, and a regression when the revert went in. Unfortunately, it's countered by the inverse in a different metric! For more fun, the change wasn't meant to touch certificate verification memory at all.

(What is necessary is seeing a trace of the certificate verification tasks that were scheduled before & after, so I can see if there is something funny about their scheduling. Or perhaps making things faster even changed what the benchmark did, which would suggest noise.)

=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : David Benjamin
  Commit : 11464f93e041d5b6ae47e2f8dfb2765ab727d7e8
  Date   : Thu Jun 22 22:14:37 2017
  Subject: Revert "Don't take a mutex around certificate verifications on Android."

Bisect Details
  Configuration: android_webview_nexus6_aosp_perf_bisect
  Benchmark    : memory.top_10_mobile
  Metric       : memory:webview:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size_avg/foreground/http_yandex_ru_touchsearch_text_science
  Change       : 4.73% | 3904739.55556 -> 4089344.0

Revision             Result                  N
chromium@481688      3904740 +- 252904       9      good
chromium@481689      4089344 +- 76406.5      6      bad       <--

Please refer to the following doc on diagnosing memory regressions:
  https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests memory.top_10_mobile

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8975822801400507488

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6126942811586560


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Hypothesis has been confirmed.
https://chromium.googlesource.com/chromium/src/+/11464f93e041d5b6ae47e2f8dfb2765ab727d7e8
is the reversal of faulty improvement.