All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=708644

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgnOyFtAsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg3P2m_QgM


Bot(s) for this bug's original alert(s):

android-nexus5X
android-nexus6

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8983135147338093856

=== Auto-CCing suspected CL author tzik@chromium.org ===

Hi tzik@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : tzik
  Commit : e5de8d551e80115b21e0e3ca7d5451d3cee6f3ba
  Date   : Thu Mar 30 18:05:46 2017
  Subject: Clear PostTaskAndReply task on the destination thread

Bisect Details
  Configuration: android_nexus5X_perf_bisect
  Benchmark    : system_health.memory_mobile
  Metric       : memory:chrome:all_processes:reported_by_os:system_memory:native_heap:proportional_resident_size_avg/load_news/load_news_flipboard
  Change       : 2.46% | 65369224.0 -> 66975709.3333

Revision             Result                   N
chromium@460748      65369224 +- 667162       6      good
chromium@460818      65345331 +- 393366       6      good
chromium@460820      65457800 +- 277744       6      good
chromium@460821      67061213 +- 436489       6      bad       <--
chromium@460823      67134771 +- 902904       6      bad
chromium@460827      66979976 +- 649033       6      bad
chromium@460836      67017352 +- 616263       6      bad
chromium@460853      67074355 +- 1024702      6      bad
chromium@460888      66975709 +- 659303       6      bad

Please refer to the following doc on diagnosing memory regressions:
  https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.news.flipboard system_health.memory_mobile

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8983135147338093856

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5291240184086528


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

 Issue 708574  has been merged into this issue.

 Issue 708639  has been merged into this issue.

Issue 708527 has been merged into this issue.

Looking.

 Issue 708558  has been merged into this issue.

 Issue 708559  has been merged into this issue.

I think I identified the mechanism. My CL moved a bunch of free() call from a thread to another thread, that is, some IOBuffer instances used by net::disk_cache::SimpleEntryImpl are allocated on the IO thread, and free()ed on the CACHE thread. Then, the malloc implementation stores the free()ed buffer into the thread local free list, and the observed memory consumption has grown.

Here is an attempt to fix it: http://crrev.com/2803023003

Issue 708537 has been merged into this issue.

 Issue 708648  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1714b61635cc8207647b1a9857e8042fbb833025

commit 1714b61635cc8207647b1a9857e8042fbb833025
Author: tzik <tzik@chromium.org>
Date: Thu Apr 13 08:12:10 2017

Avoid cross thread malloc/free pair of IOBuffer on the simple cache

This CL removes a cross thread malloc/free pair from simple disk cache backend.

After a PostTaskAndReply change in http://crrev.com/e5de8d551e80115b, it
destroys |task| part of the given tasks on the target thread, and that
introduced a number of cross thread malloc/free pairs around net::IOBuffer.
The cross thread malloc/free pair increased the apparent size of memory
usage on some Android perf bots by 2~3%, that is likely due to thread
specific free list caches.

BUG= 708644 

Review-Url: https://codereview.chromium.org/2815563002
Cr-Commit-Position: refs/heads/master@{#464338}

[modify] https://crrev.com/1714b61635cc8207647b1a9857e8042fbb833025/net/disk_cache/simple/simple_entry_impl.cc
[modify] https://crrev.com/1714b61635cc8207647b1a9857e8042fbb833025/net/disk_cache/simple/simple_entry_impl.h

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982402988639663088

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982402957999103792

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9fcfc8861ae30d5f1f5acf2d4cc0237f7f4e4f59

commit 9fcfc8861ae30d5f1f5acf2d4cc0237f7f4e4f59
Author: tzik <tzik@chromium.org>
Date: Thu Apr 13 21:31:09 2017

Revert of Avoid cross thread malloc / free pair of IOBuffer on the simple cache (patchset #2 id:20001 of https://codereview.chromium.org/2815563002/ )

Reason for revert:
Another large memory reduction seems to land in the same batch of the performance dashboard job, and it hid the improvement of this CL.
Revert this for now and will reland later.

Original issue's description:
> Avoid cross thread malloc/free pair of IOBuffer on the simple cache
>
> This CL removes a cross thread malloc/free pair from simple disk cache backend.
>
> After a PostTaskAndReply change in http://crrev.com/e5de8d551e80115b, it
> destroys |task| part of the given tasks on the target thread, and that
> introduced a number of cross thread malloc/free pairs around net::IOBuffer.
> The cross thread malloc/free pair increased the apparent size of memory
> usage on some Android perf bots by 2~3%, that is likely due to thread
> specific free list caches.
>
> BUG= 708644 
>
> Review-Url: https://codereview.chromium.org/2815563002
> Cr-Commit-Position: refs/heads/master@{#464338}
> Committed: https://chromium.googlesource.com/chromium/src/+/1714b61635cc8207647b1a9857e8042fbb833025

TBR=pmeenan@chromium.org,primiano@chromium.org,jkarlin@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 708644 

Review-Url: https://codereview.chromium.org/2814743011
Cr-Commit-Position: refs/heads/master@{#464551}

[modify] https://crrev.com/9fcfc8861ae30d5f1f5acf2d4cc0237f7f4e4f59/net/disk_cache/simple/simple_entry_impl.cc
[modify] https://crrev.com/9fcfc8861ae30d5f1f5acf2d4cc0237f7f4e4f59/net/disk_cache/simple/simple_entry_impl.h

=== Auto-CCing suspected CL author tzik@chromium.org ===

Hi tzik@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : tzik
  Commit : 1714b61635cc8207647b1a9857e8042fbb833025
  Date   : Thu Apr 13 08:12:10 2017
  Subject: Avoid cross thread malloc/free pair of IOBuffer on the simple cache

Bisect Details
  Configuration: android_nexus5X_perf_bisect
  Benchmark    : system_health.memory_mobile
  Metric       : memory:chrome:all_processes:reported_by_os:system_memory:native_heap:proportional_resident_size_avg/load_news/load_news_flipboard
  Change       : 2.26% | 65857330.6667 -> 64370141.3333

Revision             Result                  N
chromium@464310      65857331 +- 402202      6      good
chromium@464325      65895219 +- 665772      6      good
chromium@464333      65958707 +- 326309      6      good
chromium@464337      65851187 +- 319525      6      good
chromium@464338      64433629 +- 485368      6      bad       <--
chromium@464339      64363144 +- 493528      6      bad
chromium@464340      64259720 +- 192267      6      bad
chromium@464369      64370141 +- 339281      6      bad

Please refer to the following doc on diagnosing memory regressions:
  https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.news.flipboard system_health.memory_mobile

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982402957999103792

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6464280926355456


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

=== Auto-CCing suspected CL author mattm@chromium.org ===

Hi mattm@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : mattm
  Commit : 0704d236ee706ffb29fc22fd90f99537e7f15673
  Date   : Thu Apr 13 02:54:23 2017
  Subject: Convert android to use X509CertificateBytes instead of X509CertificateOpenSSL.

Bisect Details
  Configuration: android_nexus6_perf_bisect
  Benchmark    : system_health.memory_mobile
  Metric       : memory:chrome:all_processes:reported_by_chrome:malloc:effective_size_avg/load_social/load_social_tumblr
  Change       : 34.69% | 49162550.0 -> 32107664.0

Revision             Result                   N
chromium@464225      49162550 +- 1371913      6      good
chromium@464251      48721189 +- 787285       6      good
chromium@464258      48928483 +- 1674364      6      good
chromium@464261      48403439 +- 1450478      6      good
chromium@464262      32456487 +- 831833       6      bad       <--
chromium@464263      32095455 +- 416429       6      bad
chromium@464264      32461337 +- 1168806      6      bad
chromium@464277      32233399 +- 767296       6      bad
chromium@464329      32107664 +- 550282       6      bad

Please refer to the following doc on diagnosing memory regressions:
  https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.social.tumblr system_health.memory_mobile

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982402988639663088

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6671764622409728


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

I don't know how to interpret this.

1. The bug was filed on Apr 5 but my CL landed on Apr 13th. If anything, shouldn't it be a separate bug?

2. It looks like the numbers are much better after my CL, or am I just reading that wrong? Is this one of those perf tests that reports any change as a regression even if it actually got better?

mattm: Right, there's nothing to do for you about this issue. I scheduled the bisect job just to identify the significant improvement around this. Sorry for confusion.
However, as your change reduced the memory usage significantly, I recommend you to shout out it as your achievement.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/248782b9cda664a1ea6d347dd508354d23160a5a

commit 248782b9cda664a1ea6d347dd508354d23160a5a
Author: tzik <tzik@chromium.org>
Date: Fri Apr 14 04:03:27 2017

Avoid cross thread malloc/free pair of IOBuffer on the simple cache

This CL removes a cross thread malloc/free pair from simple disk cache backend.

After a PostTaskAndReply change in http://crrev.com/e5de8d551e80115b, it
destroys |task| part of the given tasks on the target thread, and that
introduced a number of cross thread malloc/free pairs around net::IOBuffer.
The cross thread malloc/free pair increased the apparent size of memory
usage on some Android perf bots by 2~3%, that is likely due to thread
specific free list caches.

BUG= 708644 

Review-Url: https://codereview.chromium.org/2815563002
Cr-Original-Commit-Position: refs/heads/master@{#464338}
Committed: https://chromium.googlesource.com/chromium/src/+/1714b61635cc8207647b1a9857e8042fbb833025
Review-Url: https://codereview.chromium.org/2815563002
Cr-Commit-Position: refs/heads/master@{#464673}

[modify] https://crrev.com/248782b9cda664a1ea6d347dd508354d23160a5a/net/disk_cache/simple/simple_entry_impl.cc
[modify] https://crrev.com/248782b9cda664a1ea6d347dd508354d23160a5a/net/disk_cache/simple/simple_entry_impl.h

 Issue 713112  has been merged into this issue.

 Issue 713099  has been merged into this issue.

Issue 713615 has been merged into this issue.