All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=694565

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDguMeo5QoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg-LbBhggM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg-JDXrQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg-ISf9QgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg-OemqgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg-MCJoAsM


Bot(s) for this bug's original alert(s):

android-nexus5
android-nexus7v2

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8987039663304415392

=== Auto-CCing suspected CL author bcwhite@chromium.org ===

Hi bcwhite@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : bcwhite
  Commit : 8cf7e79b5bb95b8108861d8d7b910afdbc1db94d
  Date   : Thu Feb 16 20:44:17 2017
  Subject: Move to MappedFile as default for persistent histograms.

Bisect Details
  Configuration: android_nexus5_perf_bisect
  Benchmark    : smoothness.top_25_smooth
  Metric       : frame_times/Wordpress
  Change       : 10.35% | 16.800826564 -> 18.539352979

Revision             Result                   N
chromium@451036      16.8008 +- 0.137208      6      good
chromium@451081      16.8022 +- 0.222845      6      good
chromium@451092      16.782 +- 0.132696       6      good
chromium@451093      19.3348 +- 0.670567      6      bad       <--
chromium@451094      18.3094 +- 0.64244       6      bad
chromium@451095      18.1536 +- 0.564068      6      bad
chromium@451098      18.8054 +- 2.17494       9      bad
chromium@451103      18.5725 +- 2.86115       9      bad
chromium@451125      18.5394 +- 1.08378       6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=Wordpress smoothness.top_25_smooth

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8987039663304415392

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6185138364350464


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Alexei, what do you make of this?

The stats are from the renderer but there are no differences between InMemory and OnDisk with regards to subprocesses.  The only difference would be when the process dies and its metrics get merged with that of the browser since the browser's versions are now stored in a memory-mapped file.

So the only thing I can think of is that the tests are starting/stopping a lot of processes (gpu? renderer?) which dirty the pages of the browser metrics which then get flushed to disk at some point by the OS.

Any other ideas?

That explanation seems plausible - it is testing top 25 sites - so I could imagine that it would be starting multiple renderers.

Looks like there's more info here about the benchmarks:
http://www.chromium.org/developers/design-documents/rendering-benchmarks

Adding nduca and ernstm to cc since they're listed as owners in that doc.

nduca@ and ernstm@ - what's the best way for Brian to debug and dig into what's actually slower here?

nduca & ernstm: Ping?

skyostil & brianderson:  Your names are on the scheduler.tough_scheduling_cases test.  Can you describe what it is measuring?  Is it entirely inside the renderer?

One of the metrics that regressed clearly (mean_frame_time) measures the end-to-end time of two successive graphics frames, indirectly including time spent in the renderer, browser and GPU processes. It also regressed on simple_text_page.html, so I don't think the process starting theory holds water.

Looking at the before[1] and after[2] traces there's now a 200ms+ pause during rendering. I think that is causing the regression and probably looks pretty bad to the user too. Any idea where it's coming from?

[1] https://00e9e64bac86ce1313e1c3eb6931e33f2db12930fdcfaec945-apidata.googleusercontent.com/download/storage/v1/b/chrome-telemetry-output/o/trace-file-id_8-2017-02-16_10-01-12-12425.html?qk=AD5uMEtbWMoI7hQQkBflBYiiOmj5imK40-wNYruqBJL_UZ26F8Rqx-2aeTl0QBK6s-E-bf7vp022lCT7orQJJuN5y1NiqDdTE2VaY1AlflEyw23t4KZODarnm2V5O_BwUSJ1AmZJTgPyFYDOFvK3ticyb88hDhDNNdGezHzy40DKZ2Rx2x_70z3MEYmxWb2cv6hszrrG4WO0MwiYWGAc4rYTs6TKL60sTJKt3YNSYvpapkGKxJOIT9oe1rVnPFrLG1MZ5p9NMyKNj9HZahzl6OF6PWNQKvSqHDA6c2uWV5Y2RGl94K7HK6Qbl_PEDwkgEZFztHzCn09_gFLKQ05xCiWIct0a8YKZEnxPTiRnNeOTDHnbQaL6kvdzWMTrvJL9LZI_O8mcsxIfJ7M_KeBeBZfbDw2k43UuSMYqykHplnpFBn3zagpao5YDFkXJfDKGni1vFugQilGRsVOMx0oZrhahxZvHV_TY36Eee4k4uSIGDhXe2dEwaAreXbu2K8aPyFbnX-pH8uTAbziojhY-9zY6LnxuoFm-Ceu_uZAiKnJ4GjEG-D57VvoJlkAltvhu73f0ooWcY3mZA6HnfKuoJwtg4YgppaFHFMQ2pL3TafuZ1i13cGnsRyMoJRl_augKIi6QHAm17zlwxe3XTHh5s-NOhAhs0e8BKezhTpMXW0myXwNdi_XQjej6xMdPBWsfDw05vVLyPm73nLW9xjqtpVJULdhrAqX6S6E0qdFLhGhVCMhPo0UasH57jvFYedBL8KAIPYul3sEkHmC8FaDp65b55krr8mwnlzrrn8fEHJAmc9rRDnZmrwoBXKi23ZJakh61oxfJ6LXZqkGfhwk9S6hjhfqbBeJcvA

[2] https://00e9e64bacb64a6acd1eac06ce95d2a868039aab6e3b2581a9-apidata.googleusercontent.com/download/storage/v1/b/chrome-telemetry-output/o/trace-file-id_8-2017-02-16_15-57-58-80344.html?qk=AD5uMEsnHQ0ajlKfyfFLFR5spyoNYSs0Q9VRS_TtFlHDyp3c_SiqnhNVte54CRV3uDguYTddE3FO_B26Kxxkmbw44Y3IPll1qSYMhXU4-jPFweXZMSQaKAQsq_uu4G0BFC563zOhchbKa5jU3dsyZjI0L8gXY__SZAsP6G0Bzm_p9XRa2StfwD7K82w8V5UgGfJRcqMLIlve0Q8JIZV7LYa_xcsMmJ14cY8IGQw2PIKpiWiI-8kHLqkviYDuxR6ld4yIMrKvz1nVy8Jon6rx6n6Ve74TyftIBTRVGRqB49UYMyvEkImJDXM47Op-we1Spfx4TOmBBDPNCGUGKbX1zeC5AXaspXWt_vLt9tcO_BK0rTL7WMcDgo161YiI-mzxgDwey-YUcWmbPXvKRdNXxf6361bljBMQWLdGNlfsvPusG10XVMhsShrNPdylRAL1oS-S2BUUd_FvuR74q3QOapBNcKU5-Wm0ICPioRBKvRG--ml_gJQdo0dfVvS9jAoA3vXnnOJyWc6Qkea8Qqb-o4u3CfjbN4qbNq68RNEeYwBeX9TaJILBTCle66rFtPEn6ff_55Ls1Dr0f-EgkvenwoP0FzXzGeSifeTk7thluyd7Wqw43PMjMj5OfJfZYmAQEZB0wgAZBMxs0Tf5mspAb6nDzAtNYt2Oxu7QCee0llCq53hFqvg9HbyOy2zGE2xfLxWrekSiufkZiCxpXAjeEiyrvugpQoGDgdLfUYGWoZyWyNRYyhcVPxGmFAspsUmcbIj0sCF8tRzPQ2M2UHLxoqPo4JUsTJoJv4b-GOMWj6FYcjR28wbMFZQnclsOUpJ918wJ_lMbhL10RhUBJBSOwacNkkXfRs7KBg

I have a Windows machine.  How can I run this test locally to test if my change is causing the issue?

You can't run locally, but you could use https://www.chromium.org/developers/telemetry/performance-try-bots

Question regarding these tests:

Exactly how are the times calculated?
- Is each render operation timed independently, then averaged?
- Is the full process run-time divided by number of tests?
- Something else?

+nednguyen@ who I think has been auditing the various perf bot tests and might be able to answer the questions in comment 12

The code that compute the rendering metrics are in https://github.com/catapult-project/catapult/blob/master/telemetry/telemetry/web_perf/metrics/smoothness.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9963f7ea173756bc0b8014ed0dcd9bd48a6d3598

commit 9963f7ea173756bc0b8014ed0dcd9bd48a6d3598
Author: bcwhite <bcwhite@chromium.org>
Date: Wed May 24 20:52:36 2017

Added support for 'spare' file that can be used at startup.

Memory-mapped files make I/O errors into bus errors if something
is wrong with the file and cannot be mapped, including if the file
is sparse and the disk is full.  To counter this, such files are now
always pre-allocated to their full size but this has a cost.

To counter the cost, create a "spare" file some time after startup
that can be used for persistent metrics during the following run,
thus eliminating the cost of pre-allocating that file during browser
startup.

This is especially important on Android where the cost of allocating
the file is significant and yet persisting metrics is the most useful.

Performance tests indicate this reverses both android regressions
but it won't be until real metrics start being collected that an
accurate assessment can be made.

BUG=721806,  694565 

Review-Url: https://codereview.chromium.org/2888563005
Cr-Commit-Position: refs/heads/master@{#474414}

[modify] https://crrev.com/9963f7ea173756bc0b8014ed0dcd9bd48a6d3598/base/metrics/persistent_histogram_allocator.cc
[modify] https://crrev.com/9963f7ea173756bc0b8014ed0dcd9bd48a6d3598/base/metrics/persistent_histogram_allocator.h
[modify] https://crrev.com/9963f7ea173756bc0b8014ed0dcd9bd48a6d3598/base/metrics/persistent_histogram_allocator_unittest.cc
[modify] https://crrev.com/9963f7ea173756bc0b8014ed0dcd9bd48a6d3598/chrome/browser/chrome_browser_field_trials.cc
[modify] https://crrev.com/9963f7ea173756bc0b8014ed0dcd9bd48a6d3598/chrome/browser/metrics/chrome_metrics_service_client.cc
[modify] https://crrev.com/9963f7ea173756bc0b8014ed0dcd9bd48a6d3598/tools/metrics/histograms/enums.xml