All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=636365

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg1ru9qAoM


Bot(s) for this bug's original alert(s):

android-nexus6

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9004709819624244128

=== Auto-CCing suspected CL author toyoshim@chromium.org ===

Hi toyoshim@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Enable WebFonts intervention v2 for slow2g on variation test waterfall
Author  : toyoshim
Commit description:
  
BUG= 578029 

Review-Url: https://codereview.chromium.org/2223753002
Cr-Commit-Position: refs/heads/master@{#410688}
Commit  : 45aa42162054faf440c1007b24abc03c83fcfa6b
Date    : Tue Aug 09 16:00:21 2016


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N  Good?
chromium@410678  12755.2  124.817  5  good
chromium@410683  12713.6  95.5238  5  good
chromium@410686  12783.2  92.2345  5  good
chromium@410687  12716.0  65.7875  5  good
chromium@410688  13341.6  15.1261  5  bad    <--
chromium@410698  13296.8  47.2779  5  bad
chromium@410717  13255.2  97.7098  5  bad

Bisect job ran on: android_nexus6_perf_bisect
Bug ID: 636365

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.datachannel
Test Metric: vm_private_dirty_final_renderer/vm_private_dirty_final_renderer
Relative Change: 3.92%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus6_perf_bisect/builds/2455
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9004709819624244128


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6406113486962688

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

 Issue 636364  has been merged into this issue.

My change was just for enabling field trial flag on variation tests, and should not affect perf tests, IIUC. But it seems multiple bisects suspect my CL with clear differences.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9004281741389356320

Kicked one more bisect.
cc: tbansal just in case if variation test flags affect perfbots.

Do you think NQE code could work differently if WebFonts study flag is enabled? I think NQE works isolated from WebFonts specific flags, but I'm not sure if calling NQE code from WebFonts could affect performance.

===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N  Good?
chromium@410683  9828.8   146.134  5  good
chromium@410688  10220.8  59.3565  5  bad
chromium@410692  10659.2  517.435  5  bad
chromium@410700  10173.6  65.8847  5  bad
chromium@410716  10191.2  73.8864  5  bad

Bisect job ran on: android_nexus5_perf_bisect
Bug ID: 636365

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.getusermedia
Test Metric: vm_private_dirty_final_renderer/vm_private_dirty_final_renderer
Relative Change: 3.69%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/3989
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9004281741389356320


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5878291718733824

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9004274309737607072

===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N  Good?
chromium@410683  9828.8   146.134  5  good
chromium@410688  10220.8  59.3565  5  bad
chromium@410692  10659.2  517.435  5  bad
chromium@410700  10173.6  65.8847  5  bad
chromium@410716  10191.2  73.8864  5  bad

Bisect job ran on: android_nexus5_perf_bisect
Bug ID: 636365

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.getusermedia
Test Metric: vm_private_dirty_final_renderer/vm_private_dirty_final_renderer
Relative Change: 3.69%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/3989
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9004281741389356320


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5878291718733824

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

The last bisect still reported in the same way, but I noticed that the result still have a range, 410684:410688.

https://chromium.googlesource.com/chromium/src/+log/59243b85c2d5bb5f049632a30fc1ebcc9663d016..45aa42162054faf440c1007b24abc03c83fcfa6b


This auto roll contains a change for WebRTC; https://chromium.googlesource.com/chromium/src/+/f4d2f3ddc2f9212514e7c3b4e6630890657f4688

https://chromium.googlesource.com/chromium/tools/build/+/0094839063efd2ef7978dfa83567c083a6dcf53b


phajdan.jr@ can you check if this roll can affect WebRTC performance?

tbansal, I heard that perf bots run with variation setteings. So my CL is still one of possible root causes. Could you answer my question at c#7?

Re#7: NQE code is independent of WebFonts study since NQE pushes data to Blink regardless of whether the WebFonts study is enabled or not.

Thanks. So this shouldn't be related to my CL. Anyway, it's also curious that only WebRTC tests have regressions by my change.

phajdan.jr: Please investigate this regression from your side.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9004181375550364352

===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Enable WebFonts intervention v2 for slow2g on variation test waterfall
Author  : toyoshim
Commit description:
  
BUG= 578029 

Review-Url: https://codereview.chromium.org/2223753002
Cr-Commit-Position: refs/heads/master@{#410688}
Commit  : 45aa42162054faf440c1007b24abc03c83fcfa6b
Date    : Tue Aug 09 16:00:21 2016


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N  Good?
chromium@410683  9855.33  74.2931  6  good
chromium@410686  9943.5   142.395  8  good
chromium@410687  9901.0   134.611  8  good
chromium@410688  10160.0  64.4981  8  bad    <--
chromium@410692  10277.6  205.535  5  bad
chromium@410700  10120.8  65.6902  5  bad
chromium@410716  10192.8  69.3628  5  bad

Bisect job ran on: android_nexus5_perf_bisect
Bug ID: 636365

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.getusermedia
Test Metric: vm_private_dirty_final_renderer/vm_private_dirty_final_renderer
Relative Change: 3.09%
Score: 99.5

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5_perf_bisect/builds/3990
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9004281741389356320


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5878291718733824

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Red herring. Rolling build into src does not affect production yet.

The last bisect suspect my CL.
I will revert the configuration for the investigation, and see what happens.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/490513e0b74a3cc3e60a0719a30333e29768f34a

commit 490513e0b74a3cc3e60a0719a30333e29768f34a
Author: toyoshim <toyoshim@chromium.org>
Date: Fri Aug 19 05:02:26 2016

WebFonts intervention v2: disable the feature on Android perf test

Bisect suspected this CL as a culprit of a WebRTC tests performance
regression. Let me revert this temporarily and see if the problem
can be fixed by reverting this.

BUG= 636365 

Review-Url: https://codereview.chromium.org/2256153002
Cr-Commit-Position: refs/heads/master@{#413053}

[modify] https://crrev.com/490513e0b74a3cc3e60a0719a30333e29768f34a/testing/variations/fieldtrial_testing_config_android.json

Hum..., this revert works.
I'd check the test itself why this test is affected by my CL.

So, I downloaded .wpr file used by this test:
third_party/catapult/third_party/gsutil/gsutil cp gs://chromium-telemetry/10ce827228dbf11155df76f1623e5d4d74846dc9 webrtc.wpr

and, then I decoded it:
./httparchive.py ls ~/webrtc.wpr (WPR tool from https://github.com/chromium/web-page-replay)

It seems that this test is downloading 3 web fonts. So, that explains the correlation in the regression with the CL.

Thanks tbansal,

The test uses this page http://webrtc.github.io/samples/src/content/datachannel/datatransfer/.

What the test results indicate is that WebFont intervention is triggered in the test, and using fallback font and layout recalculation increases memory usage. This scale of memory usage change(+200kB against 10MB) could happen in such case and would be seen only in micro benchmarks since usually such system font is already used in the renderer.

Since other tests didn't rise any similar alert for this change, I want to close this as WontFix.


tbansal: Which connection type is expected to be reported in such benchmark that does cold startup?
On Linux, following step reports Type4G.

$ rm ~/.config/chromium; ./out/Release/chrome http://webrtc.github.io/samples/src/content/datachannel/datatransfer/

But after I close the chromium without doing anything, and launch it with the same URL reports TypeUnknown.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5576960bd7f6b5e39720a65cdf57cf50c62e64f5

commit 5576960bd7f6b5e39720a65cdf57cf50c62e64f5
Author: toyoshim <toyoshim@chromium.org>
Date: Tue Aug 23 11:32:10 2016

Revert of WebFonts intervention v2: disable the feature on Android perf test (patchset #1 id:1 of https://codereview.chromium.org/2256153002/ )

Reason for revert:
Investigation was finished.

Original issue's description:
> WebFonts intervention v2: disable the feature on Android perf test
>
> Bisect suspected this CL as a culprit of a WebRTC tests performance
> regression. Let me revert this temporarily and see if the problem
> can be fixed by reverting this.
>
> BUG= 636365 
>
> Committed: https://crrev.com/490513e0b74a3cc3e60a0719a30333e29768f34a
> Cr-Commit-Position: refs/heads/master@{#413053}

TBR=isherman@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 636365 

Review-Url: https://codereview.chromium.org/2273633002
Cr-Commit-Position: refs/heads/master@{#413713}

[modify] https://crrev.com/5576960bd7f6b5e39720a65cdf57cf50c62e64f5/testing/variations/fieldtrial_testing_config_android.json

Re#24: The ECT depends on the actual performance of the connection of the testbots. If the connection is too slow (or artificially throttled), then ECT could be Slow2G or 2G.

How about the first page load from the clean profile? At this point, NQE should not have data to estimate the ECT from actual performance, but Type4G was reported in my case.