All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=724095

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg6pO24gsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg2tCCrAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgmq2_sAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgmpmluAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg2vjKogoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgmtfIpwsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg2pDTuQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg2tCYtQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgmrrdvwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg2tzwqAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgmqSwuAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg2pDPpAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg2pClqAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgmqCqpgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgmtrVogoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg6u2H0QgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgmqnzvQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgmqDetwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg6rSk4wkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg6uqL3AgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg6rif0wkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg6qLM_QkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg6tiX6AsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgqprxpAsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgquew2QoM


Bot(s) for this bug's original alert(s):

android-webview-nexus6

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8979249388736456896

=== Auto-CCing suspected CL author drott@chromium.org ===

Hi drott@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : drott
  Commit : fbec38011870753cffb3cb2ff98aa0c00c63b25c
  Date   : Wed May 17 10:49:03 2017
  Subject: Reland: Compile FreeType with HarfBuzz support

Bisect Details
  Configuration: android_webview_nexus6_aosp_perf_bisect
  Benchmark    : page_cycler_v2.intl_ar_fa_he
  Metric       : timeToFirstContentfulPaint_avg/pcv1-cold/http___www.google.com.sa_
  Change       : 134.62% | 151.635333333 -> 355.765

Revision             Result                  N
chromium@472411      151.635 +- 46.16        6      good
chromium@472412      142.856 +- 26.604       6      good
chromium@472413      357.772 +- 18.4235      6      bad       <--
chromium@472415      346.439 +- 34.4652      6      bad
chromium@472418      340.528 +- 21.9614      6      bad
chromium@472424      355.765 +- 55.0592      6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http...www.google.com.sa. page_cycler_v2.intl_ar_fa_he

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8979249388736456896

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5870336794427392


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

 Issue 724097  has been merged into this issue.

 Issue 724121  has been merged into this issue.

 Issue 724623  has been merged into this issue.

 Issue 724344  has been merged into this issue.

 Issue 724478  has been merged into this issue.

 Issue 724078  has been merged into this issue.

 Issue 725171  has been merged into this issue.

This looks like it's a significant performance hit on Android indeed. Building FreeType with HarfBuzz was a fix for  issue 617168  which was filed for Linux, so we have two options:
1) Disabling FreeType-with-HarfBuzz support on Android
2) Disabling autohinting on Android altogether. As one data point, Skia does that when built for the Android platform.

It would be helpful to know what percentage of low resolution Android phones there are and which are running Chrome, in order to decide whether switching off autohinting would be a helpful solution. For now, I think it makes sense to go with 1).

Ben, do you have more background on what's the reasoning behind Skia's compile flag which disables autohinting on Android on the OS level?

Note that AFAIU the change only affects initial font load time in FreeType, NOT each rendering.  Each rendering will be affected to the extent that previously it was NOT being autohinted (wrong) but now is (right), but that cannot be measurably high.  So I wonder if the test case is hitting cache trashing or is otherwise non-realistic.

If we figure out that it is indeed making font loading significantly slower, I can look into ideas to speed it up (I have one at least).

The graphs are quite clear that a wide range of page load times etc. are affected. For now I propose to disable this on Android in https://codereview.chromium.org/2896203002 while we can try to figure out why the effect is so broad.

Looking at the graphs linked from  issue 724623 , it's clear that timeToFirstMeaningfulPaint on Linux is affected as well by over 30% in many cases. :( This would still be inline with initial load times suffering. 

As the instructions in that bug explain, we can run this test on linux release with 
$ src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=2ch loading.desktop

On Linux, this fixes a correctness issue, which is  issue 617168 , that's why I am hesitant to disable it there for now.

Correcting my comment #17: Not page load times, but mostly timeToFirstMeaningfulPaint is affected, which is a metric for early parts of the page starting to appear on the screen, not a fully loaded page.

According to internal data this appears to be relevant for low resolution Android devices, so we would indeed benefit from knowing why this has such a high impact on timeToFirstMeaningfulPaint benchmarks.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ba0e6cd12bf6f5e864e1f60993ceafef3f4a7cae

commit ba0e6cd12bf6f5e864e1f60993ceafef3f4a7cae
Author: drott <drott@chromium.org>
Date: Wed May 24 09:08:34 2017

Do not build FreeType with HarfBuzz support on Android

Originally, building FreeType with HarfBuzz, helped improve autohinting
on Linux, see  issue 617168 . However, the performance impact we observe
on Android does not seem to justify running autohinting with HarfBuzz
support on Android.

BUG= 722980 ,  724095 

Review-Url: https://codereview.chromium.org/2896203002
Cr-Commit-Position: refs/heads/master@{#474221}

[modify] https://crrev.com/ba0e6cd12bf6f5e864e1f60993ceafef3f4a7cae/third_party/freetype/BUILD.gn
[modify] https://crrev.com/ba0e6cd12bf6f5e864e1f60993ceafef3f4a7cae/third_party/freetype/include/freetype-custom-config/ftoption.h

 Issue 726806  has been merged into this issue.

Fixed for Android after ba0e6cd12bf6f5e864e1f6. For Linux, I would mark this WontFix: I am not disabling HarfBuzz support in FreeType on Linux for now, since it is a fix for correctly autohinting ligatures, see  issue 617168 .