FWIW, my /etc/fonts/fonts.conf says:

	DO NOT EDIT THIS FILE.
	IT WILL BE REPLACED WHEN FONTCONFIG IS UPDATED.
	LOCAL CHANGES BELONG IN 'local.conf'.

Therefore, I don't know how to follow the suggestion from the error message (blank doesn't take any effect anymore. please remove it from your fonts.conf).

Let me also note that the "blank doesn't take any effect anymore" error doesn't occur before r524389.

I'm guessing that r524389 didn't run content_browsertests (and possibly other tests). This seems like a error in the configuration/scripts that select what tests are being run.

Seems similar to  bug 795662  which is about big delay when starting layout test. The delay was caused by full reconstruction of font cache in the temporary home for running layout tests. The fix is to create a symbolic link under the temporary home to .cache under the real home. Does browser tests use custom home directory?

The fontconfig warning seems unrelated to the delay but probably indicates that the statically linked fontconfig is not fully compatible (?) with the the system.

Before r524389, why didn't we fully reconstruct the font cache with the temporary home?

I've touched base with thomasanderson@ and I am attempting to revert in https://chromium-review.googlesource.com/c/chromium/src/+/834869 (I may need to work on resolving merge conflicts though).

@wjmaclean - I'm not sure why you think that r524389 didn't run content_browsertests? It looks like it ran a lot of things on linux_chromium_rel_ng:

https://chromium-review.googlesource.com/c/chromium/src/+/826403
https://ci.chromium.org/buildbot/tryserver.chromium.linux/linux_chromium_rel_ng/609440

Perhaps I didn't understand the output properly (perhaps I was looking in the wrong place) ... thought if they did run, it seems curious that the bisect picked out this CL.

c#7: Does locally reverting the CL fix the timeouts?

I tested at
- git checkout 4f1a60552748d150783655ee6f7b6f7084e3af95^ # the problem didn't repro
- git checkout 4f1a60552748d150783655ee6f7b6f7084e3af95  # the problem reproed

I haven't yet tested if the problem repros after the revert.  Let me quickly try that.

I've verified locally that the timeout doesn't happen after the revert (after the local version of the revert CL that is here: https://chromium-review.googlesource.com/c/chromium/src/+/834869).

RE: #c6-c7

Indeed - it seems that content_browsertests and browser_tests passed on the trybots.  I wonder:

- How widespread the timeouts are (it seems that me, kenrb@, wjmaclean@, wangxianzhu@ could repro the slowness;  I assume that thomasanderson@ didn't repro the slowness when working on r524389).

- What is different between my local test environment (where the timeouts affect all browser tests) and the test environment on the bots (or on machines of people who don't repro the problem).

If this is fontconfig related, I expect you're hitting differences due to the configuration of the corp image, that the bots wouldn't see. 

We've certainly seen that in the past w/ the layout tests, for example.

I have also confirmed that reverting that CL locally fixes the problem. I had to do so in order to work on a browser test that I am trying to write.

Thanks for verifying the revert fixes the timeouts.

dpranke@ does the browser test environment set up it's own fonts.conf?  If not, I think it should be.

> dpranke@ does the browser test environment set up it's own fonts.conf? 

I don't think so, but I don't remember for sure.

> If not, I think it should be.

Yes, you are probably right.

Affected devs: Could you post your fonts.conf ?
* /etc/fonts/fonts.conf
* ~/.fonts.conf
* ~/.config/fontconfig/fonts.conf

Attaching my /etc/fonts/fonts.conf.  I don't have ~/.fonts.conf or ~/.config/fontconfig/fonts.conf.  FWIW, I don't think I've ever tweaked any font configs on my machine - I think I have a plain vanilla Goobuntu desktop (trusty desktop stable "14.04 201712TD2-2").

Deleted: lukasza-etc-fonts-fonts.conf 5.4 KB

lukasza-etc-fonts-fonts.conf 5.4 KB Download

Hm.. I can't repro on Ubuntu Trusty with your fonts.conf.  The tests pass after just a few seconds.  I'm running
$ testing/xvfb.py out/Release/content_browsertests --gtest_filter=IsolatedDevToolsProtocolTest.ControlNavigationsChildFrames

Could you try temporarily clearing your fonts.conf and seeing if the tests pass?

I think you are seeing the same problem as the layout test delay during which fontconfig is reinitializing the font cache because there was no existing one in the test isolated environment. In my environment, the most slow font is the noto CJK font, so if a fonts.conf doesn't reference the CJK font, there font cache reinitialization will be probably fast.

A question: why is the statically linked fontconfig much slower than the original one? Did the original one initialize font cache much faster, or did it just skip the step?

The test still times out after:
$ sudo mv /etc/fonts/fonts.conf /etc/fonts/saved-fonts.conf-blah

Two things that might affect the repro:

1. Please note that (as shown in the repro above) I run my tests with DISPLAY=:20 - in the Chrome Remote Desktop (to avoid browser windows from popping up on my regular desktop + this is a habit that makes me able to run browser tests via ssh session):

$ DISPLAY=:20 out/dbg/content_browsertests --gtest_filter=IsolatedDevToolsProtocolTest.ControlNavigationsChildFrames

2. Please note that I am building the tests in a debug configuration which might potentially be slower than a release build:

$ cat out/dbg/args.gn 
is_component_build = true
is_debug = true
use_goma = true

BTW: The revert has landed in r525173.

Can you repro with "testing/xfvb.py out/dbg/..." instead of "DISPLAY=:20 out/dbg/..."?

Also could you attach your /etc/fonts/local.conf

Actually, could you attach a zip of your entire /etc/fonts dir

It seems that the timeout only repros when running browser tests in the Chrome Remote Desktop session (when using DISPLAY=:20).

I cannot repro the timeout with:

$ testing/xvfb.py out/dbg/content_browsertests --gtest_filter=IsolatedDevToolsProtocolTest.ControlNavigationsChildFrames

and cannot repro with:

$ out/dbg/content_browsertests --gtest_filter=IsolatedDevToolsProtocolTest.ControlNavigationsChildFrames # i.e. in the default :0 X session

Attaching my /etc/fonts/local.conf

Deleted: lukasza-etc-fonts-local.conf 1.2 KB

lukasza-etc-fonts-local.conf 1.2 KB Download

Actually, I am not sure how much I trust my statements in #c24, because right now I cannot repro the timeout at 4f1a60552748d150783655ee6f7b6f7084e3af95.  Some random hypothesis:
1. The timeout went away because my machine/CRD is not busy anymore?
2. The timeout went away because the persistent state of my machine changed when I run testing/xvfb.py (in #c24) or when I moved+moved-back /etc/fonts/fonts.conf (in #c20).

No way to repro is a problem, because it means that we cannot verify the timeout won't occur anymore after a reland...  I hope that kenrb@ and/or wjmaclean@ can still repro though...

RE: #c23

I've attached my zipped /etc/fonts dir.

Deleted: lukasza-etc-fonts-dir.zip 100 KB

lukasza-etc-fonts-dir.zip 100 KB Download

There's not really anything actionable that I can do to debug why this caused the tests to hang, because I can't reproduce the issue (I tried using an Ubuntu 14.04 VM with the provided /etc/fonts, still no luck).

I think the best way to debug this is to reland, and try to work with someone who's affected by this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/65f919d2d86e54c851b0327740a330d152b07ed5

commit 65f919d2d86e54c851b0327740a330d152b07ed5
Author: Tom Anderson <thomasanderson@chromium.org>
Date: Fri Dec 22 06:20:09 2017

Reland "Statically link fontconfig on Linux"

This reverts commit 5a8428d621c9e9c8f5bcfa9ca97ae66002207d8c

Reason for revert: See comment https://bugs.chromium.org/p/chromium/issues/detail?id=796292#c28

> This reverts commit 4f1a60552748d150783655ee6f7b6f7084e3af95.
>
> Reason for revert: This CL caused time outs in browser tests - see  https://crbug.com/796292 
>
> Original change's description:
> > Statically link fontconfig on Linux
> >
> > This CL switches Linux to use in-tree fontconfig builds.  It exposes some leaks
> > and races detected by LSAN and TSAN.  For now, these will be suppressed since
> > these issues were preexisting, but instrumenting fontconfig exposed the issues.
> >
> > R=​dnicoara@chromium.org,dpranke@chromium.org
> > BUG=795110, 795148 
> >
> > Change-Id: Ia75db4ced6ec78a5f0610af9ebc78a87840b86f7
> > Reviewed-on: https://chromium-review.googlesource.com/826403
> > Commit-Queue: Daniel Nicoara <dnicoara@chromium.org>
> > Reviewed-by: Daniel Nicoara <dnicoara@chromium.org>
> > Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#524389}
>
> TBR=dpranke@chromium.org,dnicoara@chromium.org,thomasanderson@chromium.org
>
> # Not skipping CQ checks because original CL landed > 1 day ago.
>
> Bug: 795110,  795148 
> Change-Id: Id224fdcbfb0ca3373f6219b66252a8970072675b
> Reviewed-on: https://chromium-review.googlesource.com/834869
> Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
> Reviewed-by: Łukasz Anforowicz <lukasza@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Commit-Queue: Xianzhu Wang <wangxianzhu@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#525173}

R=dpranke@chromium.org
TBR=dnicoara@chromium.org
BUG=795110, 795148 , 796292 

Change-Id: I47a2225cb9cc255721b521cbb790d9eb70e0b3b9
Reviewed-on: https://chromium-review.googlesource.com/841627
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Thomas Anderson <thomasanderson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#525945}
[add] https://crrev.com/65f919d2d86e54c851b0327740a330d152b07ed5/build/linux/unbundle/fontconfig.gn
[modify] https://crrev.com/65f919d2d86e54c851b0327740a330d152b07ed5/build/linux/unbundle/replace_gn_files.py
[modify] https://crrev.com/65f919d2d86e54c851b0327740a330d152b07ed5/build/sanitizers/lsan_suppressions.cc
[modify] https://crrev.com/65f919d2d86e54c851b0327740a330d152b07ed5/build/sanitizers/tsan_suppressions.cc
[modify] https://crrev.com/65f919d2d86e54c851b0327740a330d152b07ed5/chrome/installer/linux/debian/dist_package_versions.json
[modify] https://crrev.com/65f919d2d86e54c851b0327740a330d152b07ed5/chrome/installer/linux/debian/update_dist_package_versions.py
[modify] https://crrev.com/65f919d2d86e54c851b0327740a330d152b07ed5/chrome/installer/linux/rpm/dist_package_provides.json
[modify] https://crrev.com/65f919d2d86e54c851b0327740a330d152b07ed5/chrome/installer/linux/rpm/update_package_provides.py
[modify] https://crrev.com/65f919d2d86e54c851b0327740a330d152b07ed5/third_party/WebKit/LayoutTests/platform/linux/fast/text/unicode-fallback-font-expected.png
[modify] https://crrev.com/65f919d2d86e54c851b0327740a330d152b07ed5/third_party/fontconfig/BUILD.gn
[modify] https://crrev.com/65f919d2d86e54c851b0327740a330d152b07ed5/third_party/fontconfig/fontconfig.gni

When I launch Chrome or tests on trunk I see the error message again:
Fontconfig warning: "/etc/fonts/fonts.conf", line 146: blank doesn't take any effect anymore. please remove it from your fonts.conf

However, local browser tests now seem to be running successfully, whereas they were not with the original CL. I have no idea what might have changed.

Thanks kenrb@.  There was a harfbuzz roll which I suspect had something to do with it (There was a performance graph somewhere which showed a regression on the fontconfig roll and a speedup on the hb roll).

I think the warning is mostly harmless.  If we really wanted, we could suppress it in our bundled fontconfig.

If anyone is still seeing the timeouts, plmk.

Haven't seen any reports of tests timing out after the reland, so I'm closing this issue.  PLMK if you're still seeing any timeouts.