Hi Jeff / Quinten / Joshua,

I was unsure who this should be assigned to. It's a CSS related test which is failing, but it also a w3c imported test.

How do we handle flakiness for these tests?

Tim 'mithro' Ansell

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7f6264df1bd3a73c0f8af49adfc345126eebb1cd

commit 7f6264df1bd3a73c0f8af49adfc345126eebb1cd
Author: tansell <tansell@chromium.org>
Date: Mon May 08 04:56:11 2017

LayoutTests: Disable flaky tests on Windows.

These tests are failing a small amount of the time on the Windows Random
Order bots now they are running sharded on swarming. The tests affected
are;

 * fast/css-grid-layout/grid-self-baseline-two-dimensional.html
 * fast/css-grid-layout/grid-align-justify-stretch-with-orthogonal-flows.html
 * fast/dnd/dropEffect-for-effectAllowed.html
 * external/wpt/css/css-ui-3/outline-004.html

Bugs have been filed for each of the failing tests.

The most likely cause is that they depend on another test being run and
the sharding makes this no longer happening.

BUG= 717347 , 718717 , 719298 , 719299 
NOTRY=true

Review-Url: https://codereview.chromium.org/2872463002
Cr-Commit-Position: refs/heads/master@{#469907}

[modify] https://crrev.com/7f6264df1bd3a73c0f8af49adfc345126eebb1cd/third_party/WebKit/LayoutTests/TestExpectations

LayoutTests/W3CImportExpectations has:

## Owners: rego@igalia.com
# external/wpt/css/css-ui-3 [ Pass ]

It seems a very simple test for the "outline" property:
http://w3c-test.org/css/css-ui-3/outline-004.html

I wouldn't expect any flakiness there, or any relationship with previous tests.
So if it's flacky on Windows there might be a bug somewhere.

It'd be nice to see what's the actual failure and what's the output when it fails.
I cannot find the results on builds like:
https://build.chromium.org/p/chromium.fyi/builders/WebKit%20Win%20-%20RandomOrder/builds/3140

The results are missing there right now ( bug 717346 ), and it will be much easier to see the results once that's fixed, I think.

Removing Blink>CSS for now. Once blocking bug is fixed and it's a CSS issue, please add it back.

You can see the results manually via going to the failing shard and clicking the "Isolated Outputs" link.

Here is an example failure;
 * https://isolateserver.appspot.com/browse?namespace=default-gzip&hash=55d2060e28efc1f799f69b588a35b7dac15357d6
 * Expected - https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=38ae51b3ecd4cfea3b38298fbb70ea561bd261ca&as=layout-test-results%5Cexternal%5Cwpt%5Ccss%5Ccss-ui-3%5Coutline-004-expected.png
 * Actual - https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=7c91a2a0e9478c4045cffa938f7cd3c4a8e65a3c&as=layout-test-results%5Cexternal%5Cwpt%5Ccss%5Ccss-ui-3%5Coutline-004-actual.png
 * Diff - https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=5f287e27de3a0d96c182014f81121537e9be0624&as=layout-test-results%5Cexternal%5Cwpt%5Ccss%5Ccss-ui-3%5Coutline-004-diff.png
 * Actual (retry 1) - https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=7c91a2a0e9478c4045cffa938f7cd3c4a8e65a3c&as=layout-test-results%5Cretry_1%5Cexternal%5Cwpt%5Ccss%5Ccss-ui-3%5Coutline-004-actual.png
 * Actual (retry 2) - https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=7c91a2a0e9478c4045cffa938f7cd3c4a8e65a3c&as=layout-test-results%5Cretry_2%5Cexternal%5Cwpt%5Ccss%5Ccss-ui-3%5Coutline-004-actual.png
 * Actual (retry 3) - https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=7c91a2a0e9478c4045cffa938f7cd3c4a8e65a3c&as=layout-test-results%5Cretry_3%5Cexternal%5Cwpt%5Ccss%5Ccss-ui-3%5Coutline-004-actual.png

So it looks like in the run it failed 4 times in a row...

The rendered output in that image matches what my local Chrome (58.0.3029.96) renders when showing that file - file:///fast/chrome/src/third_party/WebKit/LayoutTests/external/wpt/css/css-ui-3/outline-004.html

Has the rendering of this test changed recently?

@tansell thank you very much for the links.

So the issue is the font, the rest of the test is working fine.
For some reason the "font-family: ahem;" is failing on Windows.
No idea of what's going on.

> The rendered output in that image matches what my local Chrome (58.0.3029.96) renders when showing that file - 
> file:///fast/chrome/src/third_party/WebKit/LayoutTests/external/wpt/css/css-ui-3/outline-004.html

I don't think what I'm going to say makes any sense,
but could you try with "font-family: Ahem"?
Theoretically font-family should be case insensitive [1],
but there are lots of tests with "Ahem" passing, so maybe it's related
(dunno really, just a wild idea).

> Has the rendering of this test changed recently?

I'm on Linux, but the output of that test is the same in M58 and M60.

[1] https://drafts.csswg.org/css-fonts-4/#font-family-casing

So, if the Ahem font is not being found, then this would cause the problem? That should cause a lot more tests to fail however?

There seems to be a *lot* of Ahem fonts in our repo....
----------------
./testing/libfuzzer/fuzzers/woff2_corpus/Ahem.woff2
./testing/libfuzzer/fuzzers/woff2_corpus/AhemSpaceLigature.woff2
./third_party/WebKit/LayoutTests/external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/flexbox/support/Ahem.ttf
./third_party/WebKit/LayoutTests/external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/support/Ahem.ttf
./third_party/WebKit/LayoutTests/external/wpt/css/fonts/ahem-extra/AHEM_Ahem!.TTF
./third_party/WebKit/LayoutTests/external/wpt/content-security-policy/support/Ahem.ttf
./third_party/WebKit/LayoutTests/fast/text/resources/AhemIVS.ttf
./third_party/WebKit/LayoutTests/resources/Ahem.ttf
./third_party/WebKit/LayoutTests/resources/Ahem.woff2
./third_party/WebKit/LayoutTests/resources/Ahem.otf
./third_party/WebKit/LayoutTests/resources/AhemNoXHeight.ttf
./third_party/WebKit/LayoutTests/resources/Ahem.woff
./third_party/WebKit/LayoutTests/http/tests/w3c/webperf/resources/Ahem.ttf
./third_party/WebKit/LayoutTests/http/tests/resources/Ahem.ttf
./third_party/WebKit/LayoutTests/third_party/AhemSpaceLigature
./third_party/WebKit/LayoutTests/third_party/AhemSpaceLigature/AhemSpaceLigature.woff
./third_party/WebKit/LayoutTests/inspector-protocol/css/resources/Ahem.ttf
./third_party/WebKit/Source/platform/testing/data/AhemSpaceLigature.woff
./third_party/WebKit/Source/platform/testing/data/Ahem.woff
./third_party/WebKit/Source/web/tests/data/Ahem.ttf
./third_party/sfntly/fuzzers/corpora/sfntly_fuzzer/Ahem.ttf
./headless/test/data/Ahem.ttf
----------------
1117a07f6f8a38b42e1e3740935a5482  ./third_party/WebKit/LayoutTests/external/wpt/css/fonts/ahem-extra/AHEM_Ahem!.TTF
3ba4d2d4f404bfa2a51ab96e40a0d914  ./third_party/WebKit/LayoutTests/fast/text/resources/AhemIVS.ttf
49c02203398871cccf6b647fd53fe035  ./testing/libfuzzer/fuzzers/woff2_corpus/Ahem.woff2
5f221786ddbc44371f334e4c9cc4b916  ./third_party/WebKit/LayoutTests/resources/Ahem.woff
5f221786ddbc44371f334e4c9cc4b916  ./third_party/WebKit/Source/platform/testing/data/Ahem.woff
8b9b5fd34719b8a4dc393fb91074b7c5  ./third_party/WebKit/LayoutTests/resources/AhemNoXHeight.ttf
9487853062f61c6550010321cce7b140  ./third_party/WebKit/LayoutTests/resources/Ahem.woff2
ae461e43368d346142e7ad1de21b4c17  ./testing/libfuzzer/fuzzers/woff2_corpus/AhemSpaceLigature.woff2
d380bf638b09b1fe766717069ba20f10  ./headless/test/data/Ahem.ttf
d380bf638b09b1fe766717069ba20f10  ./third_party/sfntly/fuzzers/corpora/sfntly_fuzzer/Ahem.ttf
d380bf638b09b1fe766717069ba20f10  ./third_party/WebKit/LayoutTests/external/wpt/content-security-policy/support/Ahem.ttf
d380bf638b09b1fe766717069ba20f10  ./third_party/WebKit/LayoutTests/external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/flexbox/support/Ahem.ttf
d380bf638b09b1fe766717069ba20f10  ./third_party/WebKit/LayoutTests/external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/support/Ahem.ttf
d380bf638b09b1fe766717069ba20f10  ./third_party/WebKit/LayoutTests/http/tests/resources/Ahem.ttf
d380bf638b09b1fe766717069ba20f10  ./third_party/WebKit/LayoutTests/http/tests/w3c/webperf/resources/Ahem.ttf
d380bf638b09b1fe766717069ba20f10  ./third_party/WebKit/LayoutTests/inspector-protocol/css/resources/Ahem.ttf
d380bf638b09b1fe766717069ba20f10  ./third_party/WebKit/LayoutTests/resources/Ahem.ttf
d380bf638b09b1fe766717069ba20f10  ./third_party/WebKit/Source/web/tests/data/Ahem.ttf
dfb4f72fd1136d52be547df640e2b053  ./third_party/WebKit/LayoutTests/third_party/AhemSpaceLigature/AhemSpaceLigature.woff
dfb4f72fd1136d52be547df640e2b053  ./third_party/WebKit/Source/platform/testing/data/AhemSpaceLigature.woff
f3a02e6142aeb21e994702aff56f47ec  ./third_party/WebKit/LayoutTests/resources/Ahem.otf
----------------

Checking the image of the results in the build,
the issue is that there's "X" instead of some boxes (Ahem font).
So it's clearly not loading the font properly,
dunno why that's happening only to this test.
There are a lot of tests using this font indeed.

The Grid Layout tests that were failing also use "ahem" instead of "Ahem"
(see  bug #718717 ).

I've landed https://codereview.chromium.org/2869353002 as a tentative fix for  bug #718717 .
Let's watch the build today and see how it goes to check if the issue is related to the font.

It seems the buildbot is happy with the grid change,
so we could do the same here.

The question is what's the best plan to move forward?
Note that this might have found a legitimate Windows issue.

My idea would be to write a patch doing 2 things:
* Using "Ahem" instead of "ahem" in "outline-004.html".
* Add a new test just checking the usage of "ahem".
Then in the next days we could check if that new test
starts to fail too, skip it and redirect the bug
to the relevant people.

What do you think? Any other alternative plan?

I've encountered issues with flaky tests due to fonts loading slowly. Not sure if it's the same, but in case it helps, this is the bug I was familiar with:  issue 507054 . That issue links to  issue 392046 , which might also be relevant.

So this is my proposal to fix this issue:
https://chromium-review.googlesource.com/c/506009/

PTAL, thanks!

This issue has been fixed in the CL linked before. So we can close this.

So it was finally an issue with the "ahem" font. More on that in  bug #724392 .