Webkit layout tests flaky on Windows because side-loaded fonts were using case-sensitive matching. |
||||||||
Issue descriptionI've got a simple CL that resets state between WebKit test runs: https://chromium-review.googlesource.com/c/chromium/src/+/1246723 This causes deterministic test failures on Windows [which I can repro locally]. Expected raw output: """ 17:26:50.510 35408 ERR: \n 17:26:50.510 35408 ERR: DevTools listening on ws://127.0.0.1:53840/devtools/browser/55943ff1-f6c6-496e-b393-c018efc9681b\n 17:26:50.510 35408 IN: "http://web-platform.test:8001/css/CSS2/text/text-indent-007.xht'--pixel-test\n 17:26:51.428 35408 OUT: Content-Type: text/plain\n 17:26:51.428 35408 OUT: layer at (0,0) size 800x600\n 17:26:51.429 35408 OUT: LayoutView at (0,0) size 800x600\n 17:26:51.429 35408 OUT: layer at (0,0) size 800x92\n 17:26:51.429 35408 OUT: LayoutBlockFlow {html} at (0,0) size 800x92\n 17:26:51.429 35408 OUT: LayoutBlockFlow {body} at (8,16) size 784x68\n 17:26:51.430 35408 OUT: LayoutBlockFlow {p} at (0,0) size 784x20\n 17:26:51.430 35408 OUT: LayoutText {#text} at (0,0) size 438x19\n 17:26:51.430 35408 OUT: text run at (0,0) width 438: "Test passes if there is one black box on this page (with no jagged edges)."\n 17:26:51.430 35408 OUT: LayoutBlockFlow {div} at (0,36) size 784x32\n 17:26:51.431 35408 OUT: LayoutBlockFlow {div} at (96,0) size 688x16\n 17:26:51.431 35408 OUT: LayoutText {#text} at (0,0) size 32x16\n 17:26:51.431 35408 OUT: text run at (0,0) width 32: "xx"\n 17:26:51.431 35408 OUT: LayoutBlockFlow {div} at (0,16) size 784x16\n 17:26:51.431 35408 OUT: LayoutText {#text} at (96,0) size 32x16\n 17:26:51.432 35408 OUT: text run at (96,0) width 32: "xx"\n 17:26:51.432 35408 OUT: #EOF\n 17:26:51.454 35408 OUT: \n 17:26:51.455 35408 OUT: ActualHash: 870ce90519d4bb5dcae4cdc6e44d9050\n 17:26:51.468 35408 ERR: #EOF\n 17:26:51.471 35408 IN: "http://web-platform.test:8001/css/CSS2/reference/pass_if_box_ahem.html'--pixel-test'870ce90519d4bb5dcae4cdc6e44d9050\n 17:26:51.523 35408 OUT: Content-Type: text/plain\n 17:26:51.535 35408 OUT: layer at (0,0) size 800x600\n 17:26:51.536 35408 OUT: LayoutView at (0,0) size 800x600\n 17:26:51.536 35408 OUT: layer at (0,0) size 800x92\n 17:26:51.536 35408 OUT: LayoutBlockFlow {HTML} at (0,0) size 800x92\n 17:26:51.536 35408 OUT: LayoutBlockFlow {BODY} at (8,16) size 784x68\n 17:26:51.537 35408 OUT: LayoutBlockFlow {P} at (0,0) size 784x20\n 17:26:51.537 35408 OUT: LayoutText {#text} at (0,0) size 438x19\n 17:26:51.537 35408 OUT: text run at (0,0) width 438: "Test passes if there is one black box on this page (with no jagged edges)."\n 17:26:51.537 35408 OUT: LayoutBlockFlow {DIV} at (96,36) size 688x32\n 17:26:51.537 35408 OUT: LayoutText {#text} at (0,0) size 32x16\n 17:26:51.538 35408 OUT: text run at (0,0) width 32: "FA"\n 17:26:51.538 35408 OUT: LayoutBR {BR} at (32,0) size 0x16\n 17:26:51.538 35408 OUT: LayoutText {#text} at (0,16) size 32x16\n 17:26:51.538 35408 OUT: text run at (0,16) width 32: "IL"\n 17:26:51.539 35408 OUT: #EOF\n 17:26:51.539 35408 OUT: \n 17:26:51.539 35408 OUT: ActualHash: 870ce90519d4bb5dcae4cdc6e44d9050\n 17:26:51.539 35408 OUT: \n 17:26:51.539 35408 OUT: ExpectedHash: 870ce90519d4bb5dcae4cdc6e44d9050\n 17:26:51.540 35408 OUT: #EOF\n 17:26:51.540 35408 ERR: #EOF\n 17:26:51.541 35408 """ Actual raw output: """ 17:16:18.653 1596 OUT: #READY\n 17:16:18.653 1596 ERR: \n 17:16:18.654 1596 ERR: DevTools listening on ws://127.0.0.1:53122/devtools/browser/a47879fd-2a96-46d9-bff6-e1bbb2f79cfc\n 17:16:18.654 1596 IN: "http://web-platform.test:8001/css/CSS2/text/text-indent-007.xht'--pixel-test\n 17:16:19.581 1596 OUT: layer at (0,0) size 800x600\n 17:16:19.581 1596 OUT: LayoutView at (0,0) size 800x600\n 17:16:19.582 1596 OUT: layer at (0,0) size 800x92\n 17:16:19.582 1596 OUT: LayoutBlockFlow {html} at (0,0) size 800x92\n 17:16:19.582 1596 OUT: LayoutBlockFlow {body} at (8,16) size 784x68\n 17:16:19.582 1596 OUT: LayoutBlockFlow {p} at (0,0) size 784x20\n 17:16:19.582 1596 OUT: LayoutText {#text} at (0,0) size 438x19\n 17:16:19.583 1596 OUT: text run at (0,0) width 438: "Test passes if there is one black box on this page (with no jagged edges)."\n 17:16:19.583 1596 OUT: LayoutBlockFlow {div} at (0,36) size 784x32\n 17:16:19.583 1596 OUT: LayoutBlockFlow {div} at (96,0) size 688x16\n 17:16:19.583 1596 OUT: LayoutText {#text} at (0,0) size 32x16\n 17:16:19.584 1596 OUT: text run at (0,0) width 32: "xx"\n 17:16:19.584 1596 OUT: LayoutBlockFlow {div} at (0,16) size 784x16\n 17:16:19.584 1596 OUT: LayoutText {#text} at (96,0) size 32x16\n 17:16:19.584 1596 OUT: text run at (96,0) width 32: "xx"\n 17:16:19.630 1596 IN: "http://web-platform.test:8001/css/CSS2/reference/pass_if_box_ahem.html'--pixel-test'870ce90519d4bb5dcae4cdc6e44d9050\n 17:16:19.650 1596 ERR: [36820:15908:0926/171619.640:ERROR:blink_test_controller.cc(390)] CreateNewWindow\n 17:16:19.732 1596 ERR: [36820:34848:0926/171619.726:ERROR:process_win.cc(155)] Unable to terminate process: Access is denied. (0x5)\n 17:16:20.446 1596 OUT: Content-Type: text/plain\n 17:16:20.447 1596 OUT: layer at (0,0) size 800x600\n 17:16:20.447 1596 OUT: LayoutView at (0,0) size 800x600\n 17:16:20.447 1596 OUT: layer at (0,0) size 800x92\n 17:16:20.448 1596 OUT: LayoutBlockFlow {HTML} at (0,0) size 800x92\n 17:16:20.448 1596 OUT: LayoutBlockFlow {BODY} at (8,16) size 784x68\n 17:16:20.448 1596 OUT: LayoutBlockFlow {P} at (0,0) size 784x20\n 17:16:20.448 1596 OUT: LayoutText {#text} at (0,0) size 438x19\n 17:16:20.448 1596 OUT: text run at (0,0) width 438: "Test passes if there is one black box on this page (with no jagged edges)."\n 17:16:20.449 1596 OUT: LayoutBlockFlow {DIV} at (96,36) size 688x32\n 17:16:20.449 1596 OUT: LayoutText {#text} at (0,-2) size 19x19\n 17:16:20.449 1596 OUT: text run at (0,-2) width 19: "FA"\n 17:16:20.449 1596 OUT: LayoutBR {BR} at (19,-2) size 0x19\n 17:16:20.449 1596 OUT: LayoutText {#text} at (0,14) size 14x19\n 17:16:20.450 1596 OUT: text run at (0,14) width 14: "IL"\n 17:16:20.450 1596 OUT: #EOF\n 17:16:20.471 1596 OUT: \n 17:16:20.472 1596 OUT: ActualHash: 83929f0dda62abc81281a8def1b815cb\n 17:16:20.472 1596 OUT: \n 17:16:20.472 1596 OUT: ExpectedHash: 870ce90519d4bb5dcae4cdc6e44d9050\n 17:16:20.472 1596 OUT: Content-Type: image/png\n """ It looks like the test itself is generating the same results -- producing the same hash 870ce90519d4bb5dcae4cdc6e44d9050. But the reference page is actually producing different results.
,
Sep 26
That sounds bad, looks like another timing issue with loading the special Ahem font. Thanks for bringing this to my attention. Koji, do you know how Ahem is loaded for WPT tests?
,
Sep 26
Lower-case "ahem" reminds me of issue 724392 .
,
Sep 27
Thanks robertma@. I've confirmed that the test deterministically passes with "Ahem" and deterministically fails with "ahem". [with CL from opening comment patched in].
,
Sep 27
It is not clear to me whether CSS Fonts are supposed to be case sensitive. font families should are case insensitive: https://drafts.csswg.org/css-fonts/#font-family-casing It looks like side_loaded fonts are case sensitive. https://cs.chromium.org/chromium/src/third_party/blink/renderer/platform/fonts/skia/font_cache_skia.cc?type=cs&q=sideloaded_fonts&g=0&l=264 Which is why the lower-case "ahem" causes failures with my CL patched in. I have not investigated to see why the test passes without my CL -- at a guess, there is state carrying through between tests. :) I'll change side-loaded fonts to be case insensitive.
,
Sep 27
Thank you Eric!
,
Sep 27
,
Sep 27
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/90d29c2c598796708cf504925c72f9cc89248291 commit 90d29c2c598796708cf504925c72f9cc89248291 Author: Erik Chen <erikchen@chromium.org> Date: Thu Sep 27 19:55:13 2018 Side loaded fonts should be case insensitive. This is a source of webkit_layout_test flakiness on Windows. Bug: 889608 Change-Id: Ib31cd068620d0c1917aaa85ce2ec7038ccd4236e Reviewed-on: https://chromium-review.googlesource.com/1249385 Commit-Queue: Erik Chen <erikchen@chromium.org> Reviewed-by: Emil A Eklund <eae@chromium.org> Reviewed-by: Koji Ishii <kojii@chromium.org> Cr-Commit-Position: refs/heads/master@{#594827} [modify] https://crrev.com/90d29c2c598796708cf504925c72f9cc89248291/third_party/WebKit/LayoutTests/platform/win/fast/canvas/canvas-blending-text-expected.txt [modify] https://crrev.com/90d29c2c598796708cf504925c72f9cc89248291/third_party/blink/renderer/platform/fonts/font_cache.h [modify] https://crrev.com/90d29c2c598796708cf504925c72f9cc89248291/third_party/blink/renderer/platform/fonts/skia/font_cache_skia.cc [modify] https://crrev.com/90d29c2c598796708cf504925c72f9cc89248291/third_party/blink/renderer/platform/fonts/win/font_cache_skia_win.cc
,
Sep 27
,
Sep 27
,
Sep 28
Thanks for finding and fixing this, well done!
,
Dec 4
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by erikc...@chromium.org
, Sep 26I think the problem here is that layout of the reference build is non-deterministic. Specifically: """ div { font: 16px/1em ahem; margin-left: 96px; } </style> <p>Test passes if there is one black box on this page (with no jagged edges).</p> <div>FA<br>IL</div> """ I think that "<div>FA<br>IL</div>" produces underspecified results, which is why the difference between expected reference rendering and actual reference rendering has slight differences: """ 7:26:51.537 35408 OUT: LayoutBlockFlow {DIV} at (96,36) size 688x32\n 17:26:51.537 35408 OUT: LayoutText {#text} at (0,0) size 32x16\n 17:26:51.538 35408 OUT: text run at (0,0) width 32: "FA"\n 17:26:51.538 35408 OUT: LayoutBR {BR} at (32,0) size 0x16\n 17:26:51.538 35408 OUT: LayoutText {#text} at (0,16) size 32x16\n 17:26:51.538 35408 OUT: text run at (0,16) width 32: "IL"\n """ vs. """ 17:16:20.449 1596 OUT: LayoutBlockFlow {DIV} at (96,36) size 688x32\n 17:16:20.449 1596 OUT: LayoutText {#text} at (0,-2) size 19x19\n 17:16:20.449 1596 OUT: text run at (0,-2) width 19: "FA"\n 17:16:20.449 1596 OUT: LayoutBR {BR} at (19,-2) size 0x19\n 17:16:20.449 1596 OUT: LayoutText {#text} at (0,14) size 14x19\n 17:16:20.450 1596 OUT: text run at (0,14) width 14: "IL"\n """ +eae for layout support