This might not because of bug 574272 . It seems that some repaint test doesn't respect waitUntilDone() and finish before running repaintTest(). When this happens, the actual text result is a layout tree instead of graphics layer tree, and the actual pixel result is in the initial state before repaintTest().
http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=inner-svg-change-viewBox-contract is an example: it has good baselines on most platforms, so the results are mostly green. On platforms with good baselines, purple results are actual failures caused by not running repaintTest(). On platforms with bad baselines (e.g. Win10), the purple results are actual succeeds. I think the bad baselines are because of flaky results of the tests, not because of bug 574272 .
Bug 620141 may be of the same situation. Looking into it.
Cc: -mfomitchev@chromium.org -dalecur...@chromium.org -grunell@chromium.org dpranke@chromium.org wkorman@chromium.org qyears...@chromium.org Components: -Blink>SVG Blink>Infra Labels: -Pri-2 Pri-1 Summary: Some layout tests sometimes finish before running any script (was: Tests still failing after rebaselining: svg/repaint)
This is not just a repaint test issue, but a common issue for tests containing scripts. The tests sometimes (mostly when the system load is high, e.g. during the first run of all layout tests) finish before running any script, including scripts like the following:
<script>
if (window.testRunner) {
testRunner.waitUntilDone();
testRunner.dumpAsText();
}
</script>
Examples:
- In https://storage.googleapis.com/chromium-layout-test-archives/mac_chromium_rel_ng/246153/layout-test-results/results.html (with 'flaky failures' checked):
- fast/sub-pixel/repaint-subpixel-layer-in-subpixel-composited-layer.html outputs an original layer tree and pixel result (the first actual failure is missing), indicating that testRunner.dumpAsText() was not executed or not respected.
- fast/table/resize-table-repaint-percent-size-cell.html
This also happened on other bots.
I can reproduce this locally by running all layout tests with --child-processes=8 (2702 tests failed during the first try). Using --child-processes=2 there were much less failing tests.
This is occurring again. Some auto-rebaselines happened to pick up the results when having this bug, for example https://codereview.chromium.org/2245063007/ created wrong baselines for some tests on Linux Trusty. The tests produced empty pixel result and dump of layout tree.
This might be also the reason of bug 637038 which is about testRunner.dumpAsText() is not respected in the tests.
Carlosk@ can you check if this has anything with your changes?
Since I was OOO I didn't land any changes since 3 weeks ago and even that one shouldn't affect any of this. So if this breakage started recently it was caused by something else. I'll take a look and see if I can figure it out.
I'm not sure when the issue starts again, just noticed it recently. Flakiness dashboard doesn't provide data of long time ago. It's also down now. I'll try to find more data when flakiness dashboard is fixed.
I bisected between my commit from #20 (good), d4c23ca618abaeeeba8b7f707bb2b1cb4460d642, and a commit I got from this morning (bad), 3839dbc4efacbba65f5d395301f2fecb9c58c1fa. During the bisect I filtered the tests I ran to just fast/harness/ as they were failing with the same missing image error and seemed to be a very good indication of problems as they test the layout test harness itself. This is the command line I used:
third_party/WebKit/Tools/Scripts/run-webkit-tests --debug --num-retries=0 --no-show-results third_party/WebKit/LayoutTests/fast/harness
The breaking change is https://codereview.chromium.org/2208603004 which changes the behavior of my fix of layout test preferences from #20. I'm almost sure it causes the "leaking" of preferences changes to following tests just like my original commit that was reverted.
Reassigning to pdr@ who landed it.
The following revision refers to this bug:
https://chromium.googlesource.com/chromium/src.git/+/ab2d1cd2ef3c8c12da17db01410d5749f1f4cb08
commit ab2d1cd2ef3c8c12da17db01410d5749f1f4cb08
Author: pdr <pdr@chromium.org>
Date: Mon Aug 22 17:48:47 2016
Ensure previous test preference overrides are reset between tests
This patch fixes a bug introduced by [1] where explicit test preference
overrides (e.g., testRunner.overridePreference(...)) would get re-used
in later tests.
BlinkTestController::PrepareForLayoutTest should completely reset
preferences before every test. On the first test run, default_prefs_
is initialized with default preferences plus compositor-test-specific
preferences. To ensure the compositor-test-specific preferences are
reset for subsequent runs, we call OverrideWebkitPrefs(&default_prefs_).
The bug in [1] was that default_prefs_ was read from the previous test
and would include explicitly overridden preferences. This patch removes
this line so default_prefs_ is no longer set to the previous test's
preferences.
Test that compositor-specific preferences are still reset:
run-webkit-tests compositing/animation/state-at-end-event-transform-layer.html fast/canvas/canvas-render-layer.html --additional-driver-flag=--enable-slimming-paint-v2 --num-retries=0 --child-processes=1
Test that explicit preference overrides are reset:
run-webkit-tests third_party/WebKit/LayoutTests/fast/harness/override-preferences.html third_party/WebKit/LayoutTests/fast/harness/results.html --num-retries=0 --child-processes=1
[1] https://crrev.com/a5191dca2b309396a7743816917f19d037ebc39a
R=carlosk@chromium.org,wangxianzhu@chromium.org
BUG= 620126
Review-Url: https://codereview.chromium.org/2264883002
Cr-Commit-Position: refs/heads/master@{#413470}
[modify] https://crrev.com/ab2d1cd2ef3c8c12da17db01410d5749f1f4cb08/content/shell/browser/layout_test/blink_test_controller.cc
Comment 1 by dmazz...@chromium.org
, Jun 15 2016