Web Animations: Large number of failing WPT tests hidden by -expected.txt files |
||||
Issue descriptionIt appears that during a WPT import some time ago, a lot of failing web-animation WPT tests were essentially 'hidden' by adding -expected.txt files that include FAIL lines in them. This has meant that whilst TestExpectations only lists 9 files that fail for WebAnimations, we actually fail something on the order of 300 tests, not a dozen: $ find third_party/WebKit/LayoutTests/external/wpt/web-animations/ -name *-expected.txt | wc -l 28 $ grep "FAIL" `find third_party/WebKit/LayoutTests/external/wpt/web-animations/ -name *-expected.txt` | wc -l 319 We should decide what to do here; putting them all in TestExpectations and deleting the -expected.txt files would be most in-line with other Blink infra, but would hide a good number of passes as well.
,
Oct 6 2017
There seem to be 4 expected files with no PASS lines: grep "FAIL" `find third_party/WebKit/LayoutTests/external/wpt/web-animations/ -name *-expected.txt`|cut -d: -f 1|uniq|xargs grep -c "PASS"|grep ":0$" third_party/WebKit/LayoutTests/external/wpt/web-animations/timing-model/animations/set-the-timeline-of-an-animation-expected.txt:0 third_party/WebKit/LayoutTests/external/wpt/web-animations/interfaces/KeyframeEffect/setTarget-expected.txt:0 third_party/WebKit/LayoutTests/external/wpt/web-animations/interfaces/KeyframeEffect/copy-contructor-expected.txt:0 third_party/WebKit/LayoutTests/external/wpt/web-animations/interfaces/KeyframeEffectReadOnly/copy-contructor-expected.txt:0 Seems like these could safely be removed and added as failures in TestExpectations without masking any failures.
,
Oct 6 2017
I'll send out a CL to do so. The more I think about it, the more I'm coming around to this -expected.txt approach for JS harness tests. It seems dangerous to mark an entire suite of tests as failing (in TestExpectations) because just one of them fails - you run the risk of causing others to start failing in future changes without noticing. Of course the downside is that you get no easy visibility (like we found!) into how many of your tests actually pass or fail.
,
Oct 6 2017
Having discussed this with Blink>Infra folks, it appears that the current approach for WPT tests is to use TestExpectations only for flaky tests if possible. Non-flaky tests should have an -expected.txt file, no matter how many of their sub-tests fail (e.g. some or all). So the correct thing to do here is actually to remove the 8 non-flaky entries from TestExpectations, and add -expected.txt for them instead.
,
Oct 9 2017
,
Oct 17 2017
Further investigation reveals that we cannot use -expectations.txt for these test, since they generate failure values that change per test run. Preferably we would not have WPT tests that do that, but that's not high enough priority to worry about atm. Closing as WontFix :(.
,
Oct 17 2017
|
||||
►
Sign in to add a comment |
||||
Comment 1 by smcgruer@chromium.org
, Oct 6 2017