New issue
Advanced search Search tips

Issue 772405 link

Starred by 3 users

Issue metadata

Status: WontFix
Owner:
Closed: Oct 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug

Blocking:
issue 772407



Sign in to add a comment

Web Animations: Large number of failing WPT tests hidden by -expected.txt files

Project Member Reported by smcgruer@chromium.org, Oct 6 2017

Issue description

It appears that during a WPT import some time ago, a lot of failing web-animation WPT tests were essentially 'hidden' by adding -expected.txt files that include FAIL lines in them.

This has meant that whilst TestExpectations only lists 9 files that fail for WebAnimations, we actually fail something on the order of 300 tests, not a dozen:

$ find third_party/WebKit/LayoutTests/external/wpt/web-animations/ -name *-expected.txt | wc -l
28
$ grep "FAIL" `find third_party/WebKit/LayoutTests/external/wpt/web-animations/ -name *-expected.txt` | wc -l
319

We should decide what to do here; putting them all in TestExpectations and deleting the -expected.txt files would be most in-line with other Blink infra, but would hide a good number of passes as well.
 
Blocking: 772407
There seem to be 4 expected files with no PASS lines:
grep "FAIL" `find third_party/WebKit/LayoutTests/external/wpt/web-animations/ -name *-expected.txt`|cut -d: -f 1|uniq|xargs grep -c "PASS"|grep ":0$"
third_party/WebKit/LayoutTests/external/wpt/web-animations/timing-model/animations/set-the-timeline-of-an-animation-expected.txt:0
third_party/WebKit/LayoutTests/external/wpt/web-animations/interfaces/KeyframeEffect/setTarget-expected.txt:0
third_party/WebKit/LayoutTests/external/wpt/web-animations/interfaces/KeyframeEffect/copy-contructor-expected.txt:0
third_party/WebKit/LayoutTests/external/wpt/web-animations/interfaces/KeyframeEffectReadOnly/copy-contructor-expected.txt:0

Seems like these could safely be removed and added as failures in TestExpectations without masking any failures.
I'll send out a CL to do so.

The more I think about it, the more I'm coming around to this -expected.txt approach for JS harness tests. It seems dangerous to mark an entire suite of tests as failing (in TestExpectations) because just one of them fails - you run the risk of causing others to start failing in future changes without noticing. Of course the downside is that you get no easy visibility (like we found!) into how many of your tests actually pass or fail.
Having discussed this with Blink>Infra folks, it appears that the current approach for WPT tests is to use TestExpectations only for flaky tests if possible. Non-flaky tests should have an -expected.txt file, no matter how many of their sub-tests fail (e.g. some or all).

So the correct thing to do here is actually to remove the 8 non-flaky entries from TestExpectations, and add -expected.txt for them instead.
Labels: Update-Quarterly
Status: Wong (was: Assigned)
Further investigation reveals that we cannot use -expectations.txt for these test, since they generate failure values that change per test run. Preferably we would not have WPT tests that do that, but that's not high enough priority to worry about atm. Closing as WontFix :(.
Status: WontFix (was: wong)

Sign in to add a comment