New issue
Advanced search Search tips

Issue 662249 link

Starred by 1 user
Status: WontFix
Owner: ----
Closed: Nov 2016
Cc:
tkent@chromium.org
jsb...@chromium.org
foolip@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

Make W3CImportExpectations a whitelist of test directories to import?

Project Member Reported by qyears...@chromium.org, Nov 4 2016 
Fairly frequently, when new directories are added in web-platform-tests or csswg-test, we also want to add new directories to W3CImportExpectations to not import that directory.

Currently, W3CImportExpectations acts like a blacklist, and newly-added directories in wpt or csswg-test will normally be imported, but sometimes newly-added directories contain a lot of new failing tests that nobody is planning to look at.

If our policy moving forward is still going to be that we want to have owners for all directories that are imported, then it will be more manageable to make W3CImportExpectations a whitelist rather than a blacklist.

Does this sound like a reasonable idea?

Also, would it be worth it to change the auto-importer to automatically skip new directories if there are (many) failing tests?

Comment 1 by jsb...@chromium.org, Nov 4 2016 

+1 to whitelist

Would this apply to all subdirectories or merely top level directories (e.g. imported/wpt/html, imported/wpt/IndexedDB)? What has the history been with newly added subdirectories? If those have generally been importable with a pass/fail rate comparable to the parent/sibling directory then I'd prefer to have the whitelist apply only to the top level so that e.g. we don't lose coverage if html/ gets rearranged, or FileAPI/ gets some new tests that happen to land in a subdir.

My guess (with no data!) is that the rate at which subdirs are added is low and when it happens our pass/fail rate is comparable, so importing w/ failures - not skipping - is preferrable.

One more feature request: auto-file a bug when the importer detects a new top-level directory that isn't on the whitelist. (that would require tracking which have been previously seen and skipped)

Comment 2 by tkent@chromium.org, Nov 7 2016 

Does the current W3CImportExpectations processing code support flipping value multiple times?  I mean:

imported/wpt [ Skip ]
imported/wpt/html [ Pass ]
imported/wpt/html/editing/dnd/svg [ Skip ]

would skip new directories and wpt/html/editing/dnd/svg, and import wpt/html/* except editing/dnd/svg?

Comment 3 by qyears...@chromium.org, Nov 7 2016

Summary: Make W3CImportExpectations a whitelist of test directories to import? (was: New directories sometimes need to be added to W3CImportExpectations every time new directories are added upstream.)
I was wondering that too, it looks like that doesn't work --

The code that reads in W3CImportExpectations and decides what tests to skip based on it is at:
https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_importer.py?l=291

It only looks at lines with Skip expectations, not Pass expectations, so if we wanted to change W3CImportExpectations to be a whitelist (or at least whitelist top-level directories, and then possibly skip particular directories within those) then that would also involve changing the logic in test_importer.py.

Not sure if it's worth it to do that now, so I'll hold off for now.

Comment 4 by foolip@chromium.org, Nov 8 2016 

FWIW, I think we should strive towards a model of importing everything verbatim and skipping tests when running instead, unless there are so many skipped tests that it'd slow things down.

Comment 5 by qyears...@chromium.org, Nov 8 2016

Status: WontFix (was: Unconfirmed)
Yeah, that makes sense.

Closing this bug for now; opened a related bug:  bug 663383 : Auto-file a bug (and/or add to skip list) when the importer detects a new top-level directory.
►

Sign in to add a comment