WPT manifest generator considers some css reftests to be manual tests |
|||||||
Issue descriptionAfter the merge of csswg-test into wpt, it appears that some tests which should be considered reftests are now listed as manual tests in MANIFEST.json, causing them to fail. Specifically, I'm seeing this happen in https://chromium-review.googlesource.com/c/462475/ with: external/wpt/css/css-writing-modes-3/background-position-vlr-003.xht external/wpt/css/css-writing-modes-3/background-position-vlr-005.xht external/wpt/css/css-writing-modes-3/background-position-vlr-007.xht external/wpt/css/css-writing-modes-3/background-position-vlr-009.xht external/wpt/css/css-writing-modes-3/background-position-vlr-011.xht external/wpt/css/css-writing-modes-3/background-position-vlr-013.xht external/wpt/css/css-writing-modes-3/background-position-vlr-015.xht external/wpt/css/css-writing-modes-3/background-position-vlr-017.xht external/wpt/css/css-writing-modes-3/background-position-vrl-002.xht external/wpt/css/css-writing-modes-3/background-position-vrl-004.xht external/wpt/css/css-writing-modes-3/background-position-vrl-006.xht external/wpt/css/css-writing-modes-3/background-position-vrl-008.xht external/wpt/css/css-writing-modes-3/background-position-vrl-010.xht external/wpt/css/css-writing-modes-3/background-position-vrl-012.xht external/wpt/css/css-writing-modes-3/background-position-vrl-014.xht external/wpt/css/css-writing-modes-3/background-position-vrl-016.xht external/wpt/css/css-writing-modes-3/background-position-vrl-018.xht external/wpt/css/css-writing-modes-3/background-position-vrl-020.xht external/wpt/css/css-writing-modes-3/background-position-vrl-022.xht external/wpt/css/css-writing-modes-3/first-page-vlr-003.xht external/wpt/css/css-writing-modes-3/first-page-vrl-002.xht external/wpt/css/css-writing-modes-3/page-flow-direction-002.xht external/wpt/css/css-writing-modes-3/page-flow-direction-003.xht external/wpt/css/css-writing-modes-3/page-flow-direction-slr-005.xht external/wpt/css/css-writing-modes-3/page-flow-direction-srl-004.xht external/wpt/css/css-writing-modes-3/text-orientation-mixed-vlr-100.html external/wpt/css/css-writing-modes-3/text-orientation-mixed-vrl-100.html external/wpt/css/css-writing-modes-3/text-orientation-sideways-vlr-100.html external/wpt/css/css-writing-modes-3/text-orientation-sideways-vrl-100.html external/wpt/css/css-writing-modes-3/text-orientation-upright-vlr-100.html external/wpt/css/css-writing-modes-3/text-orientation-upright-vrl-100.html To reproduce, first ensure that these files are present, cd to WebKit/Tools/Scripts, and run: webkitpy/thirdparty/wpt/wpt/manifest --work --tests-root ../../LayoutTests/external/wpt/
,
Mar 30 2017
Actually, it looks like the following are supposed to be ref tests, and the rest are supposed to be manual tests? external/wpt/css/css-writing-modes-3/background-position-vrl-018.xht external/wpt/css/css-writing-modes-3/background-position-vrl-020.xht external/wpt/css/css-writing-modes-3/background-position-vrl-022.xht external/wpt/css/css-writing-modes-3/text-orientation-mixed-vlr-100.html external/wpt/css/css-writing-modes-3/text-orientation-mixed-vrl-100.html external/wpt/css/css-writing-modes-3/text-orientation-sideways-vlr-100.html external/wpt/css/css-writing-modes-3/text-orientation-sideways-vrl-100.html external/wpt/css/css-writing-modes-3/text-orientation-upright-vlr-100.html external/wpt/css/css-writing-modes-3/text-orientation-upright-vrl-100.html These files contain <link rel="match"...> but are listed in the "manual" section of MANIFEST.json. The rest are manual tests which fail when run because when run, they output a render tree dump but there is no baseline present.
,
Mar 30 2017
These look to be tests written/owned by the layout team.
,
Mar 31 2017
#2: the list looks right to me. We used to import them as ref tests, so is this a problem in the manifest generator?
,
Mar 31 2017
I think I have a guess now.
background-position-vrl-018.xht for instance has:
<meta content="image interact" name="flags" />
and content_is_css_manual(self) has:
return bool(self.css_flags & {"animated", "font", "history", "interact", "paged", "speech", "userstyle"})
So I think the test author added "interact" flag meaning it's a ref test, but additional manual test is possible, as in the comment in the test:
<meta content="This test checks that 'background-position: left top' will make background-image start at left side of document root element (even in case there is resizing of the window) because background properties should not be affected by vertical writing-mode." name="assert" />
I guess our old importer ignored the flag, or "match" wins over the flag, while the manifest generator does the opposite.
Could the manifest generator prioritize "match" over "flags" if they contradict?
,
Mar 31 2017
That makes sense; filed issue https://github.com/w3c/wpt-tools/issues/196; if jgraham and others are OK with it, perhaps we could change the behavior of the manifest generator?
,
Jul 3 2017
,
Jul 3 2017
,
Jul 5 2017
https://github.com/w3c/wpt-tools/issues/196 was closed; I suspect this is going to be WontFix, and the solution in each case is going to be to change the tests so that they don't contain <meta name="flags", content="...">. |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by qyears...@chromium.org
, Mar 30 2017