[W3C auto-import] Use more concise platform specifiers in test expectation lines. |
||||||||||
Issue descriptionIn http://crrev.com/2406263004, w3c-test-autoroller added lines lines like this: # Tests added from W3C auto import bot crbug.com/626703 [ Precise Trusty Mac10.11 Mac10.10 Retina Win7 Win10 Mac10.9 ] imported/csswg-test/css-display-3/display-contents-before-after-002.html [ Failure ] crbug.com/626703 [ Precise Trusty Mac10.11 Mac10.10 Retina Win7 Win10 Mac10.9 ] imported/csswg-test/css-display-3/display-contents-tr-001.html [ Failure ] crbug.com/626703 [ Precise Trusty Mac10.11 Mac10.10 Retina Win7 Win10 Mac10.9 ] imported/csswg-test/css-display-3/display-contents-block-001.html [ Failure ] crbug.com/626703 [ Precise Trusty Mac10.11 Mac10.10 Retina Win7 Win10 Mac10.9 ] imported/csswg-test/css-display-3/display-contents-first-letter-001.html [ Failure ] crbug.com/626703 [ Precise Trusty Mac10.11 Mac10.10 Retina Win7 Win10 Mac10.9 ] imported/csswg-test/css-display-3/display-contents-inline-001.html [ Failure ] crbug.com/626703 [ Precise Trusty Mac10.11 Mac10.10 Retina Win7 Win10 Mac10.9 ] imported/csswg-test/css-display-3/display-contents-before-after-001.html [ Failure ] crbug.com/626703 [ Precise Trusty Mac10.11 Mac10.10 Retina Win7 Win10 Mac10.9 ] imported/csswg-test/css-writing-modes-3/wm-propagation-body-008.xht [ Failure ] crbug.com/626703 [ Precise Trusty Mac10.11 Mac10.10 Retina Win7 Win10 Mac10.9 ] imported/csswg-test/css-display-3/display-contents-first-line-001.html [ Failure ] crbug.com/626703 [ Precise Trusty Mac10.11 Mac10.10 Retina Win7 Win10 Mac10.9 ] imported/csswg-test/css-display-3/display-contents-block-002.html [ Failure ] crbug.com/626703 [ Precise Trusty Mac10.11 Mac10.10 Retina Win7 Win10 Mac10.9 ] imported/csswg-test/css-display-3/display-contents-td-001.html [ Failure ] Note that the test is not run on Android, and most likely it would fail there too, so these lines should probably all have no platform-specific specifiers. Another slightly more reasonable thing that w3c-test-autoroller could have done here would have been to note that we still don't know what the status is for android, but at least we can make the platform specifier [ Linux Mac Win ] instead of listing all of the specific OS versions. The place that should be changed is here: https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/update_w3c_test_expectations.py?l=254
,
Oct 12 2016
,
Oct 14 2016
,
Nov 17 2016
,
Dec 28 2016
,
Jan 3 2017
,
Jan 3 2017
,
Jan 18 2017
I had started to work on this (https://codereview.chromium.org/2607013002) and then realized that it wasn' quite as easy as I thought originally. I think what we actually want to do is to change update_w3c_test_expectations.py so that Port objects (or full port-version strings) are used in the dicts with results; then, the simplification code would simplify, taking into account ports that skip the given test (e.g. Android due to running only smoke tests). General logic might be: For each port class (linux, mac, win, android), if all versions have the same result or if the test is skipped, then the general non-version-specific specifier can be added, replacing any version-specific specifiers. Then, if specifiers for all port classes are there, then all OS specifiers should be removed.
,
Feb 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a0c9d2388e2ad1d6e500befba0aa38af5bb4e9cb commit a0c9d2388e2ad1d6e500befba0aa38af5bb4e9cb Author: qyearsley <qyearsley@chromium.org> Date: Thu Feb 09 18:10:09 2017 Preliminary refactoring to improve test expectation specifiers added by importer. This change includes clean-up and refactoring in wpt_expectations_updater without actually changing its behavior. BUG= 655201 Review-Url: https://codereview.chromium.org/2680393003 Cr-Commit-Position: refs/heads/master@{#449350} [modify] https://crrev.com/a0c9d2388e2ad1d6e500befba0aa38af5bb4e9cb/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater.py [modify] https://crrev.com/a0c9d2388e2ad1d6e500befba0aa38af5bb4e9cb/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater_unittest.py
,
Feb 11 2017
Started: https://codereview.chromium.org/2687093005
,
Feb 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/14d0f71b8deb5b63f498c92ccc37a65b46536e84 commit 14d0f71b8deb5b63f498c92ccc37a65b46536e84 Author: qyearsley <qyearsley@chromium.org> Date: Tue Feb 14 05:26:24 2017 WPT importer: Use concise lists of version specifiers in expectation lines. For example, instead of adding TestExpectation lines like crbug.com/626703 [ Precise Trusty Mac10.11 Mac10.10 Retina Win7 Win10 Mac10.9 ] external/wpt/foo.html [ Failure ] crbug.com/626703 [ Precise Trusty Mac10.11 Mac10.10 Retina Mac10.9 ] external/wpt/bar.html [ Failure ] Add lines like: crbug.com/626703 external/wpt/foo.html [ Failure ] crbug.com/626703 [ Mac ] external/wpt/bar.html [ Failure ] Note, this change includes the change in https://codereview.chromium.org/2692063002. BUG= 655201 Review-Url: https://codereview.chromium.org/2687093005 Cr-Commit-Position: refs/heads/master@{#450264} [modify] https://crrev.com/14d0f71b8deb5b63f498c92ccc37a65b46536e84/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/builder_list.py [modify] https://crrev.com/14d0f71b8deb5b63f498c92ccc37a65b46536e84/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/builder_list_unittest.py [modify] https://crrev.com/14d0f71b8deb5b63f498c92ccc37a65b46536e84/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py [modify] https://crrev.com/14d0f71b8deb5b63f498c92ccc37a65b46536e84/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater.py [modify] https://crrev.com/14d0f71b8deb5b63f498c92ccc37a65b46536e84/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater_unittest.py
,
Feb 15 2017
One last CL, and then this should be fixed: https://codereview.chromium.org/2694293003
,
Feb 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3554f65cd92e8391454b0698510b84f6864749c8 commit 3554f65cd92e8391454b0698510b84f6864749c8 Author: qyearsley <qyearsley@chromium.org> Date: Sun Feb 19 23:56:35 2017 Simplify specifiers when all platforms are covered, and sort lines. BUG= 655201 Review-Url: https://codereview.chromium.org/2694293003 Cr-Commit-Position: refs/heads/master@{#451532} [modify] https://crrev.com/3554f65cd92e8391454b0698510b84f6864749c8/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater.py [modify] https://crrev.com/3554f65cd92e8391454b0698510b84f6864749c8/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater_unittest.py
,
Feb 27 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db commit 3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db Author: qyearsley <qyearsley@chromium.org> Date: Mon Feb 27 19:12:25 2017 Fix specifier simplification and add test. There was still a bug left in logic for simplifying the specifier list for new test expectation lines: specifiers for skipped tests were added after simplification, meaning that in the (common) case where the test fails the same on all platforms and android is skipped, the specifier list is simplified to [ Linux Mac Win ] and then Android is added. The fix is to add specifiers for skipped ports before simplifying. BUG= 655201 Review-Url: https://codereview.chromium.org/2709103009 Cr-Commit-Position: refs/heads/master@{#453284} [modify] https://crrev.com/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db/third_party/WebKit/Tools/Scripts/webkitpy/common/host_mock.py [add] https://crrev.com/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/factory_mock.py [modify] https://crrev.com/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater.py [modify] https://crrev.com/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater_unittest.py
,
Feb 27 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db commit 3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db Author: qyearsley <qyearsley@chromium.org> Date: Mon Feb 27 19:12:25 2017 Fix specifier simplification and add test. There was still a bug left in logic for simplifying the specifier list for new test expectation lines: specifiers for skipped tests were added after simplification, meaning that in the (common) case where the test fails the same on all platforms and android is skipped, the specifier list is simplified to [ Linux Mac Win ] and then Android is added. The fix is to add specifiers for skipped ports before simplifying. BUG= 655201 Review-Url: https://codereview.chromium.org/2709103009 Cr-Commit-Position: refs/heads/master@{#453284} [modify] https://crrev.com/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db/third_party/WebKit/Tools/Scripts/webkitpy/common/host_mock.py [add] https://crrev.com/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/factory_mock.py [modify] https://crrev.com/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater.py [modify] https://crrev.com/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater_unittest.py
,
Feb 27 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db commit 3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db Author: qyearsley <qyearsley@chromium.org> Date: Mon Feb 27 19:12:25 2017 Fix specifier simplification and add test. There was still a bug left in logic for simplifying the specifier list for new test expectation lines: specifiers for skipped tests were added after simplification, meaning that in the (common) case where the test fails the same on all platforms and android is skipped, the specifier list is simplified to [ Linux Mac Win ] and then Android is added. The fix is to add specifiers for skipped ports before simplifying. BUG= 655201 Review-Url: https://codereview.chromium.org/2709103009 Cr-Commit-Position: refs/heads/master@{#453284} [modify] https://crrev.com/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db/third_party/WebKit/Tools/Scripts/webkitpy/common/host_mock.py [add] https://crrev.com/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/factory_mock.py [modify] https://crrev.com/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater.py [modify] https://crrev.com/3e6f43b480f54b4abe4b8a9e8d7e95abec31c2db/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater_unittest.py
,
Mar 2 2017
,
Jul 3 2017
,
Jul 3 2017
|
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by qyears...@chromium.org
, Oct 12 2016