Cc: tkent@chromium.org jsb...@chromium.org foolip@chromium.org Summary: [WPT import] new expectations for non-wpt tests (when testharness.js is updated) are not added (was: [WPT import] wpt-update-expectations failed to add timeout expectations for four new tests)
OK, so the reason why these four tests did not have expectations updated is because they're not wpt tests, and wpt-update-expectations has a rule to look at only wpt tests to avoid accidentally adding expectations for flaky non-wpt tests (https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater.py?l=270).
The underlying reason why this not correct is that changes to testharness.js can affect behavior of any test, because testharness.js is copied to LayoutTests/resources/testharness.js.
The reason why testharness.js is copied to LayoutTests/resources/testharness.js on import is to keep that copy of testharness.js in sync with LayoutTests/external/wpt/resources/testharness.js so that our testharness.js outside of external/wpt behave consistently with
There are two main options to make the behavior of wpt-update-expectations correct again:
1. Allow adding expectation lines for *any* tests, including those outside of wpt.
The downside of this change would be that bogus expectations might be added for flaky tests outide of wpt. If we do this, we might want to enable this only when
2. Stop copying resource files over to LayoutTests/resources.
The downside of this would be that it might result in testharness.js being out of date, which means that tests would behave differently when copied over into wpt,
I know we've discussed this before, so now I hope to come up with a more final/long-term solution.
Any thoughts about preferences for solution 1 vs 2? I'm leaning towards 2 now (since it's simpler in terms of import) but I could be persuaded :-)
Comment 1 by qyears...@chromium.org
, May 16 2017Summary: [WPT import] new expectations for non-wpt tests (when testharness.js is updated) are not added (was: [WPT import] wpt-update-expectations failed to add timeout expectations for four new tests)