*.any.js tests are similar, expanding scope.

 Issue 664935  has been merged into this issue.

Now we're running everything through wptserve and relying on the MANIFEST.json this should be easy, no?

We should take care of:

* Command-line interface
  % run-webkit-tests external/wpt/console/console-is-a-namespace.any.js
  should run two tests, or should we specify generated one?
  % run-webkit-tests external/wpt/console/console-is-a-namespace.any.html

* TestExpectaions
  lint-test-expectations command checks file existence of specified tests. However console-is-a-naemespace.any.html doesn't exist as a file. We need a special handling like virtual/.

Is there already logic we can re-use in w3c/web-platform-tests or w3c/wptrunner that understands what these extensions mean and can map them to real tests? I'll check.

#8,
MANIFEST.json already has necessary information.  The main change would be something like changing:

url = manifest_items[0][0]
if url[1:] != path_in_wpt:
    # TODO(tkent): foo.any.js and bar.worker.js should be accessed
    # as foo.any.html, foo.any.worker, and bar.worker with WPTServe.
    continue
tests.append(file_path)

to

for item in manifest_items:
  test_append(dirname(file_path) + basename(item[0]))

in _convert_wpt_file_paths_to_url_paths() of base.py.

annevk has suggested refactor test I care about (storage/, encoding/, etc) to be .any.js test to gain coverage. If that happens w/o this fix we'll start to lose coverage, which would be :(

So +1 / me too / starred / etc.

Note that there's also patches for large parts of fetch/ and websocket/ to make use of this convention (and once .window.js is added I expect a lot of new tests to use this as often you only need JavaScript).

This seems to do it...

--- a/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_manifest.py
+++ b/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_manifest.py
@@ -50,14 +50,7 @@ class WPTManifest(object):
     def file_path_to_url_paths(self, path_in_wpt):
         manifest_items = self._items_for_path(path_in_wpt)
         assert manifest_items is not None
-        if len(manifest_items) != 1:
-            return []
-        url = manifest_items[0][0]
-        if url[1:] != path_in_wpt:
-            # TODO(tkent): foo.any.js and bar.worker.js should be accessed
-            # as foo.any.html, foo.any.worker, and bar.worker with WPTServe.
-            return []
-        return [path_in_wpt]
+        return [item[0][1:] for item in manifest_items]

Looking to see what test coverage we have.

Ah, right -- tkent@, I think that there we were just waiting for wptserve to be enabled, right?

First stab at this in https://codereview.chromium.org/2767673002

Ran into a problem: TestExpectations needs to have an entry for 'foo.worker.html' for expectations, but that fails the lint check that the file exists. :(

I'll have to let qyearsley/jeffcarp figure out what the best path forward is.

Maybe the correct behavior of the TestExpecations file and lint test should now be changed so that the test doesn't have to be a file that exists -- a "test" could also be an item in MANIFEST.json. Some files that are "test files" are not actually tests, and some "test paths" aren't the same as their corresponding relative file path.

So for example, if MANIFEST.json contains:
"encoding/textdecoder-copy.any.js": [
  ["/encoding/textdecoder-copy.any.html", {}],
  ["/encoding/textdecoder-copy.any.worker.html", {}]
], ...

Then:
  run-webkit-tests external/wpt/encoding/textdecoder-copy.any.html
  run-webkit-tests external/wpt/encoding/textdecoder-copy.any.worker.html
should work, but
  run-webkit-tests external/wpt/encoding/textdecoder-copy.any.js
should not, and likewise,
  crbug.com/xxx external/wpt/encoding/textdecoder-copy.any.html [ Skip ]
  crbug.com/xxx external/wpt/encoding/textdecoder-copy.any.worker.html [ Skip ]
should work, but the same thing with ...any.js should make lint complain about "test doesn't exist".

This makes the idea of "test path" slightly more subtle, since it would no longer be the same as "relative file path from LayoutTests", so I'm not sure if this might break some assumptions somewhere.

Does this sound like a good change to make, in order to enable these WPT tests?

sgtm

We're going to need to educate blink engineers - how TestExpectations and RWT would work here was not obvious to me when enabling these. 

I guess the scenarios are:

* I see foo.any.html failing on the bots - how do I update TextExpectations?
  * Easy: you append an entry for foo.any.html 

* I see foo.any.html failing on the bots and want to reproduce it locally.
  * Easy: you RWT foo.any.html

* I see a failure on the bots and want to edit the test.
  * Harder: Need to educate non-WPT experts users about what .any.html means
  * We should add this to the LayoutTest docs and announce on blink-dev

* I imported foo.any.js and need to update TestExpectations
  * If a user adds foo.any.js they will get a lint failure: we could detect the .any.js/.worker.js case and make the linter emit a helpful "did you mean ...any.html and ...any.worker.html?"

* I made a change to foo.any.js and want to run it
  * For RWT I think we're silent if there's no matching file; that tends to bite me on a regular basis anyway. We could have a similar warning on no matching file, and add a helpful hint for .any.js/.worker.js?

I was wrong above, which may be confusing:

* I see foo.any.html failing on the bots and want to reproduce it locally.

Currently you need to run foo.any.js -- trying foo.any.html will give you "No tests to run"

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ee22b7dd342977e02e11f3cf0a6a10d99ad0499e

commit ee22b7dd342977e02e11f3cf0a6a10d99ad0499e
Author: Jeff Carpenter <jeffcarp@chromium.org>
Date: Mon Jul 10 22:19:18 2017

WPTServe: Run .any.js and .worker.js variations

This change is based on jsbell's crrev.com/2767673002.

When .any.js and .worker.js files are seen during MANIFEST generation
they are mapped to .any.html, .any.worker.html, and .worker.html files
as appropriate which WPTServe will generate on-the-fly. This allows
for easy testing of [Exposed=(Window,Worker)] APIs and tests that are
simply script with no boilerplate HTML. These are now supported in
run-webkit-tests.

When specifying tests to run, the source name should be specified, 
for example:

  $ run-webkit-test external/wpt/storage/interfaces.worker.js

But expectations will use the generated name, e.g.:

  external/wpt/storage/interfaces.worker-expected.txt

And failures will be reported under the generated name:

  external/wpt/storage/interfaces.worker.html

Bug:  653514 
Change-Id: I2f531e142c4ca41b1f3223a69a1e6f61343464c2
Reviewed-on: https://chromium-review.googlesource.com/461722
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Kent Tamura <tkent@chromium.org>
Commit-Queue: Jeff Carpenter <jeffcarp@chromium.org>
Cr-Commit-Position: refs/heads/master@{#485419}
[modify] https://crrev.com/ee22b7dd342977e02e11f3cf0a6a10d99ad0499e/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/ee22b7dd342977e02e11f3cf0a6a10d99ad0499e/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer.py
[modify] https://crrev.com/ee22b7dd342977e02e11f3cf0a6a10d99ad0499e/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/lint_test_expectations.py
[modify] https://crrev.com/ee22b7dd342977e02e11f3cf0a6a10d99ad0499e/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py
[modify] https://crrev.com/ee22b7dd342977e02e11f3cf0a6a10d99ad0499e/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py
[modify] https://crrev.com/ee22b7dd342977e02e11f3cf0a6a10d99ad0499e/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/test.py
[modify] https://crrev.com/ee22b7dd342977e02e11f3cf0a6a10d99ad0499e/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py
[modify] https://crrev.com/ee22b7dd342977e02e11f3cf0a6a10d99ad0499e/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_manifest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0dff02f926f63bd7fc14afe232f5a0eaf86d1f56

commit 0dff02f926f63bd7fc14afe232f5a0eaf86d1f56
Author: Keishi Hattori <keishi@chromium.org>
Date: Tue Jul 11 11:42:20 2017

Revert "WPTServe: Run .any.js and .worker.js variations"

This reverts commit ee22b7dd342977e02e11f3cf0a6a10d99ad0499e.

Reason for revert: Possibly causing Linux Trusty flake. See  crbug.com/740802 

Original change's description:
> WPTServe: Run .any.js and .worker.js variations
> 
> This change is based on jsbell's crrev.com/2767673002.
> 
> When .any.js and .worker.js files are seen during MANIFEST generation
> they are mapped to .any.html, .any.worker.html, and .worker.html files
> as appropriate which WPTServe will generate on-the-fly. This allows
> for easy testing of [Exposed=(Window,Worker)] APIs and tests that are
> simply script with no boilerplate HTML. These are now supported in
> run-webkit-tests.
> 
> When specifying tests to run, the source name should be specified, 
> for example:
> 
>   $ run-webkit-test external/wpt/storage/interfaces.worker.js
> 
> But expectations will use the generated name, e.g.:
> 
>   external/wpt/storage/interfaces.worker-expected.txt
> 
> And failures will be reported under the generated name:
> 
>   external/wpt/storage/interfaces.worker.html
> 
> Bug:  653514 
> Change-Id: I2f531e142c4ca41b1f3223a69a1e6f61343464c2
> Reviewed-on: https://chromium-review.googlesource.com/461722
> Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
> Reviewed-by: Kent Tamura <tkent@chromium.org>
> Commit-Queue: Jeff Carpenter <jeffcarp@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#485419}

TBR=jsbell@chromium.org,qyearsley@chromium.org,tkent@chromium.org,jeffcarp@chromium.org

No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  653514 
Change-Id: I685218eb840aa3d860a3709bc413d946240633d5
Reviewed-on: https://chromium-review.googlesource.com/566980
Commit-Queue: Keishi Hattori <keishi@chromium.org>
Reviewed-by: Keishi Hattori <keishi@chromium.org>
Cr-Commit-Position: refs/heads/master@{#485590}
[modify] https://crrev.com/0dff02f926f63bd7fc14afe232f5a0eaf86d1f56/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/0dff02f926f63bd7fc14afe232f5a0eaf86d1f56/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer.py
[modify] https://crrev.com/0dff02f926f63bd7fc14afe232f5a0eaf86d1f56/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/lint_test_expectations.py
[modify] https://crrev.com/0dff02f926f63bd7fc14afe232f5a0eaf86d1f56/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py
[modify] https://crrev.com/0dff02f926f63bd7fc14afe232f5a0eaf86d1f56/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py
[modify] https://crrev.com/0dff02f926f63bd7fc14afe232f5a0eaf86d1f56/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/test.py
[modify] https://crrev.com/0dff02f926f63bd7fc14afe232f5a0eaf86d1f56/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py
[modify] https://crrev.com/0dff02f926f63bd7fc14afe232f5a0eaf86d1f56/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_manifest.py