Possible fix: make base.py's test_exists() have similar logic to is_test_file() - maybe factor out is_manifest_entry()

How urgent is this? I did some work recently fiddling with MANIFEST.json so I can take this bug if nobody else wants it.

I think that taking this bug now would be helpful, since it would enable us to enable a bunch of wpt tests that we currently ignore -- feel free to take it :-)

As Joshua suggests in #1, I think this will involve changing Port.test_exists and Port.is_test_file, although those names are a bit confusing. It seems like maybe there should be a function to tell whether a given string is a valid test (which may not be a file in wpt, and should be a single test, not a directory), and another function to tell whether a given string is a test directory.

Cool - I started working on this but didn't get very far. FWIW, I ended up lazily populating a set of urls in wpt_manifest by iterating the manifest items, and exposing an is_wpt_url() method but didn't have time to get everything working.

FYI, I added my partial impl to https://codereview.chromium.org/2767673002 in case it's helpful. No tests though. :(

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0cbeffaef490b679c65e1f212a9c4922678e6002

commit 0cbeffaef490b679c65e1f212a9c4922678e6002
Author: Jeff Carpenter <jeffcarp@chromium.org>
Date: Tue Jul 11 00:42:05 2017

Add 4 TestExpecations from WebKit Linux Trusty Leak

After landing the CL enabling WPT tests that don't necessarily
correspond to real files (crrev.com/c/461722/), 4 Timeout 
TestExpectations from that CL turned up as Failures on WebKit Linux 
Trusty Leak. 

Bug:  703837 
TBR: qyearsley@chromium.org
Change-Id: Ideb2d49c866f04d5105c0ad30446b6c92124f0f2
Reviewed-on: https://chromium-review.googlesource.com/566019
Commit-Queue: Jeff Carpenter <jeffcarp@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Bret Sepulveda <bsep@chromium.org>
Cr-Commit-Position: refs/heads/master@{#485464}
[modify] https://crrev.com/0cbeffaef490b679c65e1f212a9c4922678e6002/third_party/WebKit/LayoutTests/TestExpectations

I landed this yesterday (crrev.com/c/461722/) along with a follow-up TestExpectations CL (above, crrev.com/c/566019/). However, it was reverted (crrev.com/c/566980/) in an attempt to diagnose  bug 740802 . It does not appear that the original CL caused  bug 740802 . Once that is resolved, I will attempt to re-land this CL.

 Issue 653514  has been merged into this issue.

There are a few ongoing issues causing flakiness in layout test land ( bug 741804 ,  bug 740802 ). I'm waiting until things cool down to re-land.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8d5218e690aad4bc8a45d7f11979f02a730f63ea

commit 8d5218e690aad4bc8a45d7f11979f02a730f63ea
Author: Jeff Carpenter <jeffcarp@chromium.org>
Date: Tue Jul 18 00:10:05 2017

Run WPT .any.js and .worker.js variations in run-webkit-tests

When .any.js and .worker.js files are seen during MANIFEST generation
they are mapped to .any.html, .any.worker.html, and .worker.html files
as appropriate which WPTServe will generate on-the-fly. This allows
for easy testing of [Exposed=(Window,Worker)] APIs and tests that are
simply script with no boilerplate HTML. These are now supported in
run-webkit-tests.

When specifying tests to run, the source name should be specified,
for example:

  $ run-webkit-test external/wpt/storage/interfaces.worker.js

But expectations will use the generated name, e.g.:

  external/wpt/storage/interfaces.worker-expected.txt

And failures will be reported under the generated name:

  external/wpt/storage/interfaces.worker.html

This change is based on jsbell's crrev.com/2767673002.

Bug:  703837 
Change-Id: I063b18f44e0460932c3fe8114420a327f1b1e088
Reviewed-on: https://chromium-review.googlesource.com/574645
Commit-Queue: Jeff Carpenter <jeffcarp@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Jeff Carpenter <jeffcarp@chromium.org>
Reviewed-by: Joshua Bell <jsbell@chromium.org>
Cr-Commit-Position: refs/heads/master@{#487304}
[modify] https://crrev.com/8d5218e690aad4bc8a45d7f11979f02a730f63ea/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/8d5218e690aad4bc8a45d7f11979f02a730f63ea/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer.py
[modify] https://crrev.com/8d5218e690aad4bc8a45d7f11979f02a730f63ea/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/lint_test_expectations.py
[modify] https://crrev.com/8d5218e690aad4bc8a45d7f11979f02a730f63ea/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py
[modify] https://crrev.com/8d5218e690aad4bc8a45d7f11979f02a730f63ea/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py
[modify] https://crrev.com/8d5218e690aad4bc8a45d7f11979f02a730f63ea/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/test.py
[modify] https://crrev.com/8d5218e690aad4bc8a45d7f11979f02a730f63ea/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py
[modify] https://crrev.com/8d5218e690aad4bc8a45d7f11979f02a730f63ea/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_manifest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/def8eab23c6425d654d0cbe8ced1b41680c6e98e

commit def8eab23c6425d654d0cbe8ced1b41680c6e98e
Author: Jeff Carpenter <jeffcarp@chromium.org>
Date: Tue Jul 18 02:55:05 2017

Reland "Add 4 TestExpecations from WebKit Linux Trusty Leak"

This is a reland of 0cbeffaef490b679c65e1f212a9c4922678e6002
Original change's description:
> Add 4 TestExpecations from WebKit Linux Trusty Leak
> 
> After landing the CL enabling WPT tests that don't necessarily
> correspond to real files (crrev.com/c/461722/), 4 Timeout 
> TestExpectations from that CL turned up as Failures on WebKit Linux 
> Trusty Leak. 
> 
> Bug:  703837 
> TBR: qyearsley@chromium.org
> Change-Id: Ideb2d49c866f04d5105c0ad30446b6c92124f0f2
> Reviewed-on: https://chromium-review.googlesource.com/566019
> Commit-Queue: Jeff Carpenter <jeffcarp@chromium.org>
> Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
> Reviewed-by: Bret Sepulveda <bsep@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#485464}

TBR: qyearsley@chromium.org
Bug:  703837 
Change-Id: I23f301716c62c67976b5ff9b4d26bfc80f57bc66
Reviewed-on: https://chromium-review.googlesource.com/575590
Reviewed-by: Jeff Carpenter <jeffcarp@chromium.org>
Commit-Queue: Jeff Carpenter <jeffcarp@chromium.org>
Cr-Commit-Position: refs/heads/master@{#487359}
[modify] https://crrev.com/def8eab23c6425d654d0cbe8ced1b41680c6e98e/third_party/WebKit/LayoutTests/TestExpectations

Aside from a few new TestExpectations that had to be added, the most recent attempt to land this (crrev.com/c/574645) looks like it's stable!!

These types of tests now work on Chromium master:
./run-webkit-tests external/wpt/encoding/textdecoder-copy.any.worker.html -> Found 1 test
./run-webkit-tests external/wpt/encoding/textdecoder-copy.any.js -> Found 2 tests

I want to apologize for this change taking so long to land, I was stumped for longer than I should have been by a logic error in my CL. Let me know if the change is working out for you and please re-open this bug if anything comes up!

Yay! Excellent, just confirmed that those two example invocations work.

Note though, that currently blob matching doesn't work for wpt tests the way it works for other tests, e.g.

  ./run-webkit-tests 'external/wpt/encoding*' -> Found 0 tests
  ./run-webkit-tests 'fast/forms/validation*' -> Found 5 tests

For consistency, it would be nice to either make blob matching work, or remove support for it entirely. Any thoughts?