So, looking into this a bit more:

Previously, our logic for determining whether to import the test was:

 1. Use TestParser to look at the file. If it looks like a testharness test (because it uses testharness.js), then import it. Otherwise, if it looks like a manual test (because the filename includes -manual), then don't import it.
 2. Then, if it was imported, try to run it regardless of what the MANIFEST says it is.

However, this isn't ideal because it's better to just have just one place in charge of deciding what type of test a file is, and whether it's a test, and it would be best if the manifest script could do that.

Proposal: Change the wpt manifest script to only consider tests as manual as a last resort (if they don't look like testharness or reftests), and then don't run any tests that are listed as manual tests.

This would basically mean that a test would not be run if it doesn't have <script src="/resources/testharness.js"> or <link rel="match"...>, but it would be imported and listed as a "manual" test in the manifest. The pointerevents/ tests should still be run I think since they use testharness.js.

Start of a related CL: https://codereview.chromium.org/2798663002

Can we update the importer so that adding [ Skip ] expectation for tests of which names match to *-manual.* and try results are Timeout or Missing?

In Chromium project, we have three classes of WPT manual tests:
A) Using testharness.js, and we have wpt_automation for the test
B) Using testharness.js, and we have no wpt_automation
C) Using no testharness.js

We imported A+B, and had to exclude B manually.  Even if we can distinguish A/B and C by looking MANIFEST.json, we still need manual exclusion for B.  It would be nice to have a way to skip/exclude B+C automatically.

Good point.

This will mean that we'll import all of the tests (A+B+C), and have Skip expectations for B+C, which will probably end up being 100s of lines in TestExpectations. This might not be ideal in the long run, but I think it's a good way to proceed for now.

Made CL https://codereview.chromium.org/2799683003.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2a55b708afc1a74f2f120d1c473a5f0673a1974f

commit 2a55b708afc1a74f2f120d1c473a5f0673a1974f
Author: qyearsley <qyearsley@chromium.org>
Date: Thu Apr 06 17:40:45 2017

Add skip expectations for new manual tests with a missing or timeout result.

BUG= 708241 

Review-Url: https://codereview.chromium.org/2799683003
Cr-Commit-Position: refs/heads/master@{#462533}

[modify] https://crrev.com/2a55b708afc1a74f2f120d1c473a5f0673a1974f/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater.py
[modify] https://crrev.com/2a55b708afc1a74f2f120d1c473a5f0673a1974f/third_party/WebKit/Tools/Scripts/webkitpy/w3c/wpt_expectations_updater_unittest.py

So, for now, skip expectations will be added, and this should be OK, at least for now.