Relevant comment from the original thread:

robertma@:
"More background: the most inner runner (single_test_runner) doesn't actually read manifest at all. Instead, it determines if a test uses testharness.js based on its output... Namely, it looks for the "This is a testharness.js-based test." header, which is added by our testharnessreport.js automatically.

Therefore, if a test produces empty output (or content_shell considers prematurely the test has finished), the runner would interpret the test as a reftest.

I still struggle to understand why this is even possible. Our testharnessreport.js calls `testRunner.waitUntilDone()` immediately when loaded, so it'd be a timeout if a test doesn't finish properly. And when a tests finishes, testharness.js calls the completionCallback in testharnessreport.js, which will print the header before calling `testRunner.notifyDone()`, so it's weird that the header isn't found.

testharnessreport.js: https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/resources/testharnessreport.js

Note: for WPT, we use the upstream testharness.js at https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/external/wpt/resources/testharness.js, *NOT* the copy in LayoutTests/resources."

In the case of https://chromium-review.googlesource.com/c/chromium/src/+/882002, I would suspect any weirdness of being the same as what I fixed in https://github.com/w3c/web-platform-tests/pull/9598.

Since we do update incrementally from WPT_BASE_MANIFEST.json, it seems very likely that it would have happened immediately after this was landed.

If some tests are being *randomly* interpreted as of the wrong type, that might be a different problem though.

That changes in https://github.com/w3c/web-platform-tests/pull/9598 need to be imported, and I don't recall what the procedure is. Maybe we should make it part of the rotation and semi-automated :)

Based on the log for third_party/WebKit/Tools/Scripts/webkitpy/thirdparty/wpt/wpt/tools/manifest/manifest.py, it hasnt been updated since Sept 25 2017:

commit 26a8ae1db8e8caa1f7c8b84611d6c72948120320
Author: Robert Ma <robertma@chromium.org>
Date:   Mon Sep 25 15:31:56 2017 +0000

    Roll in new version of wpt tools and modify callers correspondingly
    
    * Updated the versions in WPT_HEAD in checkout.sh and README.chromium.
    * All wpt commands now have a single entry point, wpt. Include this
      dispatch module in WPTWhiteList.
    * The commands for generating manifest, starting wptserve and running
      wpt lint are now "wpt manifest", "wpt serve" and "wpt lint". Change
      webkitpy and PRESUBMIT.py accordingly.
    * The new wptserve contains a change for
      fetch/api/abort/general-serviceworker.https.html
      (https://github.com/w3c/web-platform-tests/pull/6484),
      so the test is rebaselined.
    
    Bug:  749879 
    Change-Id: I214860078add1c391266f063193aa279db414bb0
    Reviewed-on: https://chromium-review.googlesource.com/677557
    Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
    Commit-Queue: Robert Ma <robertma@chromium.org>
    Cr-Commit-Position: refs/heads/master@{#504060}


Not sure if it was automated at all or just manual?

It's updated manually.

I don't think the manifest is used by the test runner to determine reftests vs. jsharness tests, but I could be wrong. I'll have to double check.

Meanwhile, let me do another roll of the wpt tools in case that solves the problem. (And it's been long enough since the last roll so we should update them anyway.)

I'm thinking about one potential fix: modifying the test runner to rely on the manifest instead of the test output to determine the type of a test in WPT. (Unfortunately, we still need to keep the original output-based checks for non-WPT layout tests.)

Although Stephen's CL (https://crrev.com/c/951662) seems to stick this time, acid/acid3/numbered-tests.html (issue 805463) is still flaky and the reland was reverted by sheriffs.

I suspect the bug in delta updating manifest is not the root cause (or is only part of the story). Still need to investigate further.

robertma@chromium.org ping from your friendly neighbourhood Ecosystem Infra rotation :-)

robertma@chromium.org ping from your friendly neighbourhood Ecosystem Infra rotation. Does this bug disappear into the ether when other people are on rotation?

The act of pinging bumps the modified date of the bug so that it disappears for another 60 days.

Rotation ping :)