foolip@, could you PTAL? (since you authored https://github.com/w3c/web-platform-tests/pull/11004 and also since I think you have more experience than me working with WPT and how one is supposed to use reference documents in tests)

FWIW, img-html-correctly-labeled.sub-ref.html and img-html-correctly-labeled.sub.html are almost identical - the only difference is that the URL in the <img> tag is cross-origin in the test file and is same-origin in the reference file.  Let me know if you have any questions about CORB or these specific tests.

FWIW, I've started a CL here: https://crrev.com/c/1066816 (but I don't actually understand what broke the test / wouldn't know how to fix it other than reverting CORB-related part of https://github.com/w3c/web-platform-tests/pull/11004)

The import that disabled these tests:
https://chromium.googlesource.com/chromium/src/+/a0f57ae46a4326ecae11e13f1edb408e79476e20

In that import, expectations were changed based on results from https://chromium-review.googlesource.com/c/chromium/src/+/1059309/1, for example:
https://test-results.appspot.com/data/layout_results/linux_trusty_blink_rel/29100/layout-test-results/results.html
https://test-results.appspot.com/data/layout_results/win10_blink_rel/7267/layout-test-results/results.html

(I think the rest are the same, didn't check all of them.)

OK, so what changed is that now the expected screenshot shows a blue image, where before both expected and actual was a blocked image, a blue/white box with a question mark in it.

As it turns out these three tests all use the same ref:
external/wpt/fetch/corb/img-html-correctly-labeled.sub.html
external/wpt/fetch/corb/img-png-mislabeled-as-html-nosniff.tentative.sub.html
external/wpt/fetch/corb/img-png-mislabeled-as-html.sub.html

And that ref is img-png-mislabeled-as-html.sub-ref.html. Verified by appending a unique string to each of the refs and seeing what showed up in the "expected" rendering when they failed.

When the files were called -expected.html, something outside of the wpt manifest handling took precedence, and they were matched by filename instead. robertma@, do you know what this something else, and should we consider it a bug? Or perhaps just add a wpt lint disallowing files to be called *-expected.* to avoid this kind of problem?

The fix will be to use the right refs.

Sent https://chromium-review.googlesource.com/c/chromium/src/+/1065991.

Thank you very much for the investigation and the CL with the fix - much appreciated!

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a843b673246d7978981ce847f5a6250790ceec39

commit a843b673246d7978981ce847f5a6250790ceec39
Author: Philip Jägenstedt <foolip@chromium.org>
Date: Mon May 21 14:52:39 2018

Fix CORB tests using the wrong references

Problem uncovered by a recent change in WPT, see:
https://bugs.chromium.org/p/chromium/issues/detail?id=844781#c5

Bug:  844781 
Change-Id: I6d3a51e7673ef28128b0ede4cbb0450a299730b5
Reviewed-on: https://chromium-review.googlesource.com/1065991
Reviewed-by: Łukasz Anforowicz <lukasza@chromium.org>
Commit-Queue: Philip Jägenstedt <foolip@chromium.org>
Cr-Commit-Position: refs/heads/master@{#560268}
[modify] https://crrev.com/a843b673246d7978981ce847f5a6250790ceec39/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/a843b673246d7978981ce847f5a6250790ceec39/third_party/WebKit/LayoutTests/external/wpt/fetch/corb/img-html-correctly-labeled.sub.html
[modify] https://crrev.com/a843b673246d7978981ce847f5a6250790ceec39/third_party/WebKit/LayoutTests/external/wpt/fetch/corb/img-png-mislabeled-as-html-nosniff.tentative.sub.html

*-expected.html does have special meaning in both Blink and WebKit test runners. Philip's comment in #5 is basically correct. This behaviour is confusing in external/wpt with a few potential failure modes:

1. Test authors create tests with wrong ref targets, but due to the magic filename, tests seem to work in Chromium and will fail when exported.
2. By accidentally naming a test (reference) file as *-expected.html, it becomes the reference of some other test unintentionally and hence breaks that test.

Let me see if I can disable this magic suffix in external/wpt in Chromium, which is my preferred solution. If that isn't easy, we can consider implementing lints (either in Chromium or WPT upstream).

@foolip, has this been fixed?

Yes!