I got curious when I saw  bug 807791  and its related commits, as the bug should not have happened with our WPT import infrastructure (i.e. import a test but not rebaseline it).

Looking at the import CL (https://chromium-review.googlesource.com/c/chromium/src/+/895811/1) it's possible to see that none of the *_blink_rel trybots reported a failure in external/wpt/html/dom/interfaces.html, even though the failure is present in each bot's layout-test-results.zip, outdir_json and failing_results.json in Google Storage.

It turns out https://chromium-review.googlesource.com/c/chromium/src/+/834758 (from January 10th) marked external/wpt/html/dom/interfaces.html with NeedsManualRebaseline, so the expectations file hadn't been updated since then.

While the import CL was being processed, https://chromium-review.googlesource.com/c/chromium/src/+/874802 ("Rebaseline WPT tests after harmony-function-tostring",  bug 753073 ) was also being worked on. It removed external/wpt/html/dom/interfaces.html from TestExpectations and rebaselined it without the change from https://chromium-review.googlesource.com/c/chromium/src/+/895811 (which hadn't landed yet).

Robert's harmony-function-tostring CL landed at Jan 31 2018 22:52:11 CET, whereas layout tests in the WPT import CL finished running before that and the CL landed at Jan 31 2018 23:23:51 CET. In other words, we had the right expectations for the test for about 30min, when a new version of html.idl was landed without the right expectation due to the "race condition we" had there.

Is there anything we could do to prevent this from happening again?