Okay, so all builders show this error as soon as they included the commits ranging from r2dc2e737 to aed99d7e. If the error wasn't introduced in a configuration before that, this is the regression range.

It's unlikely that the error was introduced before the first failing commit as there is no set of "last-known-good" CLs for this issue. If an earlier CL is the root, it must have landed before r5534e46f.

Now my CL cannot be committed due to this error on the try bots. I'm not sure other people's situation, but I'd like to give this higher priority. Thank you for triage!

Hmm, I can't find a good suspect. 

Complete list (oldest to newest suspect):
https://crrev.com/c/1372990 - Adds a dependency in a build file - nothing removed here.
https://crrev.com/c/1358615 - Name change of a histogram.
https://crrev.com/c/1373049 - Changes only comments.
https://crrev.com/c/1372831 - Changes a single test JS file.
https://crrev.com/c/1370486 - Changes mainly expectations and a one-line rounding hack. (Someone reverted it: https://crrev.com/c/1373285 ==> It showed no effect and Google Chrome ChromeOS fails still)

I'll check older CLs for culprits. As this is pretty hopeless: if someone knows how to create a fix for this kind of issue, please go ahead and take this.

Setting to P-0 as this renders a lot of builders completely useless, blocks CLs from landing and should absolutely take priority over any other bug.

Okay, Now "Google Chrome OS" and "android-rel" have been cycling green [1].
Let's wait for android-dbg to see whether this was the solution and whether it persists.

Or has someone touched the bots manually to fix this?


[1] android-rel with https://crrev.com/c/1373285 only and Google Chrome OS right after this commit)

It apparently got worse: The "green" builds aren't really green. They just ignore the failure.
So this opens another question: How is that possible?
Isn't the bot expectation set _somewhere_ in the chromium repo?
(https://crrev.com/c/1373285 was the first to cycle green, but it's only a revert and doesn't touch any bot config file)

This is fixed by
https://chromium-review.googlesource.com/c/chromium/tools/build/+/1373509

Okay, to record hat happened:

1. Quite a while ago, it was decided to ignore failures in this step.
2. https://crrev.com/c/1371350 stated these failures are rare and should be blocking.
3. This broke mac_asan and win_x86_msvc_rel [1]
4. https://crrev.com/c/1373509 reverted the CL from 2 to ignore these failures again.
5. Failures still exist and they are ignored as expected.

The observed red cycles happened between 2 and 4 but the changes in 2 & 4 were invisible in the regular tree as they happened in the sub-repo https://chromium.googlesource.com/chromium/tools/build/.

[1] 
https://ci.chromium.org/p/webrtc/builders/luci.webrtc.try/mac_asan/15219
https://ci.chromium.org/p/webrtc/builders/luci.webrtc.try/win_x86_msvc_rel/489