Why do we need extra trybots? Are the tests not part of the CQ?

*Dirk: we currently don't have any Android coverage on CQ, I believe

Looks like the Android breakage is not too often, so we can wait for the webkit test suites to be enabled on android-kitkat-arm-rel for now.

John: can you point out the bug to add webkit tests to android-kitkat-arm-rel?

We have a whole bunch of android configs in the CQ. Perhaps the needed tests should be in one of those.

Also, you should figure out if the blinkpy unit tests are running, and whether they'd have got these issues. They're obviously far more lightweight than running the layout tests and we should be able to run them everywhere (and they should be catching these failures; if not, we're missing test coverage).

#5: I agree that blinkpy unit tests should catch all the test breakage. But in reality, it's hard to expect blinkpy unit tests's coverage == third_party/blink/tools/run_web_tests.py. So by the time we know we miss coverage, it's already a bit late.

Historically, the test coverage would catch most if not all bugs in the python code, so if it's not now, you need more :).

I concur with #7, based on my experience, this statement is generally true; the coverage of blinkpy tests is pretty good.

In this particular case, we just didn't foresee all the combinations of target & configuration actually used in the wild, and the unit tests missed this edge case unfortunately.