I wonder if this could have been caused by some of Stephen's work for crrev.com/569341

Looking

It looks related to my change. Not sure exactly why it's doing this. I'll revert my change to be safe though.

Do you all have examples of chromedriver CLs where a test failed, and CQ blocked you from committing? I think it's possible that it's never really done that, because the failures are all "expected", which my CLs fixed. 

It blocked +Yuly's CL

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win7_chromium_rel_ng/22045
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/78366

I'm suspicious of that CL, because it also has build config changes (changes chromium.gpu.fyi.json), which means that CQ doesn't retry without patch. And my suspicion is that the issue here has to do with the retry logic being weird.

I'm going to make a test CL which breaks the tests and see what CQ does.

As I suspected, it was broken before my change. Look at https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/125483, which is based on https://chromium-review.googlesource.com/c/chromium/src/+/1111386 (latest patchset). chromedriver python tests fail, but the CL still passes CQ. I think this is because of the unexpected test results stuff.

I'm going to land my CL which fixes this (https://chromium-review.googlesource.com/c/chromium/src/+/1111218), and then re-run the CQ on https://chromium-review.googlesource.com/c/chromium/src/+/1111386 to test if it fixes this.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a7643e7c4f47b9d3a1dbe6db44001ac83f59c636

commit a7643e7c4f47b9d3a1dbe6db44001ac83f59c636
Author: Stephen Martinis <martiniss@chromium.org>
Date: Fri Jun 22 21:05:34 2018

Reland "Chromedriver: Make test runner emit proper test results"

This is a reland of 9b237478addd0c5878309677530e78122393a1e0

Original CL was reverted just to be safe, due to https://crbug.com/855290.

There's a fix to a typo in the unexpected field; it should actually be
'is_unexpected', not 'expected'.

Original change's description:
> Chromedriver: Make test runner emit proper test results
>
> The chrome test result format needs failed tests to be marked as
> unexpected to be recognized as true failures. Otherwise they're thought
> to be acceptable failures, and don't show up as actual failures when
> parsed.
>
> Bug:  533481 
> Change-Id: Ica94a18a4503a5553ef5602627643ae526155e1a
> Reviewed-on: https://chromium-review.googlesource.com/1109269
> Reviewed-by: John Chen <johnchen@chromium.org>
> Reviewed-by: Caleb Rouleau <crouleau@chromium.org>
> Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#569341}

Bug:  533481 , 855290
Change-Id: I502ace9b1f0fd97d569fe6eb6c08f660218acb01
Reviewed-on: https://chromium-review.googlesource.com/1111218
Reviewed-by: Caleb Rouleau <crouleau@chromium.org>
Reviewed-by: John Chen <johnchen@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#569770}
[modify] https://crrev.com/a7643e7c4f47b9d3a1dbe6db44001ac83f59c636/chrome/test/chromedriver/test/run_py_tests.py

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/125510 fails. It includes my CL which landed, so it seems that my CL fixes this problem, which is good!

Sorry, but these 2 still pass, while they shouldn't:
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win7_chromium_rel_ng/22949
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/79291

Those tryjobs don't include my CL. I think CQ is trying to be smart here, and it's not re-running any tryjobs, since they ran on an earlier patchset. CQ doesn't think false positives really happen, and isn't programmed to handle them well.

I'll try to make it re-run, and more tryjobs should fail.

windows mac and linux are all failing in the latest CQ dry run on https://chromium-review.googlesource.com/c/chromium/src/+/1111386

Nice, thanks for fixing!