AFAICS, all the related CLs have failures well-hidden under the "experimental tryjobs" section (win7_chromium_rel_loc_exp for those CLs). This has also bitten me before.

Currently, the importer tries to check the CQ try job results by checking the results of all try jobs that are not "blink try bots", e.g. *blink_rel.

https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_importer.py?l=231

This is kind of a hack which mostly works. What we really want is either a way to check directly whether the CQ passed, or else we want to get a list of non-experimental CQ bots.

rakuco: I didn't know there was such a thing as "experimental tryjobs" (and Gerrit actually has a button to show them). Learned something new today.

Apparently Gerrit knows which tryjobs are experimental.

The thing on Gerrit that shows the tryjob results is the Buildbucket plugin, which knows which jobs are experimental because Buildbucket knows and the plugin fetches the build info from Buildbucket: https://chromium.googlesource.com/infra/infra/+/master/appengine/cr-buildbucket/doc/index.md

Incidentally, git cl try-results also gets results from Buildbucket... and there are multiple places where it could indicate that the build is a CQ tryjob, e.g. from "tags". First, this would need to be included in the JSON produced by git cl try-results --json: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/819573

That WIP CL looks very promising! Thanks, Quinten!

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/a563d7222f84f8c17a2aef5fd8353f85c7386fb1

commit a563d7222f84f8c17a2aef5fd8353f85c7386fb1
Author: Quinten Yearsley <qyearsley@chromium.org>
Date: Tue Dec 12 18:17:15 2017

Add more fields to try-results --json for WPT importer

Context: The WPT importer consumes the JSON produced by
git cl try-results is consumed and uses it to determine
when the CQ has passed (whether all non-experimental cq
jobs were successful) and where layout test results are
stored.

Added fields:
  created_ts: This could be used to sort builds by time.
  tags: Includes user_agent tag and experimental tag, as well as
  experimental: whether the job is experimental

Bug:  792652 
Change-Id: Ic6099a21b21db668b88b1a8e54aefd84529a2a91
Reviewed-on: https://chromium-review.googlesource.com/819573
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/a563d7222f84f8c17a2aef5fd8353f85c7386fb1/tests/git_cl_test.py
[modify] https://crrev.com/a563d7222f84f8c17a2aef5fd8353f85c7386fb1/git_cl.py

Next step: Change common.net.git_cl to support checking only CQ jobs. To check only CQ jobs, we want to check jobs with "user_agent:cq" in tags and "experimental" set to False.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/140c584f103b88eb4db2061789e93c4e2ca36264

commit 140c584f103b88eb4db2061789e93c4e2ca36264
Author: Quinten Yearsley <qyearsley@chromium.org>
Date: Mon Dec 18 23:15:30 2017

WPT Import: when checking CQ status, only check CQ builds

This CL changes GitCL.wait_for_try_jobs and GitCL.try_job_results
to take an optional boolean argument called cq_only, which causes
these methods to only return results for jobs that are triggered
by the CQ and are non-experimental.

This is used in the wpt importer when checking CQ status. After
this, the expected result is that the importer should consider
the CQ to have passed even if there are failing experimental
jobs etc.

Bug:  792652 
Change-Id: I09f89e917b2a97f02df347ac803c78b228b6b744
Reviewed-on: https://chromium-review.googlesource.com/828140
Reviewed-by: Robert Ma <robertma@chromium.org>
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#524849}
[modify] https://crrev.com/140c584f103b88eb4db2061789e93c4e2ca36264/third_party/WebKit/Tools/Scripts/webkitpy/common/net/git_cl.py
[modify] https://crrev.com/140c584f103b88eb4db2061789e93c4e2ca36264/third_party/WebKit/Tools/Scripts/webkitpy/common/net/git_cl_mock.py
[modify] https://crrev.com/140c584f103b88eb4db2061789e93c4e2ca36264/third_party/WebKit/Tools/Scripts/webkitpy/common/net/git_cl_unittest.py
[modify] https://crrev.com/140c584f103b88eb4db2061789e93c4e2ca36264/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_importer.py
[modify] https://crrev.com/140c584f103b88eb4db2061789e93c4e2ca36264/third_party/WebKit/Tools/Scripts/webkitpy/w3c/test_importer_unittest.py

Fix CL now in.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d0b9283b286a0ccf335643a399664ad5d276f680

commit d0b9283b286a0ccf335643a399664ad5d276f680
Author: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Date: Thu Mar 15 16:12:28 2018

WPT Import: Use the "cq_experimental" tag to check for experimental builds

We were previously using the "experimental" field in the JSON returned by
`git-cl try-results`, but that field actually corresponds to whether a
recipe is running in experimental mode. See e.g.

    https://cs.chromium.org/chromium/infra/recipes-py/recipe_modules/runtime/api.py?q=is_experimental&sq=package:chromium&l=27&dr=CSs

What we need is the "cq_experimental" tag that really indicates whether a
trybot is scheduled in an experimental commit queue per

    https://cs.chromium.org/chromium/infra/go/src/infra/libs/bqschema/buildevent/completedbuild.go?type=cs&sq=package:chromium&l=31

This tag is also used by the Buildbucket Gerrit plugin to split bots into
experimental and non-experimental ones.

Bug:  792652 ,  821445 
Change-Id: I267bf033729b6a74382d018cb3da06c64dc7b207
Reviewed-on: https://chromium-review.googlesource.com/964121
Reviewed-by: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Cr-Commit-Position: refs/heads/master@{#543385}
[modify] https://crrev.com/d0b9283b286a0ccf335643a399664ad5d276f680/third_party/WebKit/Tools/Scripts/webkitpy/common/net/git_cl.py
[modify] https://crrev.com/d0b9283b286a0ccf335643a399664ad5d276f680/third_party/WebKit/Tools/Scripts/webkitpy/common/net/git_cl_unittest.py