So, the reason is this this builder is marked as experimental, and CQ, when searching, doesn't "include_experimental". Similarly, Gerrit doesn't display these tryjobs, probably for the same reason.


Compare [1] vs [2].

https://apis-explorer.appspot.com/apis-explorer/?base=https://cr-buildbucket.appspot.com/_ah/api#p/buildbucket/v1/buildbucket.search?include_experimental=true&max_builds=100&tag=buildset%253Apatch%252Fgerrit%252Fchromium-review.googlesource.com%252F968671%252F16&fields=builds(bucket%252Cid%252Cparameters_json%252Curl)%252Cnext_cursor&_h=4&

[2] https://apis-explorer.appspot.com/apis-explorer/?base=https://cr-buildbucket.appspot.com/_ah/api#p/buildbucket/v1/buildbucket.search?max_builds=100&tag=buildset%253Apatch%252Fgerrit%252Fchromium-review.googlesource.com%252F968671%252F16&fields=builds(bucket%252Cid%252Cparameters_json%252Curl)%252Cnext_cursor&_h=5&

CL/PS for above example: https://chromium-review.googlesource.com/c/chromium/src/+/968671/16

And so the immediate cause is this config:
 experimental: YES
https://chromium.googlesource.com/chromium/src/+/master/infra/config/global/cr-buildbucket.cfg#1033

sounds like a general problem at buildbucket level. It is possible to configure a builder so that one schedules a build and does not see it results. This is weird API.

I am thinking we should remove experimental field from buildbucket config and instead add "this build should be buidbucket-experimental" configuration at CQ config level.

WDYT?

yes, this is confusing.

however, having CQ.cfg setting for buildbucket-experimental won't be applicable to git cl try/Gerrit-scheduled tryjobs.

we don't need that in git-cl-try and gerrit. We need it in CQ only because we are abusing CQ's experiment_percentage for the use case, so.. maybe we should not add "this build should be buidbucket-experimental" feature to CQ.

ehmaldonado, it seems that your abuse of CQ does not work very well. You may want to look for a different way to test your thing. Maybe led or resort to "if random() then do_new_thing()" in the bot_update itself?

What use case is experiment_percentage meant for?
And, how would I use led for that?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e3ddc869a492c2897dde2aa610f7bc13824a46a5

commit e3ddc869a492c2897dde2aa610f7bc13824a46a5
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Mon Apr 02 23:06:24 2018

cq.cfg: temporarily remove linux_chromium_rel_ng_patch_on_gclient.

While chrome operations are deciding how to rework how CQ deals
with buildbucket-experimental builders.

R=ehmaldonado@chromium.org, nodir@chromium.org

Bug:  828111 
Change-Id: I44c10d9328763cff1bdf609d89841e5aa5aa2f0f
No-Try: True
Reviewed-on: https://chromium-review.googlesource.com/991082
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#547549}
[modify] https://crrev.com/e3ddc869a492c2897dde2aa610f7bc13824a46a5/infra/config/branch/cq.cfg

Er... I don't think this is "abuse of the CQ"... The goal is to get production traffic to a builder with an experimental feature in such a way that we can query and see the jobs that ran the experiment.

"if random(): do thing" is particularly bad because it affects production jobs, with no control, and no way to observe the result.

Using led is fine (and a good idea), but doesn't give nearly enough data (without using some script to e.g. scrape all the outstanding CLs and generate some real-looking load).

CL to let CQ support such a use case, whether it's an abuse or not: https://crrev.com/i/601224

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/0f317abc6df39078dc32be143753e51e7b12cde5

commit 0f317abc6df39078dc32be143753e51e7b12cde5
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Apr 05 21:54:13 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/2bf37e735200ed81d2bab339edd297e1fa090a76

commit 2bf37e735200ed81d2bab339edd297e1fa090a76
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Apr 05 23:26:13 2018

Deployed to prod.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/d68a6fe2f2dd4f415636f4807d6892a83b7a3273

commit d68a6fe2f2dd4f415636f4807d6892a83b7a3273
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Mon Apr 09 23:25:07 2018