cq issue, not a trooper issue

So, I went really deep and build proper tool for looking into cq logs https://chromium-review.googlesource.com/c/356650/1

Then I finally got hold of actual log files. No relevant exceptions there, here is the crux: https://storage.cloud.google.com/chrome-dumpfiles/xsoqq32kdy

I can't infer how big dataset buildbucket has been returning, but at least the presubmit job was there and it was skipped due to dry_run flag (set by git  cl try) as expected:

2016-06-27 07:04:40PM: [pid:18262 tid:140036787918592 buildbucket_util:392] Skipping bucket result 9008688655462181008 for issue 2091763004 patchset 20001: extra properties: [u'dry_run'].

After that, CQ decides to schedule what seems like all the builders in cq.cfg. I have no idea why.

I just saw this again
hitting CQ on https://codereview.chromium.org/2465813002/#ps40001


It just started 
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/323057

even though the change just had a green try run
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/323003


I think what happened is that the original "git cl try" had started
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/322877
which failed with a flake. So in the Rietveld UI, I manually selected win_chromium_rel_ng which started the 2nd link above. For some reason the CQ is not using it though. 

jam@ thanks for reporting. Pawel, can you please look at this now that we have all the extra logging enabled?

Nodir, could you check my reasoning here?

Buildbucket seems not to be returning http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/323003 .

Please see most recent [json] in https://chromium-cq-status.appspot.com/recent#issue=2465813002,patchset=40001,codereview_hostname=codereview.chromium.org,action=verifier_jobs_update .

i don't see build in 8997184818223216656 most recent  [json] in https://chromium-cq-status.appspot.com/recent#issue=2465813002,patchset=40001,codereview_hostname=codereview.chromium.org,action=verifier_jobs_update .
i see the build in https://apis-explorer.appspot.com/apis-explorer/?base=https://cr-buildbucket.appspot.com/_ah/api#p/buildbucket/v1/buildbucket.search?tag=buildset%253Apatch%252Frietveld%252Fcodereview.chromium.org%252F2465813002%252F40001&_h=1&, but probably it wasn't there when CQ checked it

root cause seems to be datastore indexes update latency (WAI), so not relying on them may improve the situation, so I will make this bug blocked on  issue 661689 

Consistent search by buildset is deployed.

Thanks, let's close as fixed.

FWIW, this is where I suggested the idea: https://goto.google.com/zltyb .