I was trying to test a set of CLs (27 total, 23 of which are 1-6 line changes in private overlays). After setting all of them to trybot-ready +1, they get picked up by a pre-cq run.

The full set of CLs involved are:

https://chromium-review.googlesource.com/q/status:open+author:saklein%2540chromium.org+freon

https://chrome-internal-review.googlesource.com/q/status:open+author:saklein%2540chromium.org+freon


Expected Results:
They should be applied and tested, and the results reported.

Actual Results:
They appear to (sometimes) be hitting a quota or rate limit in gerrit, so the trybots can't apply all of the CLs and the runs fail there.


e.g.
Build: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8938721154603966656

Relevant Logs: https://luci-logdog.appspot.com/v/?s=chromeos/buildbucket/cr-buildbucket.appspot.com/8938721154603966656/+/steps/PreCQSync/0/stdout


It does work sometimes (e.g. https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8938721151499413568), but fails at least enough the CLs can't pass all the trybots or it's preventing other issues from surfacing.




A potential side effect or related issue is a delay in messages and trybot-ready flag reset. This causes confusion about the actual state of the CLs and whether they can be retested or are currently being tested. It is unclear if there is a causal relationship here, but since it's happening relatively frequently on the same batch CLs, there is at least a strong correlation between the issues.

e.g. https://chromium-review.googlesource.com/c/chromiumos/overlays/eclass-overlay/+/1165943

2018-08-08 17:21: CLs fail and are reset to trybot-ready=0.
2018-08-09 09:08: I set it back to trybot-ready=1.
2018-08-09 09:09: Messages are received about 1) it being picked up by the pre-cq, 2) it timing out after 240 minutes, and 3) it being set back to trybot-ready=0 again.

The runs linked in the pre-cq pickup message are from the previous evening. The same 1 minute failure behavior can also be seen in other CLs around 4:30 yesterday afternoon (2018-08-08) after I fixed some bugs in the CLs that day that were found in the runs the previous day.