AFAIK, CQ does check the label before submitting, but the committing process can take significant amount of time, so it could be that you've removed the label after the check has happened. I'll try to investigate the logs to confirm that.

Based on [1], looks like CQ has started to submit the CL at 20:08:04.736 by issuing a POST request to the following URL: https://chromium-review.googlesource.com/a/changes/617540/revisions/df42a29c7cbed925f868ca6ce2918da3cc216036/submit. This is later than CQ+2 was removed. So I did further code analysis.

Rietveld PM used to check presence of the CQ bit in the _last_minute_checks method, but Gerrit PM does not actually do that: https://chrome-internal.googlesource.com/infra/infra_internal/+/e7a4e44fc7c2349482a78d47eb0c733476c3e00f/infra_internal/services/cq/pending_manager/gerrit.py#807. This means that we'd only notice removed CQ+2 if the CQ has cycled at least once after it and noticed that Gerrit does not return the issue among those to be commited. In that case, unless the issue has already been moved to committer thread, we will stop the committing process. Based on [2], CQ loop used to take about 2 minutes on that day, so looks like CQ just didn't notice removed CQ+2 and committed the change anyway.

We can o'course improve this by adding checks into _last_minute_checks, but that would only reduce the time for CQ+2 to be noticed by CQ. There is no way to eliminate the race completely without support from Gerrit API and it seems like it's not on their priority list. Andrii, can you post a link to your feature request?


[1]: https://pantheon.corp.google.com/logs/viewer?project=chrome-infra-logs&minLogLevel=0&expandAll=false&resource=gce_instance&filters=text:139983154067200&logName=projects%2Fchrome-infra-logs%2Flogs%2Fcq-chromium__gerrit__&timestamp=2017-08-16T18:08:10.071655000Z

[2]: https://pantheon.corp.google.com/logs/viewer?project=chrome-infra-logs&minLogLevel=0&expandAll=false&resource=gce_instance&filters=text:%22CQ%20loop%20starting%22&logName=projects%2Fchrome-infra-logs%2Flogs%2Fcq-chromium__gerrit__

Given that 2 minutes is very small compared to 1 hour that it normally takes CQ to land a CL in Chromium, I think this is low priority. Please let us know if you think otherwise.

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot