I have no idea why I marked this as started. Passing to secondary for triage.

Eric, looking at that CL, it's not at all clear what happened. Did you manually chump the change on the 20th?

Yeah, I didn't see a way to get the CLs in otherwise and got chump approval from the sheriffs on duty at the time.

I don't know how to reproduce the bug report. Here's what I see:

907588 depends on 7 other changes. Those 7 others all depend on 907588. All of them were rejected for actual, real failed CQ testing. Because they have to all land together, none did.

Eric chumped/cherry-picked all of them, instead.

Subsequently, 2 of the CL's were flagged for breaking the tree indicating that they really were bad:
https://chromium-review.googlesource.com/c/chromiumos/platform2/+/907637
https://chrome-internal-review.googlesource.com/c/chromeos/ap-daemons/+/565390

Please reopen if you see a bug or a think there's still something not right.

Both of those CLs were fine. The issue was only in local checkouts and there were no tree outages because of them.

I dug up an email chain with ihf@ about the first in which I suggested that re-emerging libchrome would fix the problem, and it did, which explains why there's no follow-on revert attempt or fix forward for that "breakage". The second CL similarly was broken on local checkouts and was fixed after re-emerging libchrome, and this was discussed in the comments there. A build on the CQ shouldn't run into these problems and indeed didn't on following runs.

I'm not even sure where the CQ testing would have failed. Can you point out to me the comments where commitbot actually tried them? The only time it looks like a dependent CL got "rejected" matches up with a comment on the chromiumos-overlay CL where it was rejected for infrastructure issues with the proxy server, and was going to be automatically retried.

It tried here: https://luci-milo.appspot.com/buildbot/chromeos/master-paladin/17759 . But it failed because of a lab outage.

Subsequently, you never got the CL's past the PreCQ because it looks like CIDB never got the signal that the CL's were successfully verified. At this point, it's virtually impossible to track down why that happened; a possible explanation is that the rebased version of the CL corrupted the CL Action History table for this CL. Another explanation would be that there was a CIDB connection problem in the middle of processing one of the PreCQ runs.

If it happens again, I'll remember this bug and dupe the new report. Sorry for the frustration.

Sounds good, thanks for the explanation.

If this sort of thing happens again, would it be easier just to abandon and reupload all of the relevant CLs? It sounds like that actually would have worked around the issue, and I wouldn't have had to chump a whole mess of patches... while the ones here weren't super complicated or high-risk, I anticipate having to land a bunch of stuff in parallel for the libchrome uprev and I'm not going to chump patches for that.

If this happens again, we'll be responsive and find a way to get the CL landed without abandon/reupload. That might involve manually modifying the CIDB record of the PreCQ state.