If I understand correctly, this was caused in following way:

1. CLs are picked up in master-paladin/17926.
2. CLs are tested in slave builder, e.g. caroline-paladin/2813, eve-arcnext-paladin/156, etc.
3. Some slave builders succeed (e.g. caroline-paladin/2813), others fail (e.g. eve-arcnext-paladin/156), so CLs are not submitted.
4. In the next run, succeeded slave builders do incremental build. So the chroot has packages of not-yet-submitted version.
5. Since new ebuilds are not written to handle such situation, build fails.

I believe the fundamental flaw here is slave builders doing incremental builds with not-yet-committed version packages. IIUC, slave builders do incremental builds when its previous run was green; it does not care the status of master builder. As long as we keep this behavior, same failures will continue to happen.

Thank you nya-san.

So sounds like this is generic build infra related issue, rather than caused by my CL itself.
pprabhu@, Could you unblock my CLs by removing -1 verified?

ping?

OK, I buy #1. Thanks for that analysis. Makes sense to me.

dgarrett@ tells me there is an old bug about this, I'm trying to find it to attach here.

Sending these CLs back at the CQ is not an option, since it can fail in the same way again. imho, chumping after enough sanity testing might be the lesser of the two evils here.

+nxia, can you find the bug about inconsistent clobber on slaves?
I can't find it right now, and Don tells me you had it for a while.

Also vapier@: Can you confirm if our reading of this situation is correct. This isn't about just a missing dependency blocker? (These are both cros-workon packages, so I'm guessing the answer is no)

what's the inconsistent clobber ?

I thought the CQ only used the incremental chroot when the last CQ-master was green, was that changed? 

I think the analysis in #1 is correct.  When I turn on snapshots, I'm planning to tie them to the master-paladin status instead of the local status, but I think all the builders currently look at their local status.

I believe they do look at local status, and that this is a long standing bug.

I've removed my verified bit from those CLs, and recommendation is to chump :'(

Making this bug the more general ask, because I can't find the old bug.

+jclinton, FYI.

dgarrett to own or delegate.  It's a bug, it is significant, but probably not urgent from the sound of things.  I have no idea how much a fix would cost, but dgarrett might.

Do you know which code clobbers chroot when the last build was red? Can we make it consider the master status by querying CIDB for example?

FYI: Re #4, now CLs are chump'ed.
CL:923617, CL:923926, CL:*572280, CL:*572281.

I will try to take a look at this next week.

I recommend having the master pass "--clobber" on the command line to the slaves when scheduling them. That puts ownership of the decision in the right place.

I've been discussing offline with people.  We're going to add a --revert-chroot option that is less expensive than a full clobber.  I'll take this bug since I've already been looking at it.

Now that crrev.com/c/1037193 is in, slaves will check the previous master's status before reusing a chroot.  The scenario in this bug should no longer be possible.

 Issue 796653  has been merged into this issue.