Aviv, this a a very reasonable request, but complicated enough to need design work, especially around the build request mechanism.

Not entirely sure what is meant, but I think I would invert the request here.

It seems a lot cleaner if only the master ever generates and commits a new manifest version (and increments the version counter).

Instead, sounds like what we want is the ability to have the master cut these additional versions but have the only kickoff a subset of the usual slaves.

Anyway, I'm not sure I understand the original request. Clarification:

> 1) Master was invoked and ManifestVersionSync set to N.N.N.N
> 2) Individual BoardA is triggered e.g. next day

I.e. with a force build for that particular build? What is the current behavior here? The behavior I would like to see is either that it builds the most recent manifest version or an explicitly passed in manifest version, or it just fails and says "can't kick me without a master".

> 2.5) New CLs are merged aka. A_CLS
> 3) Version for BoardA is bumped to N.N.N.N+1

Bumped how?

> 4.5) New CLs are merged aka. B_CLs
> 4) Indivual boardB is triggered 

Triggered how?

> 5) Version for BoardB stays at N.N.N.N+1 

> I.e. with a force build for that particular build? What is the current behavior here? The behavior I would like to see is either that it builds the most recent manifest version or an explicitly passed in manifest version, or it just fails and says "can't kick me without a master".

I believe the slave will generate a new version number, however it does not commit the manifest properly.

Everything 'works' except for the manifest part, so if you look back at the tools that analyze it like the crosland utility they are not necessarily accurate for builds where just a single slave builder was started, even if the contents of the manifest the slave actually used are updated. One has to manually examine the manifest used in the builder logs rather than the manifest version repo.
 


>> 2.5) New CLs are merged aka. A_CLS
>> 3) Version for BoardA is bumped to N.N.N.N+1

>Bumped how?

The slave seems to do it.
 


>> 4.5) New CLs are merged aka. B_CLs
>> 4) Indivual boardB is triggered 

>Triggered how?

Clicking the force button on the waterfall.

>I believe the slave will generate a new version number, however it does not commit the manifest properly.

Yikes, this sounds highly broken and undesirable.

And I'm pretty sure that if we have multiple builders incrementing version numbers in parallel on the same branch, we have a race conditions to deal with.

I think they do some sort of check based on time, like they don't generate new versions if they see a version bump in the last 15 minutes or something.

However if manifest updates are not correct, this implies two builds with the same version may not have exactly the same source content.

The solution today is to only start the master, this bug would allow us to optimize to be able to start individual builds.

I think we're going back and forth on what is desired here. To me, sounds like what is desired is a way to trigger the master and tell it the subset of slaves IT should trigger. Is that an acceptable approach?

That would work too, we would just need to use master to kick off boards all the time.

I would expect three options here
a) All boards (default)
b) A combination of boards 
c) A single board

(b) and (c) could be same just selecting multiple 

nxia@ dgarrett@ Is buildbucket currently being used to trigger slaves of release builders? If so, this is more tractable.

Not yet, but there are no remaining blockers.

I think the steps look like:

A) It set up the buildbucket
B) Tweak the buildbot instance to use it.
C) Tweak the release masters to use the proper buildbucket instance.

Was just pinged about this bug. Here's a summary of where we are:

For lakitu and jetstream, we will eventually pull their builders out in to their own masters so they stop interfering with our builds (such as in this case but also in others).

For the case where we need to do a board-specific fix on a release branch, we should be using a share build number across all builders. To implement this, we probably need a build parameter to each of the build slaves that indicates that master will be doing the build number incrementation. When those release builders on kicked off individually, that parameter would be false allowing those builders to increment the build number in some shared location.

This is not high enough priority to work on now but this looks like a good starter bug so will have Evan (Noogler) take a look at this in September.

That approach works, thanks for the follow up