I have a meeting with athilenius about this on Monday. Potentially, this can be simplified.

Recapping from the discussion in chat, the impetus for this is that the current structure goes something like:

  luci-scheduler(cron) -> 
    master builder ->
      generates manifest
      triggers children builds with manifests
      does post processing work after children complete

The UI needs to show the master builder next to its children builds, so it needs to know this relationship somehow.

We have 'buildset's in buildbucket, but these are attached at build creation time, and so couldn't include the master and the children in the buildset.

We discussed in chat a possible alternate architecture, which wouldn't require extensions to buildbucket or milo:

  luci-scheduler(cron) -> 
    snapshot_manifest:
      snapshots everything, commits manifest to $manifest_repo

  luci-scheduler($manifest_repo) ->
    child0..childN
    post-process(waits for all other children builds)

the buildset for all children and the post-process builder would be based on the commits to $manifest_repo. Milo would also be able to display a console for this too (so each row in the console would be a commit to the manifest repo, each column would be one of the child builders, and the last column would be the 'post-process' builder that processes the outputs from the other child builders).

Not claiming that this is sufficient, just a summary of what we discussed :)

> We have 'buildset's in buildbucket, but these are attached at build creation time, and so couldn't include the master and the children in the buildset.

Why can't buildset be updated after a buildbucket is launched? If we did that from that master, that would very simply and elegantly solve this.

> We discussed in chat a possible alternate architecture, which wouldn't require extensions to buildbucket or milo:
> 
>   luci-scheduler(cron) -> 
>     snapshot_manifest:
>       snapshots everything, commits manifest to $manifest_repo
> 
>   luci-scheduler($manifest_repo) ->
>     child0..childN
>     post-process(waits for all other children builds)

While this would work, this is a confusing and expensive re-architecting of the ways in which our builds work: it's introducing a join point that must know—when it is launched—all things that it will be joining to. And, it only happens to work because we know all builders when it is launched and have a single level of parent/child relationship now. Neither will be true later in 2019 when we migrate the SDK builder to across-machine runs with a volume server. There will be 2-3 levels of parent/child relationships with an indeterminate set of LUCI tasks to wait on.

I'm also afraid it would be confusing for users to understand: the UI would show the post-processing builder doing nothing for hours.

> the buildset for all children and the post-process builder would be based on the commits to $manifest_repo.
> 
> 
> Milo would also be able to display a console for this too (so each row in the console would be a commit to the manifest repo, each column would be one of the child builders, and the last column would be the 'post-process' builder that processes the outputs from the other child builders).

The plan of record (issue 903372) is that we would use gitiles commit metadata for row affinity (under recommendation from Nodir). Are these mutually exclusive?

If we used gitiles commit metadata for row affinity in the Milo console view and we had a parent-child relationship in BB to allow the build view to show child builders, wouldn't that get us our objectives with less work?

Per email thread and the discussion here, we believe that we will be able to get all of the Milo rendering desired using only gitiles_commit and buildset as the mechanism to correlate related builds and so no additional domain concept will be added to Buildbucket at this time.