Can you please define "persistent infrastructure" in a more precise way?

Do you mean any file on the builder? Populating builder caches? Or only external infrastructure (Gerrit, etc)?

Stepping back from the problem for a bit, what do you guys think about this (conceptually) for infra:

- The outer most layer does nothing more than invoke idempotent/non-persistent actions. Any persistence between actions is handled at this level (you can think of it as the 'wiring' if you like. For example, manifest-versions lives at this layer as it's a 'wiring' detail between steps.
- Next layer down is the stable API boundary for these actions. It is a bridge between (for example) a stable proto3 'build these packages' API and the next layer down.
- The branched infra, aka build scripts, Portage and so on. Here be dragons.

I like this layering because it strongly isolates the infrastructure we run on with the task we are trying to accomplish. The only layer specific to our underlying infra is the first (the wiring). In concrete terms, I like this layering because it would make moving over to a Kubernetes based build system much easier (ex, instead of manifest-versions would would use Docker images for snapshots).


So let's define mutable actions as anything that 'should' live in the first layer there. Unfortunately we have smashed the first two layers together in Recipe/Chromite scripts, but this is an effort to try and keep at least some of that organization.

I'm open to counter arguments though. Also this 'new' bug looks a lot like crbug.com/914988

Lets do design discussion in the doc: https://docs.google.com/document/d/1uRC5iP_bKkfGRpoTwYxyFOjDHBkrYV58BTzjpt73zAo/edit#heading=h.xwmvhrkcqt47

> Also this 'new' bug looks a lot like crbug.com/914988

Well dangit why didn't you use that for the TODO link?

> > Also this 'new' bug looks a lot like crbug.com/914988

> Well dangit why didn't you use that for the TODO link?

Because this bug was something else before you changed it ;p

> Stepping back from the problem for a bit

I think this fits in with the overall "Build vs CI" boundary, just looking at it from a different perspective.

> Because this bug was something else before you changed it ;p

I didn't change it what are you talking about. Ignore those deleted comments.

Can I get concrete examples of what is / is not "persistent infrastructure"?

Maybe. How many peer bonuses are you offering?

Mutable actions to persistent infra:
- Pushing to a manifest repo, including annealing and LKGM
- Pushing portage package uprevs
- Pretty much any `git push` actually
- Writing to CIDB/Buildbucket from a build
- Publishing artifacts to Cloud Storage
- And so on

The way I think of it is, anything that could be run multiple times and in parallel with safety (modulo quota limits) is non-mutable. Anywhere that has a chance of collision when run in parallel is mutable. Checking out to ToT, building, running VM tests and so on is non-mutable, as is a human-triggered build started in Swarming (assuming the build itself doesn't mutate things). No idea if that makes any sense, it's just what I'm thinking :p

Oh you mean "recipe modules". I'm not sure it makes sense to make that distinction there. Recipe modules represent reusable, coherent chunks of code that can be composed in a recipe. I think of it more as recipes being the "main" function and recipe modules being libraries. I do want to make this distinction at the recipes/chromite boundary; that is, (ideally) CI calls into chromite won't expose side effects outside of that build.

Think of it this way: we want each recipe to correspond to a builder "kind": CQ, annealing, post-submit, and in the future release, firmware, factory, etc. Recipe modules (and possibly "sub-recipes", depending on how parallel recipes work out) are the reusable parts that are composed into these builders.

If we really want to do this we could get rid of the dev.dryrun state and just do `module.step(dryrun=recipe_dryrun)` for each mutating step. 

Sorry I have confused things. I'm talking about two different things. There is

- This bug, which is overall how recipe is structured, I would like any action that mutates infra to be signaled from the top-level recipe, rather than buried down in some module somewhere.

- A discussion on the overall design philosophy of our infrastructure (that is unrelated to how recipes work, apart from the note that we have kind of collapsed the first two layers down into recipe/chromite at this point.

I updated the bug description to maybe be a bit more clear.

Re#13: You many anything "off builder". Okay, thanks, got it.