It seems like a reasonable way to catch integration problems before production as we put swarming into production.

the packages and their versions that we push to build machines, including build bootstrapper, python, git, are defined in so called "swarming_task_template". We have 2 of them:

prod: https://chrome-internal.googlesource.com/infradata/config/+/master/configs/cr-buildbucket/swarming_task_template.json
canary: https://chrome-internal.googlesource.com/infradata/config/+/master/configs/cr-buildbucket/swarming_task_template_canary.json

by default (bucket setting) 10% of LUCI builds use the canary template, so if you see that builds sometimes fail for unclear reasons, that may be because the build was canary ("canary" attr of buildbucket build is true).

Currently we have very limited alerting on canary being broken:  bug 721571 .

related: bug 695511 (milo: clearly communicate that a build is canary)

--

there is also dev versions of servers, e.g. cr-buildbucket-dev and luci-scheduler-dev. Those use HEAD versions of our packages. Jobs must be configured manually on dev servers. We and Fuchsia did so: https://luci-scheduler-dev.appspot.com/

consider doing so too

So, it's not a dedicated staging environment, instead production builds are randomly opted in at a low percentage?

Or.... (and this sounds better to me), when we get a staging version of our environment working, we can opt in to HEAD and test your new stuff as well.

The only downside, is that CrOps developers get no direct feedback until after they've committed changes that break us.

We are different enough that this scares me.

> So, it's not a dedicated staging environment, instead production builds are randomly opted in at a low percentage?

correct. Note that when scheduling a buildbucket build, you force canary. I guess that's how you can ensure that critical builds won't break when canary is promoted to prod. +iannucci who was thinking about related things.

Exactly what is added to the buildbucket request to opt in?

set "canary_preference" of a build request to "CANARY"
also see put() API in go/buildbucket-design

I’m not sure what’s expected from foundation team here 

Duping against the bug to setup a controlled way to do canary infra builds for ChromeOS.