Yes.

This requires a way to trigger builds based on changes to a repo branch. LUCI Scheduler does not support this.

Waterfalls only had support because of this builder, which does repo sync's on multiple builders, then explicitly triggers waterfall builds.

https://uberchromegw.corp.google.com/i/chromeos.infra/builders/CrOS%20Repo%20Sync

I thought this was available now? https://chromium.googlesource.com/infra/infra/+/master/infra/config/global/luci-scheduler.cfg#206

Are you saying that setting refs to anything other than master doesn't work?

Luci Scheduler understands triggering from git repositories, but not repo checkouts (which have multiple git repositories). Vadim has been unwilling to extend LUCI Scheduler to understand repo in the past, and it would be very difficult for it to do so.

So... all of the possible solutions I see:

A) List every firmware relevant git repo in the LUCI Scheduler configuration. This list can vary by branch, and there would be no way to automatically know if we missed anything. I've been considering this highly error prone and avoiding it.

B) Create a repo checkout poller like was mentioned in #3.

C) Run the annealing recipe against every branch, and use LUCI Scheduler to trigger builds from changes to the manifests git repo on the branch after annealing updates snapshot.xml on the branch.

C has the advantage of being standard across all branches, while allowing LUCI Scheduler to serialize multiple triggers efficiently.

Vadim, what's your take on this?

Would it be possible to repeat what has been done on Buildbot? (Option C I presume, but I don't know what annealing is):

1. Setup a periodic recipes-based job that watches for changes to manifests.
2. It can use Google storage to keep its own state.
3. On changes, use Scheduler recipes module to submit triggering events to corresponding jobs (https://chromium.googlesource.com/infra/luci/recipes-py/+/master/recipe_modules/scheduler/api.py).

Implementing "Repo checkout poller" in LUCI scheduler is exactly same logically, except it would need to be fully reimplemented in Go and run in constrained Appengine environment.

As a middle ground it is fairly easy to write a LUCI Scheduler poller that instead of doing all the work on GAE, launches a Buildbucket task. The task would return a set of triggers, and LUCI Scheduler will dispatch them to whoever is subscribed to the poller (like it does for gitiles poller currently).

The advantage of the later approach is more uniform description of what is triggering what: it will all be in luci-scheduler.cfg.

I like option C described by Don. It's not the same as what Vadim proposed.

Vadim, annealing builder basically does this:
 def RunSteps(api):
   pinned = {}
   for repo, ref in unpinned_manifest:
     pinned[(repo, ref)] = api.gitiles.head_of(repo, ref)
   saveToManifestRepo(api, pinned)

The idea is that CQ will be submit changes to multiple repos of CrOS checkout, and annealing builder(s) will run in cron mode and update pinned manifest. This way, future CQ and CI runs will work against pinned manifest, revision of which will allow to reproduce the whole checkout.

So, IIUC Don's Option C, LUCI scheduler can be told to watch different refs of repository containing pinned manifests and trigger necessary CI builders, merging triggers as necessary.

Don/Jason: please correct me if I am wrong.

The annealing builder will also uprev ebuilds, but triggering branch builds only after ebuilds are uprevved seems like a good thing.

For context, the Firmware team is unhappy with the high volume of builds making it difficult for them to bisect. Changes land very rarely in firmware branches and the extra builds are adding a lot of noise that they'd rather not see.

A simple solution here is to do option A but much reduced: the Firmware team only has a tiny handful of repos that they frequently land changes on. We can simply communicate that we watch these handful and, if they want and additional builds beyond those produced by the watched, they can trigger those manually. This will expand the LUCI Scheduler watch configuration somewhat but not so much so that it's unwieldy.

I think this solve the business requirement without adding a lot of complexity.

We can take that approach, but similar issues will soon exist for factory builds.

The SIE team, the primary customer of factory builds, wants builds the branches to be built on a regular basis so that breakages can be found before a new commit is made. So, we'll do those daily.