My prevailing theory is that remote_run was doing a full checkout (~100MB) on every build.  Because that specific bot is essentially 20 buildbot slaves, the effects were more pronounced.

I'm gathering data to see if this theory correlates with the git bandwidth increases.

Here is a historical graph comparing remote_run vs annotated_run for this year:
https://datastudio.google.com/open/0Bys_VarwdXE0bTIwYkpfYnBVQm8

There appears to be no correlation with git bandwidth usage.

Downgrading priority because Pri1 needs someone assigned + this has been open for 2+ months without any update.
In fact, based on comment 3 above, this probably should be Archived.

Andrii, is it now possible to convert chromium.swarm to a set of luci cron tasks instead?

As discussed with maruel@ in IM: Yes it is possible.

Easiest is defining a typical Job with schedule like */10 * * * * that triggers a buildbucket builder. The builder should be defined in buildbucket.cfg through swarmbucket definition. That builder would then run existing recipe.

However, the biggest chunk of work is actually on modifying the recipe that currently uses too much of git bandwidth. In LUCI (ie swarmbucket), named cache perhaps should be used.

In fact, just modifying the recipe might be sufficient while keeping existing buildbot slave, but I don't know how multi-buildbot-slaves per machine behave wrt to named cache.