proposed design:

- introduce `peek_percentage`: an attribute of a bucket, an integer value [0..100]
- add get_peek_percentage(bucket) and set_peek_percentage(bucket) APIs
- in the peek API, filter builds: `build_id % 100 < percentage` in frontend
  Note: this is ineffective for low non-zero values of percentage, however it should be happen only during incidents
- deprecate pausing in favor of this (I could not come up with a smooth transition from binary paused/non-paused to a probability)

+dnj and vadimsh for opinion

the mental model I had was something more along the lines of throttling to X running builds, but maybe it's not obvious what the ranges for X are without also knowing what percentage X is of the total.

once build is scheduled on buildbot, buildbucket does not have control over if/when it will run. At that point, buildbot has full control. In practice, number of running builds is limited by the number of connected slaves.

The proposed design controls how many builds buildbot discovers.

We talked about this a few times in person and I think the conclusion was that it's more important that this works in LUCI than buildbot if the implementation would be different. Nodir also said we could change swarmbucket to use task queues so it's pushed based like buildbot for easier throttling.