VM creation is already rate limited at 100 per instance group manager per minute. This was done to avoid overwhelming the cr-puppet token server.

I understand GCE is worried about our deletion rate as well.

Cloud Platform thinks they've solved the mass deletion problem and we don't need to do anything on our end for it.

We are still stressing our own systems when mass respawning bots. In particular Puppet and Isolate.

Creations are already limited to 100 per minute per managed instance group (we have one managed instance group), do we need to limit that even further for our own systems' sake?

Oh, actually we have two managed instance groups because each is limited to 1000 and we need 1300 instances.

That means it's possible for us to spike to 200 creations per minute.

The rate of changes should be as smooth as possible. Are there any downsides for making lease durations randomized in Swarming? It seems to be a simple change.

It doesn't fix the problem. The 810 always-on leases already have their lease end times randomized because they have naturally spread out across the several hours due to random delays in the provisioning steps (and there are a lot of provisioning steps). Adding an additional randomization factor should be unnecessary.

The 500 extra VMs are the problem, but their lease duration is not the issue. At 9pm tonight when 500 extra VMs all expire at once, 500 will be deleted (no longer a big deal for Cloud Platform), but we won't have 500 creations, because GCE Backend only maintains 10% more than the number of leased VMs. Since only 810 VMs are leased overnight, 500 VMs are deleted but only 81 replacements are made. Presumably we can handle a single burst of 81 creations per minte.

The problem arises tomorrow morning at 7am when Swarming asks for 500 more VMs. GCE Backend will supply the 81 it held overnight right away. Then with 891 leases it will spin up 89 more. Once those are leased (980 total leased) it will bring up 98 more. Once those are leased (1078 total) it will bring up at most 108 (if they are in two different managed instance groups). Once those are leased (1186) it will bring up another 117. Once leased (1303) it will bring up 7 more to reach the configured maximum of 1310.

It's these hundreds being brought up every minute that are liable to cause problems.

The deletions might have been an issue under the old scheme where at UTC midnight we actually had to delete the VMs and recreate them all in the middle of the day, but the new scheme that went into effect today doesn't require that.

Is this still an issue?

See #7, I think this is wontfix.