Infra folks / troopers: this is blocking our LUCI migration; can someone please take a look quickly? Thanks.

Logged onto one:

[I2018-03-14T20:05:40.454000 2800 2820 chromebuild-startup:78] Calling: 'C:\\Program Files\\OpenSSH-Win64\\ssh-keygen.exe -f C:\\Program Files\\OpenSSH-Win64\\ssh_host_rsa_key -P  -t rsa'
[I2018-03-14T20:05:40.454000 2800 2820 chromebuild-startup:79]    in C:\Windows\system32
[D2018-03-14T20:05:41.110000 2800 2820 chromebuild-startup:100] Generating public/private rsa key pair.
[D2018-03-14T20:05:41.110000 2800 2820 chromebuild-startup:100] C:\Program Files\OpenSSH-Win64\ssh_host_rsa_key already exists.

I can respawn the bots for now, but I feel like OpenSSH is causing more harm than it is helping us, and we should disable it for now and fix it later (Since we can still RDP into the bot).

Any recommendations you have hinoka@ for getting these bots back online sooner rather than later are welcome. Thanks.

I respawned them all but that didn't seem to help.  It's still getting stuck on SSH key regeneration.  I'm wondering how these worked previously and if we need to push a new image.

That might've been a red herring.  I logged onto a different bot and it's stuck at starting up due to infra-python not being around.  infra-python is installed by puppet.

I've been informed that there was a CIPD issue that was causing 403s to be served.  That might've been related.

The CIPD issue should be fixed.  I'll try respawning again.

I spot checked 5 of them and they all look online.  I'll mark this as fixed.  Root cause was CIPD being broken due to Google Storage giving us bad signed URLs, and therefore the instances could not finish bootstrap.

Hm... I'm not sure if that's the /root/ cause. That error only cropped up in the last hour or so.

hinoka@: thanks for your quick action. At least one of these slaves, swarm768-c4, still seems missing:
https://chromium-swarm.appspot.com/bot?id=swarm768-c4&sort_stats=total%3Adesc

Can you please take another look?

Respawning... in the meantime, filed crbug.com/822032

It took 4 respawns... but it's back online now.  We really need openssh to be less broken.

Jeez. Thanks Ryan.

These bots all appear to be offline again.  Can someone take a look?

wtf. Let's see...

I would suggest a postmortem on this when this is resolved. Andrii can you take the lead on drafting one? 

efoo@ there are two distinct causes here: both of which are handled in diff bugs, see Ryan's reply above:
 issue 822032
 issue 819355 which results in Win bots getting auto-updated and getting stuck in privacy options screen

Yep, this is issue 819355 again. Unfortunately the only thing I can do it respawn them.

efoo@ cit-pm-75 should be written for issue 819355, but first it has to be resolved.

$ ./ccompute ri swarm7{49..68}-c4

4 bots came back...

All back. However, not for long. If they go offline again (and I bet they will), please poke issue 819355

Thanks for following up, will keep future requests in that bug.

To clarify, my request is to create a postmortem on the flaky OpenSSH issues detailed here. This is separate from the auto-update security updates.

Pinging smut@ and vadmsh@ directly.