New issue
Advanced search Search tips

Issue 817645 link

Starred by 1 user
Status: Fixed
Owner:
pprabhu@chromium.org
Closed: Mar 2018
Cc:
ayatane@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocking:
issue 813863



Sign in to add a comment

SSH key rotation broke SSH from inside SSP container

Project Member Reported by pprabhu@chromium.org, Mar 1 2018 
http://shortn/_vjeIkVpyFL

The problem is in puppet/modules/lab/manifests/autotest_repo_postinstall.pp
This shamelessly uses a files resource from the profiles/base module. I updated the profiles one, but obviously didn't update this. Why would I, eh?

P0 because this means all SSP testing is broken?
Project Member

Comment 1 by bugdroid1@chromium.org, Mar 1 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/2282ae3b9eaf293566a0a2e3c9421f3e18043a4d

commit 2282ae3b9eaf293566a0a2e3c9421f3e18043a4d
Author: Prathmesh Prabhu <pprabhu@google.com>
Date: Thu Mar 01 01:31:27 2018

Comment 2 by pprabhu@chromium.org, Mar 1 2018

Labels: -Pri-0 Pri-2
Forcing a puppet run across the fleet to deploy #1.

Comment 3 by pprabhu@chromium.org, Mar 1 2018

Blocking: 813863
Project Member

Comment 4 by bugdroid1@chromium.org, Mar 8 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/e58203bbaf14469d8d51b8047309f987e51d1a2c

commit e58203bbaf14469d8d51b8047309f987e51d1a2c
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Thu Mar 08 00:21:28 2018

Comment 6 by pprabhu@chromium.org, Mar 8 2018 

Keeping an eye on: http://shortn/_CWlv0tXCtI

Comment 7 by pprabhu@chromium.org, Mar 8 2018 

There were some SSP test failures *again*.

I poked in prod on one of these and found that some devservers still hadn't gotten the new public key. This is because puppet run was failing on these devservers because they didn't have puppet decryption key deployed.

Comment 8 by pprabhu@chromium.org, Mar 8 2018 

All but the following servers have been updated: 

pprabhu@pprabhu:ansible$ grep 'unreachable=1' /tmp/deploy_key.log 
android1758-infra-devserver5.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos1-dev-infra-devserver.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos1-dev-infra-devserver1.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos15-infra-devserver3.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos15-infra-devserver5.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos9-infra-devserver4.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos9-infra-devserver5.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos9-infra-devserver6.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos9-infra-devserver7.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0

Comment 9 by pprabhu@chromium.org, Mar 8 2018 

Filed  issue 820242  for puppet dashboard not showing when puppet resources fail.

Comment 10 by pprabhu@chromium.org, Mar 8 2018 

Also forced a puppet run across the fleet to pick up public keys on all devservers.

Comment 11 by pprabhu@chromium.org, Mar 9 2018

Status: Fixed (was: Started)
Don't see any new failures here. Declare victory.
►

Sign in to add a comment