New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 735821 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

SSL failures on slave

Project Member Reported by emso@chromium.org, Jun 22 2017

Issue description

I'm having issues setting up credentials for the slaves connected to the new internal code search master (for instance, slave102-c7). estaab@ found SSL errors in the puppet logs on the new slaves for the internal master (compared to no errors in the corresponding slave for the old public master).

From /var/log/puppet/run_puppet.log on slave102-c7:

Starting: Wed Jun 21 06:50:01 PDT 2017
Sleeping 20 seconds to spread master load...
Pinging puppetm2.golo.chromium.org to ensure connectivity...
PING puppetm2.golo.chromium.org (74.125.248.82) 56(84) bytes of data.
64 bytes from puppetm2 (74.125.248.82): icmp_seq=1 ttl=54 time=28.1 ms
64 bytes from puppetm2 (74.125.248.82): icmp_seq=2 ttl=54 time=69.7 ms
64 bytes from puppetm2 (74.125.248.82): icmp_seq=3 ttl=54 time=27.6 ms
--- puppetm2.golo.chromium.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 27.679/41.858/69.700/19.689 ms
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 
alert certificate revoked
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://puppetm2.golo.chromium.org/pluginfacts: SSL_connect returned=1 errno=0 state
=SSLv3 read server session ticket A: sslv3 alert certificate revoked
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 aler
t certificate revoked
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://puppetm2.golo.chromium.org/plugins: SSL_connect returned=1 errno=0 state=SSLv3 r
ead server session ticket A: sslv3 alert certificate revoked
Info: Loading facts
Fact file /var/lib/puppet/facts.d/windows_defender.ps1 was parsed but returned an empty data set
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked
Finshed: Wed Jun 21 06:50:35 PDT 2017 with exit status 1

compared to slave14-c1:

Starting: Wed Jun 21 07:12:01 PDT 2017
Sleeping 42 seconds to spread master load...
Pinging puppetm2.golo.chromium.org to ensure connectivity...
PING puppetm2.golo.chromium.org (74.125.248.82) 56(84) bytes of data.
64 bytes from puppetm2 (74.125.248.82): icmp_seq=1 ttl=54 time=27.9 ms
64 bytes from puppetm2 (74.125.248.82): icmp_seq=2 ttl=54 time=27.5 ms
64 bytes from puppetm2 (74.125.248.82): icmp_seq=3 ttl=54 time=27.6 ms

--- puppetm2.golo.chromium.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 27.597/27.733/27.973/0.217 ms
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Fact file /var/lib/puppet/facts.d/windows_defender.ps1 was parsed but returned an empty data set
Could not retrieve fact='gclient_deps', resolution='<anonymous>': undefined method `[]' for nil:NilClass
Info: Caching catalog for slave14-c1.c.chromecompute.google.com.internal
Info: Applying configuration version '1498054349'
Notice: Finished catalog run in 2.82 seconds
Finshed: Wed Jun 21 07:13:26 PDT 2017 with exit status 0

 

Comment 1 by pschm...@google.com, Jun 22 2017

Cc: vadimsh@chromium.org
Cert mismatch for sure.   Not sure if Vadim wants to look at this first?   

root@puppetm:~# puppet cert fingerprint slave102-c7.c.chromecompute.google.com.internal
slave102-c7.c.chromecompute.google.com.internal (SHA256) C9:A4:64:F8:97:12:31:41:7B:3E:44:43:7B:88:1B:58:48:4E:23:69:86:5A:88:BA:0D:D0:1A:81:53:94:85:DF

root@slave102-c7:~# puppet agent -t --fingerprint
(SHA256) 1A:3E:75:9C:58:35:B5:4D:69:40:29:FF:17:55:8C:A0:2B:05:18:E6:50:2C:DB:F0:FC:69:14:E0:5B:CA:B7:44
Owner: vadimsh@chromium.org
Status: Fixed (was: Untriaged)
The cert on the machine has SN 378076, which is too far from current ones (420000) to debug easily...

Rebootstrapped the cert, Puppet agent runs normal now.

Sign in to add a comment