the full error indicates problems talking to cidb.  maybe Jason has an idea where to look.

08:01:41: INFO: Running cidb query on pid 4872, repr(query) starts with <sqlalchemy.sql.expression.Select at 0x7f0a0e87f1d0; Select object>
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/cbuildbot/builders/generic_builders.py", line 297, in Run
  File "/b/c/cbuild/repository/chromite/cbuildbot/builders/simple_builders.py", line 377, in GetSyncInstance
  File "/b/c/cbuild/repository/chromite/cbuildbot/builders/generic_builders.py", line 74, in _GetStageInstance
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/sync_stages.py", line 912, in __init__
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/sync_stages.py", line 645, in __init__
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/sync_stages.py", line 372, in __init__
  File "/b/c/cbuild/repository/chromite/cbuildbot/buildbucket_lib.py", line 119, in __init__
  File "/b/c/cbuild/repository/chromite/lib/auth.py", line 163, in __init__
  File "/b/c/cbuild/repository/chromite/lib/auth.py", line 147, in GetAccessToken
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 122, in GenericRetry
  File "/b/c/cbuild/repository/chromite/lib/auth.py", line 121, in _TokenAndLoginIfNeed
  File "/b/c/cbuild/repository/chromite/lib/auth.py", line 88, in Token
  File "/b/c/cbuild/repository/chromite/lib/auth.py", line 41, in GetAuthUtil
  File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 116, in GetCIPDFromCache
  File "/b/c/cbuild/repository/chromite/lib/cache.py", line 37, in new_f
  File "/b/c/cbuild/repository/chromite/lib/cache.py", line 175, in SetDefault
  File "/b/c/cbuild/repository/chromite/lib/cache.py", line 46, in new_f
  File "/b/c/cbuild/repository/chromite/lib/cache.py", line 121, in _Assign
  File "/b/c/cbuild/repository/chromite/lib/cache.py", line 267, in _Insert
  File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 97, in _Fetch
  File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 76, in _DownloadCIPD
  File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 54, in _ChromeInfraRequest
  File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 1593, in request
  File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 1335, in _request
  File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 1257, in _conn_request
  File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 1021, in connect
  File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 80, in _ssl_wrap_socket
  File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 241, in __init__
    ciphers)
SSLError: [Errno 185090050] _ssl.c:344: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib
08:01:41: INFO: Running cidb query on pid 4872, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f0a0e88a6d0>

Actually, this is an SSL problem related to making a call to the Buildbucket API; the log messages before and after an red herrings.

We haven't seen this one before so I'm going to ping the Troopers to find out if CIPD, auth.py or cache.py are using their own SSL certificate stores.

Just to note that reef-firmware-9042.B builder is also failing as well (and seems to be the same reason).

https://uberchromegw.corp.google.com/i/chromeos.branch/builders/reef%20firmware%20firmware-reef-9042.B/builds/237

Bump to P1 as we need to provide partners an official build for them to validate.

Thanks.

Troopers, ping.

Just an additional note, tryjob looks to be OK, though.

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8940735646277162720

Raising priority in hopes that a Trooper will reply; not actually a P0.

It only appeared in the queue after changing to P0 (I believe it's due to already having an owner). Let me take a look.

Why was this moved to foundation? This seems like a specific client issue in a codebase where we don't have any expertise. I have no idea what the Report step is supposed to do or how the chromite repository is set up. This issue seems localized entirely within chromite and/or cbuildbot code, neither of which are our codebases, so unfortunately I don't think we can be much help here.

Generally speaking, SSLError 185090050 from httplib2 is related to cacerts.txt.  https://crbug.com/780727  contains some prior art for debugging (it looks like your bots experienced this issue before). Unfortunately it doesn't seem like there's a resolution problem in there.

Troopers, see the question in #2.

I'm not sure why the log messages would be a red herring. You want to know if auth.py, cache.py, or cipd.py uses its own ca store. Which auth.py, cache.py, and cipd.py are you referring to? The logs suggest these are your scripts:
https://chromium.googlesource.com/chromiumos/chromite/+/master/lib/auth.py
https://chromium.googlesource.com/chromiumos/chromite/+/master/lib/cache.py
https://chromium.googlesource.com/chromiumos/chromite/+/master/lib/cipd.py

These scripts seem to be using httplib2, which means you're using httplib2/cacerts.txt. Again, this is your httplib2 and your cacerts.txt:
https://chromium.googlesource.com/chromiumos/chromite/+/master/third_party/httplib2/
https://chromium.googlesource.com/chromiumos/chromite/+/master/third_party/httplib2/cacerts.txt

Our CIPD doesn't appear to be failing. Your cipd.py is failing when it calls httplib2.Http().request():
https://chromium.googlesource.com/chromiumos/chromite/+/3e90a8083f46a10597e550aca274b99c5ee2f92e/lib/cipd.py#54

It sort of seems like the same issue in  https://crbug.com/780727#c18 . In #0, the bot checked out chromite at 5f74c8ef3938e5251e0fd868c80fb2b8e259fda1 then as part of the recipe it switches to 3e90a8083f46a10597e550aca274b99c5ee2f92e, which is from 2016.

And let's not use false P0s. If you want a trooper to take a look at something, unassign it. Assigning it assumes it's being handled.

Re #12: This log message:
> 08:01:41: INFO: Running cidb query on pid 4872, repr(query) starts with <sqlalchemy.sql.expression.Select at 0x7f0a0e87f1d0; Select object>

is a red herring: this is not a CIDB issue; this log message isn't relevant to this bug. CIDB isn't the same thing as CIPD.

Yes, I don't see any evidence that we are using Foundation CIPD tooling, system Python, or system cacerts.txt. Was hoping you'd see some what that we are falling back on those.

#13: Please add this to go/bugatrooper

I'm not talking about cidb, I'm talking about these lines from the traceback:

  File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 97, in _Fetch
  File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 76, in _DownloadCIPD
  File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 54, in _ChromeInfraRequest

This is your cipd.py script, not ours. See the last link in #12.

Whom are you replying to? My comment in #2 about a red herring was replying to comment #1 guessing that this was a CIDB problem.

what details are looking for at this point ?  seems to have stalled ...

At this point, this bug is waiting for me to investigate further.