InitSDK failure with Download failed with certificate error? Try "sudo c_rehash" |
||||||||
Issue descriptionWe had a couple of reports of PreCQ flake failures like this: https://luci-milo.appspot.com/buildbot/chromiumos.tryserver/no_vmtest_pre_cq/147878 17:27:44: INFO: RunCommand: /b/c/cbuild/repository/chromium/tools/depot_tools/ensure_bootstrap in /b/c/cbuild/repository/chromium/tools/depot_tools Bootstrapping cipd client for linux-amd64 from https://chrome-infra-packages.appspot.com/client?platform=linux-amd64&version=git_revision:0aa5e4d06ec4d3b90980fcb0d0b7c34a5e65b9e1... curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. /b/c/cbuild/repository/chromium/tools/depot_tools/vpython: line 12: /b/c/cbuild/repository/chromium/tools/depot_tools/.cipd_bin/vpython: No such file or directory 17:27:46: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros_sdk --buildbot-log-version --nouse-image --create --replace in /b/c/cbuild/repository 17:27:46: NOTICE: Deleting chroot. STEP_TEXT: 2017.10.12.072709 17:27:47: NOTICE: Downloading SDK tarball... cros_sdk: Unhandled exception: Traceback (most recent call last): File "/b/c/cbuild/repository/chromite/bin/cros_sdk", line 169, in <module> DoMain() File "/b/c/cbuild/repository/chromite/bin/cros_sdk", line 165, in DoMain commandline.ScriptWrapperMain(FindTarget) File "/b/c/cbuild/repository/chromite/lib/commandline.py", line 911, in ScriptWrapperMain ret = target(argv[1:]) File "/b/c/cbuild/repository/chromite/scripts/cros_sdk.py", line 1115, in main sdk_cache, urls, 'stage3' if options.bootstrap else 'SDK') File "/b/c/cbuild/repository/chromite/scripts/cros_sdk.py", line 145, in FetchRemoteTarballs capture_output=True) File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 409, in RunCurl 'Download failed with certificate error? Try "sudo c_rehash".') chromite.lib.retry_util.DownloadError: Download failed with certificate error? Try "sudo c_rehash". 17:27:47: ERROR: return code: 1; command: /b/c/cbuild/repository/chromite/bin/cros_sdk --buildbot-log-version --nouse-image --create --replace cmd=['/b/c/cbuild/repository/chromite/bin/cros_sdk', '--buildbot-log-version', '--nouse-image', '--create', '--replace'], cwd=/b/c/cbuild/repository, extra env={'USE': u'chrome_internal', 'FEATURES': 'separatedebug'}
,
Oct 18 2017
In the builds linked by Mike, the bot_update step is printing:
return client.RunOnDeps('recurse', args, ignore_requirements=True,
AttributeError: 'NoneType' object has no attribute 'RunOnDeps'
I don't know if this error is expected, this would look like the .gclient/.gclient_entries files are broken. +Ryan for this. e.g.
https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromiumos.tryserver%2Fno_vmtest_pre_cq%2F150239%2F%2B%2Frecipes%2Fsteps%2Fbot_update%2F0%2Fstdout
The rest of the failure is in chromite, e.g. https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromiumos.tryserver%2Fno_vmtest_pre_cq%2F150239%2F%2B%2Frecipes%2Fsteps%2FInitSDK%2F0%2Fstdout
,
Oct 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools/+/e6e0641b1536d0b85d11d41977679ab5e562e564 commit e6e0641b1536d0b85d11d41977679ab5e562e564 Author: Marc-Antoine Ruel <maruel@chromium.org> Date: Wed Oct 18 18:20:34 2017 gclient: print an error message instead of crashing When 'gclient recurse' is called from a unconfigured directory. No functional change otherwise. R=agable@chromium.org Bug:774749 Change-Id: Ifb2b74c042ce31faf3e1287c5614739e58e9178c Reviewed-on: https://chromium-review.googlesource.com/726419 Reviewed-by: Aaron Gable <agable@chromium.org> Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org> [modify] https://crrev.com/e6e0641b1536d0b85d11d41977679ab5e562e564/gclient.py
,
Oct 18 2017
The commit above is just to convert the crash observed into an error message, it is not fixing the problem, as I don't know what the problem is.
,
Oct 19 2017
the InitSDK failure is chromite code running curl outside of the CrOS chroot. as in curl that's installed in bot distro. i'm not sure there's anything we're doing there that'd affect curl's ability to process certificates.
,
Oct 19 2017
I've looked through a bunch of builds and it doesn't really seem certificate-specific. That is, there seem to be a lot of failures, some on InitSDK, some elsewhere, and the ones on InitSDK aren't all suggesting "sudo c_rehash" either (which I tried locally on a bot, and it didn't actually help). I'm seeing a whole of Pre-CQ failures on this now too, still looking...
,
Oct 19 2017
,
Oct 20 2017
Obvious question is obvious, but... what version of cacert.pem is being used inside the chroot?
,
Oct 20 2017
That file does not exist as a standalone file inside the chroot (or my workstation), but I could strongly expect it to be 100% consistent across all chroots associated with a given branch.
,
Oct 20 2017
All the listed builds in #0 and #1 ran on build247-m2. Assuming that machine (and its certs) are corrupt, I filed bug 776846. In the meantime, I've taken that machine offline so it shouldn't be corrupting precq runs.
,
Oct 20 2017
Machine has been reimaged and is back online. I'll watch it for any further failures. Down-grading pri in the meantime.
,
Oct 30 2017
This seems fixed? Reopen if it isn't.
,
Jan 22 2018
,
Jan 23 2018
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by vapier@chromium.org
, Oct 18 2017Labels: -Pri-3 Infra-Troopers OS-Chrome Pri-1