builders fail in X509_load_cert_crl_file whenever "clobber build" option is set |
||||||||||||||||
Issue descriptionPFQ: Exception thrown, but all stages marked successful. This is an internal error, because the stage that threw the exception should be marked as failing. Happened on https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/4704, which was a clobber build.
,
Aug 11 2017
FixIt to see if we can move the CIPD stuff out of the constructor.
,
Aug 11 2017
This seems to happen when the clobber build option is used. Clobber build: https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/4703 Clobber build: https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/4704 No clobber build: https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/4705
,
Aug 11 2017
I can't find the exception in #4703, only in #4704. I believe it was network flake.
,
Aug 11 2017
4703 looks like a legit failure. At least, multiple slave builders failed.
,
Aug 11 2017
Ah, sorry, I was off by one. Clobber build: https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/4704 Clobber build: https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/4705 No clobber build: https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/4706
,
Aug 11 2017
Sigh. Yeah.
,
Aug 12 2017
,
Aug 17 2017
Moving to the secondary to find an owner.
,
Apr 20 2018
,
Apr 20 2018
Issue 835485 has been merged into this issue.
,
Apr 20 2018
(from dup'd bug) Just happened again: ssl.SSLError: [Errno 185090050] _ssl.c:344: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib Full log: https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos_release%2Fmaster-release_release-R66-10452.B%2F50%2F%2B%2Frecipes%2Fsteps%2FReport%2F0%2Fstdout
,
Apr 20 2018
Full log has:
buildbot: Unhandled exception:
Traceback (most recent call last):
File "/b/c/cbuild/repository/chromite/bin/cbuildbot", line 169, in <module>
File "/b/c/cbuild/repository/chromite/bin/cbuildbot", line 165, in DoMain
File "/b/c/cbuild/repository/chromite/lib/commandline.py", line 911, in ScriptWrapperMain
File "/b/c/cbuild/repository/chromite/scripts/cbuildbot.py", line 1019, in main
File "/b/c/cbuild/repository/chromite/scripts/cbuildbot.py", line 1016, in main
File "/b/c/cbuild/repository/chromite/scripts/cbuildbot.py", line 174, in _RunBuildStagesWrapper
File "/b/c/cbuild/repository/chromite/cbuildbot/builders/generic_builders.py", line 308, in Run
File "/b/c/cbuild/repository/chromite/cbuildbot/builders/simple_builders.py", line 484, in GetSyncInstance
File "/b/c/cbuild/repository/chromite/cbuildbot/builders/generic_builders.py", line 76, in _GetStageInstance
File "/b/c/cbuild/repository/chromite/cbuildbot/stages/sync_stages.py", line 543, in __init__
File "/b/c/cbuild/repository/chromite/cbuildbot/stages/sync_stages.py", line 387, in __init__
File "/b/c/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 318, in GetBuildbucketClient
File "/b/c/cbuild/repository/chromite/lib/buildbucket_lib.py", line 224, in __init__
File "/b/c/cbuild/repository/chromite/lib/auth.py", line 179, in __init__
File "/b/c/cbuild/repository/chromite/lib/auth.py", line 163, in GetAccessToken
File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 244, in GenericRetry
File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 177, in _Wrapper
File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 243, in _run
File "/b/c/cbuild/repository/chromite/lib/auth.py", line 135, in _TokenAndLoginIfNeed
File "/b/c/cbuild/repository/chromite/lib/auth.py", line 94, in Token
File "/b/c/cbuild/repository/chromite/lib/auth.py", line 44, in GetLuciAuth
File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 117, in GetCIPDFromCache
File "/b/c/cbuild/repository/chromite/lib/cache.py", line 38, in new_f
File "/b/c/cbuild/repository/chromite/lib/cache.py", line 176, in SetDefault
File "/b/c/cbuild/repository/chromite/lib/cache.py", line 47, in new_f
File "/b/c/cbuild/repository/chromite/lib/cache.py", line 122, in _Assign
File "/b/c/cbuild/repository/chromite/lib/cache.py", line 272, in _Insert
File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 98, in _Fetch
File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 77, in _DownloadCIPD
File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 55, in _ChromeInfraRequest
File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 1593, in request
File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 1335, in _request
File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 1257, in _conn_request
File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 1021, in connect
File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 80, in _ssl_wrap_socket
File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 241, in __init__
ciphers)
ssl.SSLError: [Errno 185090050] _ssl.c:344: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib
,
Apr 20 2018
CCed @akeshet and @dgarrett who are familiar w/ cros infra.
,
Apr 20 2018
This looks like clobber is implicated again. From: https://uberchromegw.corp.google.com/i/chromeos_release/builders/master-release%20release-R66-10452.B we can see that the last two failing builds both had clobber set. The 3 build also failed but for a different reason. All the successful builds do not have clobber set. So, uh, who is setting clobber?
,
Apr 20 2018
,
Apr 21 2018
,
Apr 21 2018
I forced a new build on the same branch without clobber to see if that works around the issue.
,
Apr 21 2018
That succeeded: https://uberchromegw.corp.google.com/i/chromeos_release/builders/master-release%20release-R66-10452.B/builds/51 So, the immediate worry is over. We need to find out how "clobber" was being set on the two failed builds.
,
Apr 23 2018
Someone manually hit "Force Build" with the clobber checkbox checked. "Clobber" in the buildbot configuration, or the recipe would happen every single time. Further, clobber is always supposed to be safe, and was often required on that waterfall before I wrote cbuildbot_launch.
,
Apr 25 2018
Okay, let's use this bug to track what is wrong with the clobber option in the first place.
,
Sep 5
,
Sep 5
,
Sep 5
Issue 863648 has been merged into this issue.
,
Sep 5
I think I see the issue: 1) "clobber" causes ClearBuildRoot to run: crrev.com/8a49a283b96c36a16538c6e3893a5661c987501f/cbuildbot/repository.py#96 2) ClearBuildRoot runs something like: find /b/c/cbuild/repository -mindepth 1 -maxdepth 1 '!' -name cbuildbot_logs -exec rm -rf '{}' + 3) This deletes almost everything in /b/c/cbuild/repository, including chromite/third_party/httplib2/cacerts.txt 4) The still-resident cbuildbot that was executing from /b/c/cbuild/repository/chromite calls into httplib2, which tries to read cacerts.txt and dies. I'm looking into a fix.
,
Sep 6
We'll see if crrev.com/c/1208369 helps. Interestingly it looks like there is some old code to prevent this issue which trips in a local tryjob; I haven't investigated into why it doesn't work elsewhere: https://chromium.googlesource.com/chromiumos/chromite/+/8a49a283b96c36a16538c6e3893a5661c987501f/cbuildbot/commands.py#158
,
Sep 8
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/8cc2eabce89d49c7da82b12b4cda560d8817bd78 commit 8cc2eabce89d49c7da82b12b4cda560d8817bd78 Author: Lann Martin <lannm@chromium.org> Date: Sat Sep 08 01:36:56 2018 cbuildbot: Preserve chromite/ in CleanUp stage Since cbuildbot may be running from the target repository, don't delete chromite out from under the running code. BUG= chromium:754859 TEST=none Change-Id: I178f8de52720d470462a289d407caa9a13535989 Reviewed-on: https://chromium-review.googlesource.com/1208369 Commit-Ready: Lann Martin <lannm@chromium.org> Tested-by: Lann Martin <lannm@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/8cc2eabce89d49c7da82b12b4cda560d8817bd78/scripts/cbuildbot.py
,
Sep 10
I am optimistic that the above change will fix this, but I'm not sure how to verify without merging back into a branch. Given that this was a low impact bug I'm going to close and see if it crops up again.
,
Sep 10
Since we have seen this on firmware-grunt-11031.B branch, can I suggest that we merge it there and give it a try to see if the problem persists? Thanks.
,
Sep 10
,
Sep 10
,
Sep 10
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/c0ad292c028265ca9bd3a2acc1d324cf2d22ba8e commit c0ad292c028265ca9bd3a2acc1d324cf2d22ba8e Author: Lann Martin <lannm@chromium.org> Date: Mon Sep 10 16:08:42 2018 cbuildbot: Preserve chromite/ in CleanUp stage Since cbuildbot may be running from the target repository, don't delete chromite out from under the running code. BUG= chromium:754859 TEST=none Change-Id: I178f8de52720d470462a289d407caa9a13535989 Reviewed-on: https://chromium-review.googlesource.com/1208369 Commit-Ready: Lann Martin <lannm@chromium.org> Tested-by: Lann Martin <lannm@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> (cherry picked from commit 8cc2eabce89d49c7da82b12b4cda560d8817bd78) Reviewed-on: https://chromium-review.googlesource.com/1216882 Reviewed-by: YH Lin <yueherngl@chromium.org> Commit-Queue: YH Lin <yueherngl@chromium.org> [modify] https://crrev.com/c0ad292c028265ca9bd3a2acc1d324cf2d22ba8e/scripts/cbuildbot.py
,
Sep 10
Can you apply the fix to duped bugs as well?
,
Sep 10
#36: Sure, once verified I'll do so.
,
Sep 10
I am trying a clobbered pre-flight build, which used to exhibit this kind of problem. Now we will see the problem still there.. https://uberchromegw.corp.google.com/i/chromeos.branch/builders/grunt%20pre-flight%20firmware-grunt-11031.B/builds/10
,
Sep 10
The build from #35 has finished the step that previously failed. I'm going to go ahead with cherry picking this fix into other branches.
,
Sep 10
Any value in merging this back into 69/70?
,
Sep 11
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/38fc8e14b29fe644621ed6bf11aaa002868d580d commit 38fc8e14b29fe644621ed6bf11aaa002868d580d Author: Lann Martin <lannm@chromium.org> Date: Tue Sep 11 00:59:29 2018 cbuildbot: Preserve chromite/ in CleanUp stage Since cbuildbot may be running from the target repository, don't delete chromite out from under the running code. BUG= chromium:754859 TEST=none Change-Id: I178f8de52720d470462a289d407caa9a13535989 Reviewed-on: https://chromium-review.googlesource.com/1208369 Commit-Ready: Lann Martin <lannm@chromium.org> Tested-by: Lann Martin <lannm@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> (cherry picked from commit 8cc2eabce89d49c7da82b12b4cda560d8817bd78) Reviewed-on: https://chromium-review.googlesource.com/1216886 Reviewed-by: Nick Sanders <nsanders@chromium.org> Commit-Queue: Lann Martin <lannm@chromium.org> [modify] https://crrev.com/38fc8e14b29fe644621ed6bf11aaa002868d580d/scripts/cbuildbot.py |
||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||
Comment 1 by dgarr...@chromium.org
, Aug 11 2017Ignoring the actual failure, and only looking at the reporting.... while far from optimal, this is expected behavior. The code in question was running in the context of the builder class (which instantiates the various stages). Because the builder class isn't part of a stage, no stage failed. It was actually stage code that failed, but constructor code, not "Run" code. I am really surprised that we are doing HTTP connections during the constructor, instead of while running. That seems wrong. Traceback (most recent call last): File "/b/c/cbuild/repository/chromite/bin/cbuildbot", line 168, in <module> File "/b/c/cbuild/repository/chromite/bin/cbuildbot", line 164, in DoMain File "/b/c/cbuild/repository/chromite/lib/commandline.py", line 829, in ScriptWrapperMain File "/b/c/cbuild/repository/chromite/scripts/cbuildbot.py", line 1318, in main File "/b/c/cbuild/repository/chromite/scripts/cbuildbot.py", line 1315, in main File "/b/c/cbuild/repository/chromite/scripts/cbuildbot.py", line 248, in _RunBuildStagesWrapper File "/b/c/cbuild/repository/chromite/cbuildbot/builders/generic_builders.py", line 308, in Run File "/b/c/cbuild/repository/chromite/cbuildbot/builders/simple_builders.py", line 416, in GetSyncInstance File "/b/c/cbuild/repository/chromite/cbuildbot/builders/generic_builders.py", line 75, in _GetStageInstance File "/b/c/cbuild/repository/chromite/cbuildbot/stages/sync_stages.py", line 811, in __init__ File "/b/c/cbuild/repository/chromite/cbuildbot/stages/sync_stages.py", line 548, in __init__ File "/b/c/cbuild/repository/chromite/cbuildbot/stages/sync_stages.py", line 392, in __init__ File "/b/c/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 315, in GetBuildbucketClient File "/b/c/cbuild/repository/chromite/lib/buildbucket_lib.py", line 173, in __init__ File "/b/c/cbuild/repository/chromite/lib/auth.py", line 163, in __init__ File "/b/c/cbuild/repository/chromite/lib/auth.py", line 147, in GetAccessToken File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 243, in GenericRetry File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 176, in _Wrapper File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 242, in _run File "/b/c/cbuild/repository/chromite/lib/auth.py", line 121, in _TokenAndLoginIfNeed File "/b/c/cbuild/repository/chromite/lib/auth.py", line 88, in Token File "/b/c/cbuild/repository/chromite/lib/auth.py", line 41, in GetAuthUtil File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 116, in GetCIPDFromCache File "/b/c/cbuild/repository/chromite/lib/cache.py", line 37, in new_f File "/b/c/cbuild/repository/chromite/lib/cache.py", line 175, in SetDefault File "/b/c/cbuild/repository/chromite/lib/cache.py", line 46, in new_f File "/b/c/cbuild/repository/chromite/lib/cache.py", line 121, in _Assign File "/b/c/cbuild/repository/chromite/lib/cache.py", line 271, in _Insert File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 97, in _Fetch File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 76, in _DownloadCIPD File "/b/c/cbuild/repository/chromite/lib/cipd.py", line 54, in _ChromeInfraRequest File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 1593, in request File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 1335, in _request File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 1257, in _conn_request File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 1021, in connect File "/b/c/cbuild/repository/chromite/third_party/httplib2/__init__.py", line 80, in _ssl_wrap_socket File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket ciphers=ciphers) File "/usr/lib/python2.7/ssl.py", line 241, in __init__ ciphers) ssl.SSLError: [Errno 185090050] _ssl.c:344: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib