Signing does not retry gsutil on SSL errors |
||||||
Issue descriptionFor example in this build: https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Frambi-release%2F1254%2F%2B%2Frecipes%2Fsteps%2FSigning%2F0%2Fstdout 06:03:39: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-releases/dev-channel/rambi/8865.0.0/ChromeOS-recovery-R55-8865.0.0-rambi.instructions.json [1;33m06:04:06: WARNING: GS_ERROR: Traceback (most recent call last): File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil", line 22, in <module> gsutil.RunMain() File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil.py", line 106, in RunMain sys.exit(gslib.__main__.main()) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/__main__.py", line 377, in main perf_trace_token=perf_trace_token) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/__main__.py", line 565, in _RunNamedCommandAndHandleExceptions parallel_operations, perf_trace_token=perf_trace_token) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/command_runner.py", line 280, in RunNamedCommand return_code = command_inst.RunCommand() File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/commands/cat.py", line 144, in RunCommand end_byte=end_byte) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/cat_helper.py", line 98, in CatUrlStrings decryption_tuple=decryption_tuple, provider=storage_url.scheme) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/cloud_api_delegator.py", line 252, in GetObjectMedia decryption_tuple=decryption_tuple) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/gcs_json_api.py", line 963, in GetObjectMedia decryption_tuple=decryption_tuple) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/gcs_json_api.py", line 1053, in _PerformDownload additional_headers=additional_headers, use_chunks=False) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 522, in StreamMedia additional_headers=additional_headers) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 411, in __GetChunk retries=self.num_retries) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 349, in MakeRequest http, http_request, e, retry, max_retry_wait)) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 340, in MakeRequest check_response_func=check_response_func) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 389, in _MakeRequestNoRetry redirections=redirections, connection_type=connection_type) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/gcs_json_media.py", line 423, in NewRequest connection_type=connection_type) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/oauth2client/oauth2client/client.py", line 596, in new_request redirections, connection_type) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/httplib2/python2/httplib2/__init__.py", line 1570, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/gcs_json_media.py", line 331, in OverrideRequest headers) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/gcs_json_media.py", line 520, in _conn_request conn.connect() File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/httplib2/python2/httplib2/__init__.py", line 1021, in connect self.disable_ssl_certificate_validation, self.ca_certs) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/httplib2/python2/httplib2/__init__.py", line 80, in _ssl_wrap_socket cert_reqs=cert_reqs, ca_certs=ca_certs) File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket ciphers=ciphers) File "/usr/lib/python2.7/ssl.py", line 243, in __init__ self.do_handshake() File "/usr/lib/python2.7/ssl.py", line 405, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [Errno 8] _ssl.c:510: EOF occurred in violation of protocol (Temp log for crbug.com/642986 )[0m [1;31m06:04:06: ERROR: Traceback (most recent call last): File "/b/cbuild/internal_master/chromite/cbuildbot/stages/generic_stages.py", line 546, in Run self.PerformStage() File "/b/cbuild/internal_master/chromite/cbuildbot/stages/release_stages.py", line 296, in PerformStage self._WaitForSigningResults(self.instruction_urls_per_channel) File "/b/cbuild/internal_master/chromite/cbuildbot/stages/release_stages.py", line 241, in _WaitForSigningResults timeout=self.SIGNING_TIMEOUT, period=self.SIGNING_PERIOD) File "/b/cbuild/internal_master/chromite/lib/timeout_util.py", line 193, in WaitForReturnTrue WaitForReturnValue([True], *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/timeout_util.py", line 215, in WaitForReturnValue return WaitForSuccess(_Retry, *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/timeout_util.py", line 297, in WaitForSuccess return retry() File "/b/cbuild/internal_master/chromite/lib/timeout_util.py", line 270, in retry value = func(*func_args, **func_kwargs) File "/b/cbuild/internal_master/chromite/cbuildbot/stages/release_stages.py", line 190, in _CheckForResults signer_json = self._JsonFromUrl(gs_ctx, url) File "/b/cbuild/internal_master/chromite/cbuildbot/stages/release_stages.py", line 134, in _JsonFromUrl signer_txt = gs_ctx.Cat(url) File "/b/cbuild/internal_master/chromite/lib/gs.py", line 512, in Cat return self.DoCommand(['cat', path], **kwargs).output File "/b/cbuild/internal_master/chromite/lib/gs.py", line 726, in DoCommand raise GSCommandError(e.msg, e.result, e.exception) GSCommandError: return code: 1; command: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-releases/dev-channel/rambi/8865.0.0/ChromeOS-recovery-R55-8865.0.0-rambi.instructions.json Traceback (most recent call last): File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil", line 22, in <module> gsutil.RunMain() File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil.py", line 106, in RunMain sys.exit(gslib.__main__.main()) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/__main__.py", line 377, in main perf_trace_token=perf_trace_token) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/__main__.py", line 565, in _RunNamedCommandAndHandleExceptions parallel_operations, perf_trace_token=perf_trace_token) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/command_runner.py", line 280, in RunNamedCommand return_code = command_inst.RunCommand() File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/commands/cat.py", line 144, in RunCommand end_byte=end_byte) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/cat_helper.py", line 98, in CatUrlStrings decryption_tuple=decryption_tuple, provider=storage_url.scheme) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/cloud_api_delegator.py", line 252, in GetObjectMedia decryption_tuple=decryption_tuple) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/gcs_json_api.py", line 963, in GetObjectMedia decryption_tuple=decryption_tuple) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/gcs_json_api.py", line 1053, in _PerformDownload additional_headers=additional_headers, use_chunks=False) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 522, in StreamMedia additional_headers=additional_headers) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 411, in __GetChunk retries=self.num_retries) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 349, in MakeRequest http, http_request, e, retry, max_retry_wait)) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 340, in MakeRequest check_response_func=check_response_func) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 389, in _MakeRequestNoRetry redirections=redirections, connection_type=connection_type) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/gcs_json_media.py", line 423, in NewRequest connection_type=connection_type) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/oauth2client/oauth2client/client.py", line 596, in new_request redirections, connection_type) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/httplib2/python2/httplib2/__init__.py", line 1570, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/gcs_json_media.py", line 331, in OverrideRequest headers) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gslib/gcs_json_media.py", line 520, in _conn_request conn.connect() File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/httplib2/python2/httplib2/__init__.py", line 1021, in connect self.disable_ssl_certificate_validation, self.ca_certs) File "/b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/third_party/httplib2/python2/httplib2/__init__.py", line 80, in _ssl_wrap_socket cert_reqs=cert_reqs, ca_certs=ca_certs) File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket ciphers=ciphers) File "/usr/lib/python2.7/ssl.py", line 243, in __init__ self.do_handshake() File "/usr/lib/python2.7/ssl.py", line 405, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [Errno 8] _ssl.c:510: EOF occurred in violation of protocol cwd=None, extra env={'BOTO_CONFIG': '/b/build/site_config/.boto'} I guess this might just have been a network fluke.. but we should probably retry instead of failing the whole build? Luigi, I am not sure who would be the right person to look into that, could you help me find someone?
,
Oct 12 2016
Dan, Richard, any idea?
,
Oct 12 2016
,
Oct 12 2016
That error happened 100% inside the gsutil tool, which we don't own. However, I THINK the chromite/lib/gs.py wrapper would have retried that by reinvoking the command.
,
Oct 12 2016
And I was wrong, gs.py raised a GSCommandError, which causes the stage to exit. Instead, we should have at least retried at the stage level. Aseda already has a CL to add a new retry reason to gs.py, maybe we can add this one as well.
,
Oct 15 2016
CL adding new retry reason here: https://chromium-review.googlesource.com/#/c/399679/
,
Oct 17 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/bfd179865c093d65c4c3598b74c90f6d2680687b commit bfd179865c093d65c4c3598b74c90f6d2680687b Author: Aseda Aboagye <aaboagye@google.com> Date: Sat Oct 15 00:27:33 2016 gs: Add a new SSL error to RetryFilter. It's been seen that a SSL error can surface possibly due to network flakes. This error gets raised to a GSCommandError which can fail a stage. The error that is encountered is the following: `ssl.SSLError: [Errno 8] _ssl.c:510: EOF occurred in violation of protocol` This commit adds portions of that string to the RetryFilter. BUG= chromium:653137 BRANCH=None TEST=python gs_unittest --network Change-Id: I6c63259e4efdfcfd1f97a854f67e7de95714989b Signed-off-by: Aseda Aboagye <aaboagye@google.com> Reviewed-on: https://chromium-review.googlesource.com/399679 Commit-Ready: Aseda Aboagye <aaboagye@chromium.org> Tested-by: Aseda Aboagye <aaboagye@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/bfd179865c093d65c4c3598b74c90f6d2680687b/lib/gs_unittest.py [modify] https://crrev.com/bfd179865c093d65c4c3598b74c90f6d2680687b/lib/gs.py
,
Oct 18 2016
Verified in unit test. Feel free to reopen if this resurfaces. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by sheriffbot@chromium.org
, Oct 7 2016