master paladin fails at CommitQueueCompletion stage because of GOBError: Length Required |
|||
Issue descriptionhttps://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/11720/steps/CommitQueueCompletion/logs/stdio logs: [patch_branch 5757b75] Increase test timeout allowances for canaries. Author: Richard Barnette <jrbarnette@chromium.org> Date: Fri Jul 8 09:41:23 2016 -0700 4 files changed, 142 insertions(+), 141 deletions(-) 14:44:09: WARNING: A transient error occured while querying chromium-review.googlesource.com: POST https://chromium-review.googlesource.com/a/changes/356691/check HTTP/1.0 HTTP/1.0 411 Length Required Response body: '<!DOCTYPE html>\n<html lang=en>\n <meta charset=utf-8>\n <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">\n <title>Error 411 (Length Required)!!1</title>\n <style>\n *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\n </style>\n <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\n <p><b>411.</b> <ins>That\xe2\x80\x99s an error.</ins>\n <p>POST requests require a <code>Content-length</code> header. <ins>That\xe2\x80\x99s all we know.</ins>\n' X-ErrorId: None 14:44:09: WARNING: peer name unavailable @@@STEP_FAILURE@@@ 14:44:09: ERROR: Traceback (most recent call last): File "/b/cbuild/internal_master/chromite/cbuildbot/stages/generic_stages.py", line 523, in Run self.PerformStage() File "/b/cbuild/internal_master/chromite/cbuildbot/stages/completion_stages.py", line 836, in PerformStage super(CommitQueueCompletionStage, self).PerformStage() File "/b/cbuild/internal_master/chromite/cbuildbot/stages/completion_stages.py", line 288, in PerformStage self.HandleSuccess() File "/b/cbuild/internal_master/chromite/cbuildbot/stages/completion_stages.py", line 483, in HandleSuccess self.sync_stage.pool.SubmitPool(reason=constants.STRATEGY_CQ_SUCCESS) File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 2407, in SubmitPool throttled_ok=throttled_ok) File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 1986, in SubmitChanges by_repo) File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 2055, in SubmitLocalChanges changes, errors = self._SubmitRepo(repo, verified_cls) File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 2092, in _SubmitRepo sha1=sha1s.get(change)) File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 2271, in _CheckChangeWasSubmitted gob_util.CheckChange(helper.host, change.gerrit_number, sha1=sha1) File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 427, in CheckChange headers=headers) File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 246, in FetchUrlJson fh = FetchUrl(*args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 238, in FetchUrl _FetchUrlHelper, sleep=SLEEP) File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 127, in RetryException return GenericRetry(exc_retry, max_retry, functor, *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 88, in GenericRetry ret = functor(*args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 235, in _FetchUrlHelper raise GOBError(response.status, response.reason) GOBError: Length Required
,
Jul 11 2016
or it could be a gerrit flake???
,
Jul 11 2016
If it only happens once, flake is extremely likely.
,
Jul 11 2016
ok, wait for next run result
,
Jul 12 2016
It fails again, every time it finishes pushing whatever the first CL, it will fail with this "GOBError: Length Required" https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/11721/steps/CommitQueueCompletion/logs/stdio
,
Jul 12 2016
From comment 0: POST https://chromium-review.googlesource.com/a/changes/358143/check HTTP/1.0 HTTP/1.0 411 Length Required Response body: [...]<p>POST requests require a <code>Content-length</code> header Why is this a POST request? https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#check-change says /check is a GET.
,
Jul 12 2016
,
Jul 12 2016
CheckChange already sets Content-Length to 0 when the body is empty. I would have expected the underlying request library to take care of setting content-length when there is a body, but maybe that doesn't happen. Does something like this help?
diff --git i/lib/gob_util.py w/lib/gob_util.py
index d1a6301..532fdf2 100644
--- i/lib/gob_util.py
+++ w/lib/gob_util.py
@@ -118,6 +118,7 @@ def CreateHttpConn(host, path, reqtype='GET', headers=None, body=None):
if body:
body = json.JSONEncoder().encode(body)
headers.setdefault('Content-Type', 'application/json')
+ headers.setdefault('Content-Length', str(len(body)))
if logging.getLogger().isEnabledFor(logging.DEBUG):
logging.debug('%s https://%s/a/%s', reqtype, host, path)
for key, val in headers.iteritems():
,
Jul 12 2016
https://chromium-review.googlesource.com/#/c/357963/ I find this CL removes the Content-Length on line 409. It probably causes the failures
,
Jul 12 2016
Ah, good catch.
,
Jul 12 2016
It's still a mystery to me because https://docs.python.org/2/library/httplib.html#httplib.HTTPConnection.request says If one is not provided in headers, a Content-Length header is added automatically for all methods if the length of the body can be determined, either from the length of the str representation, or from the reported size of the file on disk. If body is None the header is not set except for methods that expect a body (PUT, POST, and PATCH) in which case it is set to 0. But I think you found the problem.
,
Jul 12 2016
I think it would help, but I don't get why that CL needs to remove the Content-Length. Already leave comments on that CL.
,
Jul 12 2016
*nod* The content-length change should not have been part of that CL. Thanks for catching it. I still don't understand the symptom, but *shrug*.
,
Jul 12 2016
The following latest build still fails. https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/11722/steps/CommitQueueCompletion/logs/stdio I don't think the CL https://chromium-review.googlesource.com/#/c/357963/ CL is the issue.
,
Aug 5 2016
|
|||
►
Sign in to add a comment |
|||
Comment 1 by shuqianz@chromium.org
, Jul 11 2016Owner: dnojiri@chromium.org