New issue
Advanced search Search tips

Issue 792679 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Dec 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 3
Type: Bug



Sign in to add a comment

git cl upload fails with GerritError

Project Member Reported by skobes@chromium.org, Dec 6 2017

Issue description

$ git cl upload
Running presubmit upload checks ...

Presubmit checks took 20.4s to calculate.

Presubmit checks passed.
 third_party/WebKit/LayoutTests/FlagExpectations/root-layer-scrolls |  1 -
 third_party/WebKit/Source/core/frame/LocalFrameView.cpp            | 16 +++++++++-------
 third_party/WebKit/Source/core/frame/LocalFrameView.h              |  2 +-
 3 files changed, 10 insertions(+), 9 deletions(-)
remote: Processing changes: new: 1, done            
remote: (W) a15c5be: commit subject >50 characters; use shorter first paragraph        
remote: 
remote: New Changes:        
remote:   https://chromium-review.googlesource.com/#/c/chromium/src/+/812384 Update LFV::MainThreadScrollingReasonsAsText for RLS. [WIP]        
remote: 
To https://chromium.googlesource.com/chromium/src.git
 * [new branch]                a15c5be02b4020814dc9b7e77eeb3f97b3739cdd -> refs/for/refs/heads/master%wip,m=Initial_upload,hashtag=update-lfv
[W2017-12-06 14:26:39,332 19401 140089210701632 gerrit_util.py] A transient error occurred while querying chromium-review.googlesource.com:
POST https://chromium-review.googlesource.com/a/changes/812384/revisions/current/review HTTP/1.1
HTTP/1.1 500 Internal Server Error
[W2017-12-06 14:26:39,332 19401 140089210701632 gerrit_util.py] ... will retry 4 more times.
[W2017-12-06 14:26:39,972 19401 140089210701632 gerrit_util.py] A transient error occurred while querying chromium-review.googlesource.com:
POST https://chromium-review.googlesource.com/a/changes/812384/revisions/current/review HTTP/1.1
HTTP/1.1 500 Internal Server Error
[W2017-12-06 14:26:39,972 19401 140089210701632 gerrit_util.py] ... will retry 3 more times.
[W2017-12-06 14:26:41,098 19401 140089210701632 gerrit_util.py] A transient error occurred while querying chromium-review.googlesource.com:
POST https://chromium-review.googlesource.com/a/changes/812384/revisions/current/review HTTP/1.1
HTTP/1.1 404 Not Found
[W2017-12-06 14:26:41,098 19401 140089210701632 gerrit_util.py] ... will retry 2 more times.
[W2017-12-06 14:26:43,234 19401 140089210701632 gerrit_util.py] A transient error occurred while querying chromium-review.googlesource.com:
POST https://chromium-review.googlesource.com/a/changes/812384/revisions/current/review HTTP/1.1
HTTP/1.1 404 Not Found
[W2017-12-06 14:26:43,234 19401 140089210701632 gerrit_util.py] ... will retry 1 more times.
[W2017-12-06 14:26:47,382 19401 140089210701632 gerrit_util.py] A transient error occurred while querying chromium-review.googlesource.com:
POST https://chromium-review.googlesource.com/a/changes/812384/revisions/current/review HTTP/1.1
HTTP/1.1 404 Not Found
Traceback (most recent call last):
  File "/b/depot_tools/git_cl.py", line 6193, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/b/depot_tools/git_cl.py", line 6175, in main
    return dispatcher.execute(OptionParser(), argv)
  File "/b/depot_tools/subcommand.py", line 252, in execute
    return command(parser, args[1:])
  File "/b/depot_tools/git_cl.py", line 5036, in CMDupload
    return cl.CMDUpload(options, args, orig_args)
  File "/b/depot_tools/git_cl.py", line 1665, in CMDUpload
    ret = self.CMDUploadChange(options, git_diff_args, custom_cl_base, change)
  File "/b/depot_tools/git_cl.py", line 3129, in CMDUploadChange
    notify=bool(options.send_mail))
  File "/b/depot_tools/gerrit_util.py", line 684, in AddReviewers
    resp = ReadHttpJsonResponse(conn, accept_statuses=accept_statuses)
  File "/b/depot_tools/gerrit_util.py", line 394, in ReadHttpJsonResponse
    fh = ReadHttpResponse(conn, accept_statuses)
  File "/b/depot_tools/gerrit_util.py", line 388, in ReadHttpResponse
    raise GerritError(response.status, reason)
gerrit_util.GerritError: Not Found: Not found: 812384


 
Unfortunately, this is a known issue when Gerrit is experiencing replication lag, and the initial "git push" goes to a different datacenter than the followup HTTP POSTs.

We have the retries in there to try to make up for it, with backoff (immediate, 1s, 2s, 4s) but I'm not really sure what to do when even that fails.
I think a slower retry might have helped (like 10 s, 20 s...)
Owner: aga...@chromium.org
Status: Started (was: Unconfirmed)
There's a balancing act between retrying enough to get it to work (desirable for users) and stopping reasonably quickly on a bot.

Here's a minor tweak that will quadruple the total number of seconds covered by retries for this particular operation: https://chromium-review.googlesource.com/#/c/chromium/tools/depot_tools/+/812564
Project Member

Comment 4 by bugdroid1@chromium.org, Dec 7 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/92e9f388659a2a4104766b5abbc5ab485984a7d7

commit 92e9f388659a2a4104766b5abbc5ab485984a7d7
Author: Aaron Gable <agable@chromium.org>
Date: Thu Dec 07 19:53:45 2017

gerrit_util: Slightly modify retry backoff behavior

This is a bandaid (over Gerrit's replication lag), but hopefully
this will increase the likelihood of success and improve the
messaging around retries at the same time.

R=tandrii

Bug:  792679 
Change-Id: Ia6e3133cc23ea9b2abfeacc05750c002f8804e8b
Reviewed-on: https://chromium-review.googlesource.com/812564
Commit-Queue: Aaron Gable <agable@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/92e9f388659a2a4104766b5abbc5ab485984a7d7/gerrit_util.py

Status: Fixed (was: Started)
Hopefully that will help.

Sign in to add a comment