Retry worked on at least one of the bots: https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_compile_dbg_ng/builds/416568

relevant portion of the log here:

===Running git reset --hard (attempt #1)===
In directory: src
HEAD is now at 4ae97f1 Add ModuleDatabaseObserver to observe loaded modules.
===Succeeded in 0.1 mins===
===Running git retry fetch https://chromium.googlesource.com/chromium/src refs/changes/16/502016/2 (attempt #1)===
In directory: src
fatal: Couldn't find remote ref refs/changes/16/502016/2
===Failed in 0.0 mins===

It runs "git retry fetch <repo> <ref>", fails, and then... doesn't retry.

The reason the ref didn't seem to exist is almost certainly gerrit's replication latency. So retrying with some exponential backoff is exactly the right thing to do. It's not clear to me why the recipe isn't doing this.

Dan, I know you wrote git-retry, can you take a look?

The bots (presumably) are now using the git wrapper and the `git retry` command is just a passthrough.

However, I would have expected to see some retry evidence from the git wrapper in the logs.

So this is actually really interesting. The Git wrapper is being used (see "version" up top), and local testing shows that it does successfully retry that fetch error. However, it has an explicit whitelist of commands that it will retry, and that doesn't include the "retry" subcommand.

The design was that "git_retry.py" would just fall through to "git" which would, in turn, kick off another Git wrapper round that actually does the fetch and retries it; however, the Git invocation for the first "git retry" prepends "/usr/lib/git-core" to PATH, causing all subsequent Git invocations to circumvent the wrapper and call Git directly.

So actually calling "git retry" explicitly is blocking retries :/

Short term I think the solution here is for "git_retry.py" to pop the Git path from PATH. A better solution is to stop calling "git retry" in general.

More info on the prepend: https://github.com/gitlabhq/gitlabhq/issues/8045#issuecomment-59495603

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/e2af38e019298d7f5a2762830fdda228d8205bd9

commit e2af38e019298d7f5a2762830fdda228d8205bd9
Author: Dan Jacques <dnj@chromium.org>
Date: Tue May 16 00:59:07 2017

[git retry] Fix Git wrapper fallthrough.

Currently, when the Infra Git wrapper is installed in PATH and a user
calls "git retry", the "git_retry.py" script is supposed to ignore the
request and fall through to the underlying command, relying on the Git
wrapper to handle retries.

However, Git apparently prepends its own executable path to PATH when it
is called, preventing the Git wrapper from being invoked the second time
and removing retries altogether.

We circumvent this by removing Git's new PATH influence when falling
through.

BUG= chromium:721450 
TEST=local
  - Ran locally before and after patch, confirmed that after
  successfully retries throgh the wrapper.

R=agable@chromium.org, iannucci@chromium.org

Change-Id: Iae3d7a8bf805a5ba2bf827b06006a990d94e96d9
Reviewed-on: https://chromium-review.googlesource.com/506374
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/e2af38e019298d7f5a2762830fdda228d8205bd9/git_retry.py

This should be fixed now.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/a3ba888db8f00dce68cbd6e36fd058ae1d471bc1

commit a3ba888db8f00dce68cbd6e36fd058ae1d471bc1
Author: Dan Jacques <dnj@chromium.org>
Date: Tue May 16 01:16:39 2017

[bot_update] Remove "git retry" call.

The Git wrapper now implicitly retries, so this call is unnecessary.

BUG= chromium:721450 
TEST=None
R=hinoka@chromium.org, iannucci@chromium.org

Change-Id: I79e42aa050f6fec14506b685d379a76e8296fea3
Reviewed-on: https://chromium-review.googlesource.com/506493
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/a3ba888db8f00dce68cbd6e36fd058ae1d471bc1/recipes/recipe_modules/bot_update/resources/bot_update.py