New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 801211 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug



Sign in to add a comment

(Apparently) transitory problems fetching kernel from gerrit: fatal: Couldn't find remote ref HEAD

Project Member Reported by diand...@chromium.org, Jan 11 2018

Issue description

Looking through the CQ build failures from early this morning, I see these paladins got the same type of faiure:

  https://luci-milo.appspot.com/buildbot/chromeos/zoombini-paladin/1904
  https://luci-milo.appspot.com/buildbot/chromiumos/moblab-generic-vm-paladin/540
  https://luci-milo.appspot.com/buildbot/chromeos/lakitu-st-paladin/2030
  https://luci-milo.appspot.com/buildbot/chromeos/lakitu-paladin/8905
  https://luci-milo.appspot.com/buildbot/chromeos/lakitu-gpu-paladin/2488
  https://luci-milo.appspot.com/buildbot/chromeos/betty-paladin/1905
  https://luci-milo.appspot.com/buildbot/chromeos/betty-arc64-paladin/1510

The failure was in CommitQueueSync and looked like this (using betty-paladin as an example):

06:11:59: INFO: RunCommand: git fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel in /b/c/cbuild/repository/src/third_party/kernel/v3.8
06:17:41: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 177, in _Wrapper
    ret = func(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 243, in _run
    return functor(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 654, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 128; command: git fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel
fatal: Couldn't find remote ref HEAD
[W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n"
[W git.go:294] Retrying after 3s (rc=128): transient error string encountered
fatal: Couldn't find remote ref HEAD
[W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n"
[W git.go:294] Retrying after 4.5s (rc=128): transient error string encountered
fatal: Couldn't find remote ref HEAD
[W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n"
[W git.go:294] Retrying after 6.75s (rc=128): transient error string encountered
fatal: Couldn't find remote ref HEAD
[W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n"
[W git.go:294] Retrying after 10.125s (rc=128): transient error string encountered
fatal: Couldn't find remote ref HEAD
[W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n"
[W git.go:294] Retrying after 15.1875s (rc=128): transient error string encountered
fatal: Couldn't find remote ref HEAD
[W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n"
[W git.go:294] Retrying after 22.78125s (rc=128): transient error string encountered
fatal: Couldn't find remote ref HEAD
[W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n"
[W git.go:294] Retrying after 34.171875s (rc=128): transient error string encountered
fatal: Couldn't find remote ref HEAD
[W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n"
[W git.go:294] Retrying after 51.2578125s (rc=128): transient error string encountered
fatal: Couldn't find remote ref HEAD
[W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n"
[W git.go:294] Retrying after 1m16.88671875s (rc=128): transient error string encountered
fatal: Couldn't find remote ref HEAD
[W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n"
[W git.go:294] Retrying after 1m55.330078125s (rc=128): transient error string encountered
fatal: Couldn't find remote ref HEAD
[W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n"
[W git.go:317] Command completed with rc 128 after 10 transient failure(s).
...
...
...

---

Here is the same failure on many to get an idea for timing:

06:17:41: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying
06:18:12: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying
06:19:28: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying
06:15:54: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying
06:18:06: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying
06:15:55: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying
06:17:33: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying

 
Similar to  bug #801062 .  Mark as dupe?
Cc: adityakali@google.com sawlani@google.com
This is still happening.  Looking at "betty-arc64-paladin" #1511 I see:

08:24:03: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 177, in _Wrapper
    ret = func(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 243, in _run
    return functor(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 654, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 128; command: git fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel
fatal: Couldn't find remote ref HEAD

Labels: -Pri-2 Pri-0
This is blocking the CQ.  We need resolution sooner rather than later.
Cc: iannucci@chromium.org
Components: -Infra>Client>ChromeOS Infra
Owner: iannucci@chromium.org
Status: Assigned (was: Untriaged)
The stuff that's failing is mediated by code in Chrome Infra, not
Cros Infra.

So, roping in a trooper.

The lakitu paladins went green. But not a full sign of recovery yet.
We believe that this failure and  bug 801062  were both caused by a
GoB failure detailed at b/71872654.

Status: Fixed (was: Assigned)
It ain't happenin' no more, and GoB claims its fixed, so, let's
declare victory.

Sign in to add a comment