(Apparently) transitory problems fetching kernel from gerrit: fatal: Couldn't find remote ref HEAD |
||||||
Issue descriptionLooking through the CQ build failures from early this morning, I see these paladins got the same type of faiure: https://luci-milo.appspot.com/buildbot/chromeos/zoombini-paladin/1904 https://luci-milo.appspot.com/buildbot/chromiumos/moblab-generic-vm-paladin/540 https://luci-milo.appspot.com/buildbot/chromeos/lakitu-st-paladin/2030 https://luci-milo.appspot.com/buildbot/chromeos/lakitu-paladin/8905 https://luci-milo.appspot.com/buildbot/chromeos/lakitu-gpu-paladin/2488 https://luci-milo.appspot.com/buildbot/chromeos/betty-paladin/1905 https://luci-milo.appspot.com/buildbot/chromeos/betty-arc64-paladin/1510 The failure was in CommitQueueSync and looked like this (using betty-paladin as an example): 06:11:59: INFO: RunCommand: git fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel in /b/c/cbuild/repository/src/third_party/kernel/v3.8 [1;33m06:17:41: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying Traceback (most recent call last): File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 177, in _Wrapper ret = func(*args, **kwargs) File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 243, in _run return functor(*args, **kwargs) File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 654, in RunCommand raise RunCommandError(msg, cmd_result) RunCommandError: return code: 128; command: git fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel fatal: Couldn't find remote ref HEAD [W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n" [W git.go:294] Retrying after 3s (rc=128): transient error string encountered fatal: Couldn't find remote ref HEAD [W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n" [W git.go:294] Retrying after 4.5s (rc=128): transient error string encountered fatal: Couldn't find remote ref HEAD [W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n" [W git.go:294] Retrying after 6.75s (rc=128): transient error string encountered fatal: Couldn't find remote ref HEAD [W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n" [W git.go:294] Retrying after 10.125s (rc=128): transient error string encountered fatal: Couldn't find remote ref HEAD [W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n" [W git.go:294] Retrying after 15.1875s (rc=128): transient error string encountered fatal: Couldn't find remote ref HEAD [W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n" [W git.go:294] Retrying after 22.78125s (rc=128): transient error string encountered fatal: Couldn't find remote ref HEAD [W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n" [W git.go:294] Retrying after 34.171875s (rc=128): transient error string encountered fatal: Couldn't find remote ref HEAD [W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n" [W git.go:294] Retrying after 51.2578125s (rc=128): transient error string encountered fatal: Couldn't find remote ref HEAD [W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n" [W git.go:294] Retrying after 1m16.88671875s (rc=128): transient error string encountered fatal: Couldn't find remote ref HEAD [W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n" [W git.go:294] Retrying after 1m55.330078125s (rc=128): transient error string encountered fatal: Couldn't find remote ref HEAD [W git.go:283] Transient error string identified in STDERR: "fatal: Couldn't find remote ref HEAD\n" [W git.go:317] Command completed with rc 128 after 10 transient failure(s). ... ... ... --- Here is the same failure on many to get an idea for timing: 06:17:41: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying 06:18:12: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying 06:19:28: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying 06:15:54: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying 06:18:06: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying 06:15:55: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying 06:17:33: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying
,
Jan 11 2018
,
Jan 11 2018
This is still happening. Looking at "betty-arc64-paladin" #1511 I see: 08:24:03: WARNING: git reported transient error (cmd=fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel); retrying Traceback (most recent call last): File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 177, in _Wrapper ret = func(*args, **kwargs) File "/b/c/cbuild/repository/chromite/lib/retry_util.py", line 243, in _run return functor(*args, **kwargs) File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 654, in RunCommand raise RunCommandError(msg, cmd_result) RunCommandError: return code: 128; command: git fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel fatal: Couldn't find remote ref HEAD
,
Jan 11 2018
This is blocking the CQ. We need resolution sooner rather than later.
,
Jan 11 2018
,
Jan 11 2018
The stuff that's failing is mediated by code in Chrome Infra, not Cros Infra. So, roping in a trooper.
,
Jan 11 2018
The lakitu paladins went green. But not a full sign of recovery yet.
,
Jan 12 2018
A few of the paladin builds failed again with this cause. https://luci-milo.appspot.com/buildbot/chromeos/betty-arc64-paladin/1514 https://luci-milo.appspot.com/buildbot/chromeos/betty-paladin/1909 https://luci-milo.appspot.com/buildbot/chromeos/lakitu-gpu-paladin/2492 https://luci-milo.appspot.com/buildbot/chromeos/lakitu-paladin/8909 https://luci-milo.appspot.com/buildbot/chromeos/lakitu-st-paladin/2034
,
Jan 12 2018
We believe that this failure and bug 801062 were both caused by a GoB failure detailed at b/71872654.
,
Jan 12 2018
It ain't happenin' no more, and GoB claims its fixed, so, let's declare victory. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by diand...@chromium.org
, Jan 11 2018