Various builders failing in InitialCheckout -- cannot fetch chromiumos/third_party/kernel |
||||||||
Issue description
As of about 3am this morning (July 29) the chromiumos-sdk builder started failing consistently in the InitialCheckout stage, with the error "cannot fetch chromiumos/third_party/kernel".
All of the tryjobs are also failing in this way.
Fetching project chromium/llvm-project/cfe/tools/clang-format
Fetching project chromiumos/third_party/dbus-cplusplus
error: Cannot fetch chromiumos/third_party/kernel
error: Exited sync due to fetch errors
real 0m56.381s
[1;33m06:50:31: WARNING: Command failed with retriable error.
return code: 1; command: repo --time sync --force-sync '--cache-dir=/b/swarming/w/ir/cache/git' -n
cmd=['repo', '--time', 'sync', '--force-sync', '--cache-dir=/b/swarming/w/ir/cache/git', '-n'], cwd=/b/swarming/w/ir/cache/cbuild/repository[0m
[1;31m06:50:31: ERROR: return code: 1; command: repo --time sync --force-sync '--cache-dir=/b/swarming/w/ir/cache/git' -n
cmd=['repo', '--time', 'sync', '--force-sync', '--cache-dir=/b/swarming/w/ir/cache/git', '-n'], cwd=/b/swarming/w/ir/cache/cbuild/repository[0m
06:50:31: INFO: Waiting for ts_mon flushing process to finish...
06:50:40: INFO: Finished waiting for ts_mon process.
cbuildbot_launch: Unhandled exception:
Traceback (most recent call last):
File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch", line 169, in <module>
DoMain()
File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch", line 165, in DoMain
commandline.ScriptWrapperMain(FindTarget)
File "/b/swarming/w/ir/kitchen-workdir/chromite/lib/commandline.py", line 911, in ScriptWrapperMain
ret = target(argv[1:])
File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 490, in main
return _main(argv)
File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 462, in _main
InitialCheckout(repo)
File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 72, in wrapped_functor
result = functor(*args, **kwargs)
File "/b/swarming/w/ir/kitchen-workdir/chromite/scripts/cbuildbot_launch.py", line 314, in InitialCheckout
repo.Sync(detach=True)
File "/b/swarming/w/ir/kitchen-workdir/chromite/cbuildbot/repository.py", line 702, in Sync
raise SrcCheckOutException(err_msg)
chromite.cbuildbot.repository.SrcCheckOutException: return code: 1; command: repo --time sync --force-sync '--cache-dir=/b/swarming/w/ir/cache/git' -n
cmd=['repo', '--time', 'sync', '--force-sync', '--cache-dir=/b/swarming/w/ir/cache/git', '-n'], cwd=/b/swarming/w/ir/cache/cbuild/repository
I would not be surprised if other "waterfall" builders start failing this way, too. This needs to be fixed asap.
,
Jul 29
It seems that "refs/heads/intel-wifi" was conflicted with "refs/heads/intel-wifi/*", please help to remove (or rename) the "refs/heads/intel-wifi" head.
,
Jul 29
,
Jul 29
I am attempting to fix but I do not have permissions on the kernel project.
,
Jul 29
,
Jul 29
I've deleted that branch now (01a2c378bd85641f74e2a179e6654c77ed3e1991) we should file a bug with the Gerrit team to disallow creating that kind of thing again not sure if we can easily find out who made it in the first place
,
Jul 29
Outage appears to be over. Leaving open for followup on Monday.
,
Jul 29
,
Jul 29
Normally "git show <branch head>" would show the last committer, but of course that doesn't work for deleted branches. Plus it looks like intel engineers may have permission to forge committers in intel-wireless/. Either case, does gerrit keep a push log ?
,
Jul 29
,
Jul 29
the git log data shows metadata about git commits, not about the branch itself (by design). so any commit could have been pushed by anyone and the metadata wouldn't match.
,
Jul 30
sorry for the delayed response - this was me and I am sorry. I was working on https://b.corp.google.com/issues/111819239 I noticed it as well, but the error log showed me that it didn't go through, so I didn't know to check for further damage. I corrected the command immediately afterward and it even went through. Console output here: https://paste.googleplex.com/5116289685127168?raw git push cros iwl7000/v4.14-core35:refs/heads/intel-wifi Enumerating objects: 649, done. Counting objects: 100% (567/567), done. Delta compression using up to 48 threads. Compressing objects: 100% (120/120), done. Writing objects: 100% (493/493), 63.92 KiB | 5.81 MiB/s, done. Total 493 (delta 435), reused 430 (delta 373) remote: Resolving deltas: 100% (435/435) remote: Processing changes: done remote: (W) 01a2c37: commit subject >50 characters; use shorter first paragraph remote: (W) 01a2c37: too many commit message lines longer than 72 characters; manually wrap lines [... snip ...] To https://chromium.googlesource.com/chromiumos/third_party/kernel * [new branch] iwl7000/v4.14-core35 -> intel-wifi error: update_ref failed for ref 'refs/remotes/cros/intel-wifi': cannot lock ref 'refs/remotes/cros/intel-wifi': 'refs/remotes/cros/intel-wifi/v3.10-Core18' exists; cannot create 'refs/remotes/cros/intel-wifi' ==== git push cros iwl7000/v4.14-core35:refs/heads/intel-wifi/v4.14-Core35 Total 0 (delta 0), reused 0 (delta 0) remote: Processing changes: done To https://chromium.googlesource.com/chromiumos/third_party/kernel 2d7fe0332980..01a2c378bd85 iwl7000/v4.14-core35 -> intel-wifi/v4.14-Core35
,
Jul 30
it's OK, i just wanted to make sure we had closure on the source to hopefully keep it from happening again i filed http://b/111993237 for GoB itself to reject this scenario. i think that covers everything on our side. |
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by xiaojs...@gmail.com
, Jul 29