One example:
https://ci.chromium.org/b/8930012596068633696
https://logs.chromium.org/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8930012596068633696/+/steps/PublishUprevChanges/0/stdout
...
Marking 9999 ebuild for chromeos-base/chromeos-firmware-nocturne as stable.
Change-Id: I20a1c52fc1eb10ef3f8617a1f8f5d69b27928495
21:19:28: INFO: Successfully pushed /b/swarming/wGfv7iv/ir/cache/cbuild/repository/src/private-overlays/overlay-nocturne-private to cros-internal merge_branch:refs/heads/release-R71-11151.B
21:19:29: INFO: For /b/swarming/wGfv7iv/ir/cache/cbuild/repository/src/private-overlays/chromeos-partner-overlay, using description Marking set of ebuilds as stable
samus-PREFLIGHT_BINHOST.conf: updating PREFLIGHT_BINHOST
Change-Id: Ic22b0e53e08c9a77e3a80f6c9cc6a4058d3227c1
21:19:32: INFO: Successfully pushed /b/swarming/wGfv7iv/ir/cache/cbuild/repository/src/private-overlays/chromeos-partner-overlay to cros-internal merge_branch:refs/heads/release-R71-11151.B
cros_mark_as_stable: Unhandled exception:
Traceback (most recent call last):
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/bin/cros_mark_as_stable", line 169, in <module>
DoMain()
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/bin/cros_mark_as_stable", line 165, in DoMain
commandline.ScriptWrapperMain(FindTarget)
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/commandline.py", line 912, in ScriptWrapperMain
ret = target(argv[1:])
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/scripts/cros_mark_as_stable.py", line 303, in main
_WorkOnPush(options, overlay_tracking_branch, git_project_overlays)
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/scripts/cros_mark_as_stable.py", line 321, in _WorkOnPush
parallel.RunTasksInProcessPool(_PushOverlays, inputs)
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 809, in RunTasksInProcessPool
queue.put((idx, input_args))
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 750, in BackgroundTaskRunner
queue.put(_AllTasksComplete())
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 750, in BackgroundTaskRunner
queue.put(_AllTasksComplete())
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 561, in ParallelTasks
raise BackgroundFailure(exc_infos=errors)
chromite.lib.parallel.BackgroundFailure: <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: 128; command: git rebase refs/remotes/cros/release-R71-11151.B
error: Could not open metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.49_rc-r1: No such file or directory
error: Failed to merge in the changes.
First, rewinding head to replay your work on top of it...
Applying: Marking 9999 ebuild for sys-kernel/linux-firmware as stable.
Applying: regen cache
Using index info to reconstruct a base tree...
A metadata/md5-cache/chromeos-base/chromeos-chrome-71.0.3578.48_rc-r1
M metadata/md5-cache/chromeos-base/chromeos-chrome-9999
A metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.48_rc-r1
M metadata/md5-cache/sys-kernel/chromeos-kernel-3_14-3.14-r2074
M metadata/md5-cache/sys-kernel/chromeos-kernel-3_14-9999
M metadata/md5-cache/sys-kernel/chromeos-kernel-3_18-3.18-r2222
M metadata/md5-cache/sys-kernel/chromeos-kernel-3_18-9999
M metadata/md5-cache/sys-kernel/chromeos-kernel-3_8-3.8.11-r749
M metadata/md5-cache/sys-kernel/chromeos-kernel-3_8-9999
M metadata/md5-cache/sys-kernel/chromeos-kernel-4_4-4.4.159-r1704
M metadata/md5-cache/sys-kernel/chromeos-kernel-4_4-9999
Falling back to patching base and 3-way merge...
Auto-merging metadata/md5-cache/sys-kernel/linux-firmware-0.0.1-r208
CONFLICT (rename/rename): Rename "metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.48_rc-r1"->"metadata/md5-cache/chromeos-base/chromium-source-71.0.3575.1_rc-r1" in branch "HEAD" rename "metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.48_rc-r1"->"metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.49_rc-r1" in "regen cache"
Patch failed at 0002 regen cache
Use 'git am --show-current-patch' to see the failed patch
Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".
cmd=['git', 'rebase', u'refs/remotes/cros/release-R71-11151.B'], cwd=/b/swarming/wGfv7iv/ir/cache/cbuild/repository/src/third_party/chromiumos-overlay
Traceback (most recent call last):
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 602, in TaskRunner
task(*x, **task_kwargs)
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 800, in <lambda>
fn = lambda idx, task_args: out_queue.put((idx, task(*task_args)))
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/scripts/cros_mark_as_stable.py", line 340, in _PushOverlays
cwd=overlay, staging_branch=options.staging_branch)
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/scripts/cros_mark_as_stable.py", line 130, in PushChange
git.SyncPushBranch(cwd, remote_ref.remote, remote_ref.ref)
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/git.py", line 1323, in SyncPushBranch
RunGit(git_repo, [subcommand, target], **kwargs)
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/git.py", line 785, in RunGit
return cros_build_lib.RunCommand(['git'] + cmd, **kwargs)
File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/cros_build_lib.py", line 647, in RunCommand
raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 128; command: git rebase refs/remotes/cros/release-R71-11151.B
error: Could not open metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.49_rc-r1: No such file or directory
error: Failed to merge in the changes.
First, rewinding head to replay your work on top of it...
Applying: Marking 9999 ebuild for sys-kernel/linux-firmware as stable.
Applying: regen cache
Using index info to reconstruct a base tree...
A metadata/md5-cache/chromeos-base/chromeos-chrome-71.0.3578.48_rc-r1
M metadata/md5-cache/chromeos-base/chromeos-chrome-9999
A metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.48_rc-r1
M metadata/md5-cache/sys-kernel/chromeos-kernel-3_14-3.14-r2074
M metadata/md5-cache/sys-kernel/chromeos-kernel-3_14-9999
M metadata/md5-cache/sys-kernel/chromeos-kernel-3_18-3.18-r2222
M metadata/md5-cache/sys-kernel/chromeos-kernel-3_18-9999
M metadata/md5-cache/sys-kernel/chromeos-kernel-3_8-3.8.11-r749
M metadata/md5-cache/sys-kernel/chromeos-kernel-3_8-9999
M metadata/md5-cache/sys-kernel/chromeos-kernel-4_4-4.4.159-r1704
M metadata/md5-cache/sys-kernel/chromeos-kernel-4_4-9999
Falling back to patching base and 3-way merge...
Auto-merging metadata/md5-cache/sys-kernel/linux-firmware-0.0.1-r208
CONFLICT (rename/rename): Rename "metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.48_rc-r1"->"metadata/md5-cache/chromeos-base/chromium-source-71.0.3575.1_rc-r1" in branch "HEAD" rename "metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.48_rc-r1"->"metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.49_rc-r1" in "regen cache"
Patch failed at 0002 regen cache
Use 'git am --show-current-patch' to see the failed patch
Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".
cmd=['git', 'rebase', u'refs/remotes/cros/release-R71-11151.B'], cwd=/b/swarming/wGfv7iv/ir/cache/cbuild/repository/src/third_party/chromiumos-overlay
[1;31m21:19:48: ERROR:
return code: 1; command: /b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/bin/cros_mark_as_stable push --buildroot /b/swarming/wGfv7iv/ir/cache/cbuild/repository --overlay-type both
cmd=['/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/bin/cros_mark_as_stable', 'push', '--buildroot', '/b/swarming/wGfv7iv/ir/cache/cbuild/repository', '--overlay-type', u'both'], cwd=/b/swarming/wGfv7iv/ir/cache/cbuild/repository[0m
[1;31m21:19:48: ERROR: /b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/bin/cros_mark_as_stable failed (code=1)[0m
...
This causes unexpected missing Chrome uprevs on the branch, for instance last night it failed and we now are a day behind in the release for a critical device.
Comment 1 by ayatane@chromium.org
, Nov 27