New issue
Advanced search Search tips

Issue 904868 link

Starred by 2 users

Issue metadata

Status: Untriaged
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Chrome branch PFQs occasionally fail PublishUprevChanges with a git rebase failure

Project Member Reported by bhthompson@google.com, Nov 13

Issue description

One example:
https://ci.chromium.org/b/8930012596068633696

https://logs.chromium.org/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8930012596068633696/+/steps/PublishUprevChanges/0/stdout

...
Marking 9999 ebuild for chromeos-base/chromeos-firmware-nocturne as stable.

Change-Id: I20a1c52fc1eb10ef3f8617a1f8f5d69b27928495

21:19:28: INFO: Successfully pushed /b/swarming/wGfv7iv/ir/cache/cbuild/repository/src/private-overlays/overlay-nocturne-private to cros-internal merge_branch:refs/heads/release-R71-11151.B
21:19:29: INFO: For /b/swarming/wGfv7iv/ir/cache/cbuild/repository/src/private-overlays/chromeos-partner-overlay, using description Marking set of ebuilds as stable

samus-PREFLIGHT_BINHOST.conf: updating PREFLIGHT_BINHOST

Change-Id: Ic22b0e53e08c9a77e3a80f6c9cc6a4058d3227c1

21:19:32: INFO: Successfully pushed /b/swarming/wGfv7iv/ir/cache/cbuild/repository/src/private-overlays/chromeos-partner-overlay to cros-internal merge_branch:refs/heads/release-R71-11151.B
cros_mark_as_stable: Unhandled exception:
Traceback (most recent call last):
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/bin/cros_mark_as_stable", line 169, in <module>
    DoMain()
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/bin/cros_mark_as_stable", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/commandline.py", line 912, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/scripts/cros_mark_as_stable.py", line 303, in main
    _WorkOnPush(options, overlay_tracking_branch, git_project_overlays)
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/scripts/cros_mark_as_stable.py", line 321, in _WorkOnPush
    parallel.RunTasksInProcessPool(_PushOverlays, inputs)
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 809, in RunTasksInProcessPool
    queue.put((idx, input_args))
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 750, in BackgroundTaskRunner
    queue.put(_AllTasksComplete())
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 750, in BackgroundTaskRunner
    queue.put(_AllTasksComplete())
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 561, in ParallelTasks
    raise BackgroundFailure(exc_infos=errors)
chromite.lib.parallel.BackgroundFailure: <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: 128; command: git rebase refs/remotes/cros/release-R71-11151.B
error: Could not open metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.49_rc-r1: No such file or directory
error: Failed to merge in the changes.

First, rewinding head to replay your work on top of it...
Applying: Marking 9999 ebuild for sys-kernel/linux-firmware as stable.
Applying: regen cache
Using index info to reconstruct a base tree...
A	metadata/md5-cache/chromeos-base/chromeos-chrome-71.0.3578.48_rc-r1
M	metadata/md5-cache/chromeos-base/chromeos-chrome-9999
A	metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.48_rc-r1
M	metadata/md5-cache/sys-kernel/chromeos-kernel-3_14-3.14-r2074
M	metadata/md5-cache/sys-kernel/chromeos-kernel-3_14-9999
M	metadata/md5-cache/sys-kernel/chromeos-kernel-3_18-3.18-r2222
M	metadata/md5-cache/sys-kernel/chromeos-kernel-3_18-9999
M	metadata/md5-cache/sys-kernel/chromeos-kernel-3_8-3.8.11-r749
M	metadata/md5-cache/sys-kernel/chromeos-kernel-3_8-9999
M	metadata/md5-cache/sys-kernel/chromeos-kernel-4_4-4.4.159-r1704
M	metadata/md5-cache/sys-kernel/chromeos-kernel-4_4-9999
Falling back to patching base and 3-way merge...
Auto-merging metadata/md5-cache/sys-kernel/linux-firmware-0.0.1-r208
CONFLICT (rename/rename): Rename "metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.48_rc-r1"->"metadata/md5-cache/chromeos-base/chromium-source-71.0.3575.1_rc-r1" in branch "HEAD" rename "metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.48_rc-r1"->"metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.49_rc-r1" in "regen cache"
Patch failed at 0002 regen cache
Use 'git am --show-current-patch' to see the failed patch

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".


cmd=['git', 'rebase', u'refs/remotes/cros/release-R71-11151.B'], cwd=/b/swarming/wGfv7iv/ir/cache/cbuild/repository/src/third_party/chromiumos-overlay
Traceback (most recent call last):
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 602, in TaskRunner
    task(*x, **task_kwargs)
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/parallel.py", line 800, in <lambda>
    fn = lambda idx, task_args: out_queue.put((idx, task(*task_args)))
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/scripts/cros_mark_as_stable.py", line 340, in _PushOverlays
    cwd=overlay, staging_branch=options.staging_branch)
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/scripts/cros_mark_as_stable.py", line 130, in PushChange
    git.SyncPushBranch(cwd, remote_ref.remote, remote_ref.ref)
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/git.py", line 1323, in SyncPushBranch
    RunGit(git_repo, [subcommand, target], **kwargs)
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/git.py", line 785, in RunGit
    return cros_build_lib.RunCommand(['git'] + cmd, **kwargs)
  File "/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/lib/cros_build_lib.py", line 647, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 128; command: git rebase refs/remotes/cros/release-R71-11151.B
error: Could not open metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.49_rc-r1: No such file or directory
error: Failed to merge in the changes.

First, rewinding head to replay your work on top of it...
Applying: Marking 9999 ebuild for sys-kernel/linux-firmware as stable.
Applying: regen cache
Using index info to reconstruct a base tree...
A	metadata/md5-cache/chromeos-base/chromeos-chrome-71.0.3578.48_rc-r1
M	metadata/md5-cache/chromeos-base/chromeos-chrome-9999
A	metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.48_rc-r1
M	metadata/md5-cache/sys-kernel/chromeos-kernel-3_14-3.14-r2074
M	metadata/md5-cache/sys-kernel/chromeos-kernel-3_14-9999
M	metadata/md5-cache/sys-kernel/chromeos-kernel-3_18-3.18-r2222
M	metadata/md5-cache/sys-kernel/chromeos-kernel-3_18-9999
M	metadata/md5-cache/sys-kernel/chromeos-kernel-3_8-3.8.11-r749
M	metadata/md5-cache/sys-kernel/chromeos-kernel-3_8-9999
M	metadata/md5-cache/sys-kernel/chromeos-kernel-4_4-4.4.159-r1704
M	metadata/md5-cache/sys-kernel/chromeos-kernel-4_4-9999
Falling back to patching base and 3-way merge...
Auto-merging metadata/md5-cache/sys-kernel/linux-firmware-0.0.1-r208
CONFLICT (rename/rename): Rename "metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.48_rc-r1"->"metadata/md5-cache/chromeos-base/chromium-source-71.0.3575.1_rc-r1" in branch "HEAD" rename "metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.48_rc-r1"->"metadata/md5-cache/chromeos-base/chromium-source-71.0.3578.49_rc-r1" in "regen cache"
Patch failed at 0002 regen cache
Use 'git am --show-current-patch' to see the failed patch

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".


cmd=['git', 'rebase', u'refs/remotes/cros/release-R71-11151.B'], cwd=/b/swarming/wGfv7iv/ir/cache/cbuild/repository/src/third_party/chromiumos-overlay

21:19:48: ERROR: 
return code: 1; command: /b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/bin/cros_mark_as_stable push --buildroot /b/swarming/wGfv7iv/ir/cache/cbuild/repository --overlay-type both
cmd=['/b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/bin/cros_mark_as_stable', 'push', '--buildroot', '/b/swarming/wGfv7iv/ir/cache/cbuild/repository', '--overlay-type', u'both'], cwd=/b/swarming/wGfv7iv/ir/cache/cbuild/repository

21:19:48: ERROR: /b/swarming/wGfv7iv/ir/cache/cbuild/repository/chromite/bin/cros_mark_as_stable failed (code=1)
...

This causes unexpected missing Chrome uprevs on the branch, for instance last night it failed and we now are a day behind in the release for a critical device.
 
Components: -Infra>Client>ChromeOS Infra>Client>ChromeOS>CI
Labels: -Pri-1 Pri-2

Sign in to add a comment