New issue
Advanced search Search tips

Issue 889622 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Oct 15
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocking:
issue 889286



Sign in to add a comment

firmwarebranch: WorkspacePublishBuildspec fails if manifest-versions updated.

Project Member Reported by dgarr...@chromium.org, Sep 26

Issue description

Example build:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8934287332633320208


Publishing the buildspec fails if manifest-versions is updated:


15:08:15: INFO: Updating version file to: 10775.24.0
15:08:15: INFO: Pushing to branch (_RemoteRef(remote=u'cros', ref=u'refs/heads/firmware-nami-10775.B', project_name=u'chromiumos/overlays/chromiumos-overlay')) with message: Incremented to version: 10775.24.0 
15:08:20: INFO: Creating buildspec: buildspecs/69/10775.24.0.xml
15:08:24: INFO: Creating buildspec: buildspecs/69/10775.24.0.xml as /b/swarming/w/ir/cache/cbuild/repository/manifest-versions-internal/buildspecs/69/10775.24.0.xml
15:09:03: ERROR: Traceback (most recent call last):
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 703, in Run
    self.PerformStage()
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/stages/workspace_stages.py", line 198, in PerformStage
    dryrun=self._run.options.debug)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/manifest_version.py", line 495, in GenerateAndPublishOfficialBuildSpec
    dryrun)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/manifest_version.py", line 443, in PopulateAndPublishBuildSpec
    manifest_versions_int, rel_build_spec, manifest, dryrun)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/cbuildbot/manifest_version.py", line 417, in _CommitAndPush
    git.PushBranch(PUSH_BRANCH, manifest_repo, dryrun=dryrun)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/git.py", line 1371, in PushBranch
    GitPush(git_repo, branch, remote_ref, skip=dryrun)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/git.py", line 1272, in GitPush
    **kwargs)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/git.py", line 785, in RunGit
    return cros_build_lib.RunCommand(['git'] + cmd, **kwargs)
  File "/b/swarming/w/ir/cache/cbuild/repository/chromite/lib/cros_build_lib.py", line 647, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: git push origin temp_auto_checkin_branch:refs/heads/master
To https://chrome-internal.googlesource.com/chromeos/manifest-versions
 ! [rejected]                temp_auto_checkin_branch -> master (fetch first)
error: failed to push some refs to 'https://chrome-internal.googlesource.com/chromeos/manifest-versions'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/swarming/w/ir/cache/cbuild/repository/manifest-versions-internal

15:09:03: INFO: Translating result return code: 1; command: git push origin temp_auto_checkin_branch:refs/heads/master
To https://chrome-internal.googlesource.com/chromeos/manifest-versions
 ! [rejected]                temp_auto_checkin_branch -> master (fetch first)
 
Cc: la...@chromium.org
This will cause these builds to fail in flaky ways until it's addressed. It's possible to resolve this with retry loops, or by allowing the push to rebase to TOT.

I'm not yet sure how common this will be, but it needs to be addressed.
Status: Started (was: Untriaged)
Blocking: 889286
Labels: -Pri-3 Pri-2
Project Member

Comment 5 by bugdroid1@chromium.org, Oct 12

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/0758ae8cb95e3dc956e0d8046e85dc0ea6cf34db

commit 0758ae8cb95e3dc956e0d8046e85dc0ea6cf34db
Author: Don Garrett <dgarrett@google.com>
Date: Fri Oct 12 22:18:27 2018

WorkspacePublishBuildspecStage: Add retries.

Revamp the manifest_version CreateAndPublish functions to retry
manifest version commits, in case it was updated remotely. This change
moves the sync operation for the manifest version checkouts down into
the commit code, so that retries will pull in fresh versions.

BUG= chromium:889622 
TEST=run_tests

Change-Id: I9707eed9f64f2448ef9a3bfeb6770518f208e7df
Reviewed-on: https://chromium-review.googlesource.com/1273965
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Alec Thilenius <athilenius@google.com>

[modify] https://crrev.com/0758ae8cb95e3dc956e0d8046e85dc0ea6cf34db/cbuildbot/stages/workspace_stages.py
[modify] https://crrev.com/0758ae8cb95e3dc956e0d8046e85dc0ea6cf34db/cbuildbot/manifest_version.py
[modify] https://crrev.com/0758ae8cb95e3dc956e0d8046e85dc0ea6cf34db/cbuildbot/manifest_version_unittest.py

Status: Fixed (was: Started)
Believed fixed.

Sign in to add a comment