New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 920914 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jan 15
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

Multiple builders for factory branch (ex: octopus and kalista) got failed in Sync_Chrome stage.

Project Member Reported by marcochen@chromium.org, Jan 11

Issue description

[ADD relevant info inline]
Octopus and Kalista factory builders failed to build since Jan 10 2019

Link to build or pfq page.
https://uberchromegw.corp.google.com/i/chromeos.branch/builders/octopus%20factory%20factory-octopus-11512.B
https://uberchromegw.corp.google.com/i/chromeos.branch/builders/kalista%20factory%20factory-nami-10715.B

build # for that buildbot.
https://uberchromegw.corp.google.com/i/chromeos.branch/builders/octopus%20factory%20factory-octopus-11512.B/builds/1
https://uberchromegw.corp.google.com/i/chromeos.branch/builders/kalista%20factory%20factory-nami-10715.B/builds/33

Snippet of log that contains the failure.
tools_internal (ERROR)
----------------------------------------
[0:00:12] Started.
[0:00:13] running "git cat-file -e 7f3f19539cdd5bfc2ccdabdac38ef721d408ffce^{commit}" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps"
[0:00:13] fatal: Not a valid object name 7f3f19539cdd5bfc2ccdabdac38ef721d408ffce^{commit}
[0:00:13] /b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps has 2 .pack files, re-bootstrapping if >50
[0:00:13] running "git config gc.autodetach 0" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps"
[0:00:13] running "git config gc.autopacklimit 0" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps"
[0:00:13] running "git config core.deltaBaseCacheLimit 2g" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps"
[0:00:13] running "git config remote.origin.url https://chrome-internal.googlesource.com/a/chrome/tools/build/internal.DEPS.git" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps"
[0:00:13] running "git config --replace-all remote.origin.fetch +refs/heads/*:refs/heads/* \+refs/heads/\*:.*" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps"
[0:00:13] Fetching +refs/heads/*:refs/heads/*
[0:00:13] running "git fetch origin +refs/heads/*:refs/heads/*" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps"
[0:00:14] From https://chrome-internal.googlesource.com/a/chrome/tools/build/internal.DEPS
[0:00:14]  * [new branch]      chromium/3668 -> chromium/3668
[0:00:14] fetch +refs/heads/*:refs/heads/* took 0.0 minutes
[0:00:14] Fetching +refs/tags/*:refs/tags/*
[0:00:14] running "git fetch origin +refs/tags/*:refs/tags/*" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps"
[0:00:14] fetch +refs/tags/*:refs/tags/* took 0.0 minutes
[0:00:14] Fetching +refs/branch-heads/*:refs/branch-heads/*
[0:00:14] running "git fetch origin +refs/branch-heads/*:refs/branch-heads/*" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps"
[0:00:15] fetch +refs/branch-heads/*:refs/branch-heads/* took 0.0 minutes
[0:00:15] From /b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps
[0:00:15]  * [new branch]      chromium/3668 -> origin/chromium/3668
[0:00:15] _____ tools_internal : Attempting rebase onto 7f3f19539cdd5bfc2ccdabdac38ef721d408ffce...
[0:00:15] 
[0:00:15] Rebase produced error output:
fatal: Needed a single revision
Does not point to a valid commit: 7f3f19539cdd5bfc2ccdabdac38ef721d408ffce
----------------------------------------
Error: 7> Unrecognized error, please merge or rebase manually.

....

Error: Command 'git checkout --quiet 7f3f19539cdd5bfc2ccdabdac38ef721d408ffce' returned non-zero exit status 128 in /b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal/tools_internal
fatal: reference is not a tree: 7f3f19539cdd5bfc2ccdabdac38ef721d408ffce



Warnings:
Conflicting directory /b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal/tools_internal deleted.
Conflicting directory /b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal/src-internal deleted.
sync_chrome: Unhandled exception:
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/bin/sync_chrome", line 169, in <module>
    DoMain()
  File "/b/c/cbuild/repository/chromite/bin/sync_chrome", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/c/cbuild/repository/chromite/lib/commandline.py", line 912, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/c/cbuild/repository/chromite/scripts/sync_chrome.py", line 76, in main
    SyncChrome(gclient_path, options)
  File "/b/c/cbuild/repository/chromite/scripts/sync_chrome.py", line 52, in SyncChrome
    ignore_locks=options.ignore_locks)
  File "/b/c/cbuild/repository/chromite/lib/gclient.py", line 262, in Sync
    return cros_build_lib.RunCommand(cmd, cwd=cwd, **run_args)
  File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 646, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: /b/c/cbuild/repository/chromium/tools/depot_tools/gclient sync --with_branch_heads --with_tags --reset --force --delete_unversioned_trees --nohooks --verbose --ignore_locks
cmd=['/b/c/cbuild/repository/chromium/tools/depot_tools/gclient', 'sync', '--with_branch_heads', '--with_tags', '--reset', '--force', '--delete_unversioned_trees', '--nohooks', '--verbose', '--ignore_locks'], cwd=/b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal
21:23:58: WARNING: Command failed with retriable error.
return code: 1; command: /b/c/cbuild/repository/chromite/bin/sync_chrome --reset --ignore_locks --internal --tag 73.0.3654.0 /b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal
cmd=['/b/c/cbuild/repository/chromite/bin/sync_chrome', '--reset', '--ignore_locks', '--internal', '--tag', '73.0.3654.0', '/b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal'], cwd=/b/c/cbuild/repository
21:23:58: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/sync_chrome --reset --ignore_locks --internal --tag 73.0.3654.0 /b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal in /b/c/cbuild/repository
21:23:58: INFO: RunCommand: /b/c/cbuild/repository/chromium/tools/depot_tools/gclient revert --nohooks in /b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal
 
Cc: chuntsen@chromium.org
Owner: mikenichols@chromium.org
Status: Assigned (was: Untriaged)
The failures in the builders seem odd.  They are reproducible but the same errors were not showing up prior to the builders having to be rebuilt.  The manual intervention to get them to the point of failing is also questionable.  

The best approach now is to abandon our physical machines, since we no longer need the contiguous IP block, and move our machines back to GCE.  This will ensure our puppet config is the same and reduce some outliers.  

-- Mike
What do you mean by "The manual intervention to get them to the point of failing"?

Any idea on an ETA for factory builders on GCE?
Octopus plans on having factory builds working shortly.
Discussed offline.  The plan will be to move the factory builders on Monday.  The actual move is rather quick once we get the machines drained and available.  

I'll update the bug with additional status once the move process begins.  

-- Mike
Project Member

Comment 6 by bugdroid1@chromium.org, Jan 14

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/97c9b9494fc7a80fad83365b6b92177eb2d3a0bc

commit 97c9b9494fc7a80fad83365b6b92177eb2d3a0bc
Author: Mike Nichols <mikenichols@chromium.org>
Date: Mon Jan 14 15:51:34 2019

Project Member

Comment 7 by bugdroid1@chromium.org, Jan 14

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/7b84b4509bb2d7691251398f34396251f55121bb

commit 7b84b4509bb2d7691251398f34396251f55121bb
Author: Mike Nichols <mikenichols@chromium.org>
Date: Mon Jan 14 17:42:13 2019

Project Member

Comment 8 by bugdroid1@chromium.org, Jan 14

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build/+/5d08e816344458f065a5ec38154f9778e65ebe6b

commit 5d08e816344458f065a5ec38154f9778e65ebe6b
Author: Mike Nichols <mikenichols@chromium.org>
Date: Mon Jan 14 18:11:18 2019

Project Member

Comment 9 by bugdroid1@chromium.org, Jan 14

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build/+/3111ed7bd2fe42bbe174e36ffc853da14145c74e

commit 3111ed7bd2fe42bbe174e36ffc853da14145c74e
Author: Mike Nichols <mikenichols@chromium.org>
Date: Mon Jan 14 18:45:21 2019

Hi Mike,

Thanks for the support and I see there is a force build for octopus factory branch. So I will also track the status.
Hi Marco,

I have finished the migration (took a bit longer than expected as buildbot did not support our newer kernel images) and kicked off two builds for Octopus:

Factory pre-flight (finished successfully): https://uberchromegw.corp.google.com/i/chromeos.branch/builders/octopus%20pre-flight%20factory-octopus-11512.B/builds/3

Factory octopus (still running):
https://uberchromegw.corp.google.com/i/chromeos.branch/builders/octopus%20factory%20factory-octopus-11512.B/builds/2

I'll keep an eye on them and we'll adjust accordingly.

-- Mike
Status: Started (was: Assigned)
Hi Mike,

Thanks very much for supporting this issue and octopus-factory build is passed now. But there is a question around image version number -

  I can make sure I bumped the CHROMEOS_BRANCH from 0 to 1 [1] in factory-octopus-11512.B so I expect the generated image version would be like 11512.X.0 . But it turns out that the version is 11512.6.3 . 

  In the stage of ManifestVersionedSync, it shows logs about [2]. May I know should I file an new issue for it and who can help or this issue can also try to handle it? Thanks.

  On the other hand, there is really a stabilize-atlas-11512.6.B which is the same 11512 version. But I thought it should increment version by 11512.6.X and not impact 11512.X.0 from octopus factory branch?


[1] https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/df5ddcd2445a8d728e9c607b069dbccb3d1c3836%5E%21/#F0
[2]
RELEASETAG: 11512.6.3

BUILDROOT: /b/c/cbuild/repository
TRACKING BRANCH: factory-octopus-11512.B
NEXT MANIFEST: /b/c/cbuild/repository/manifest-versions-internal/buildspecs/73/11512.6.3.xml
Hi Marco,

Please open another bug related to the version number, especially if you are seeing the same behavior in multiple builders.  The factory builders appear to be working again therefore investigating the build behavior should be a separate initiative. 

Thanks,
Mike
Status: Fixed (was: Started)
Please cc me on the new bug, I have a little insight.

Sign in to add a comment