Multiple builders for factory branch (ex: octopus and kalista) got failed in Sync_Chrome stage. |
||||
Issue description[ADD relevant info inline] Octopus and Kalista factory builders failed to build since Jan 10 2019 Link to build or pfq page. https://uberchromegw.corp.google.com/i/chromeos.branch/builders/octopus%20factory%20factory-octopus-11512.B https://uberchromegw.corp.google.com/i/chromeos.branch/builders/kalista%20factory%20factory-nami-10715.B build # for that buildbot. https://uberchromegw.corp.google.com/i/chromeos.branch/builders/octopus%20factory%20factory-octopus-11512.B/builds/1 https://uberchromegw.corp.google.com/i/chromeos.branch/builders/kalista%20factory%20factory-nami-10715.B/builds/33 Snippet of log that contains the failure. tools_internal (ERROR) ---------------------------------------- [0:00:12] Started. [0:00:13] running "git cat-file -e 7f3f19539cdd5bfc2ccdabdac38ef721d408ffce^{commit}" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps" [0:00:13] fatal: Not a valid object name 7f3f19539cdd5bfc2ccdabdac38ef721d408ffce^{commit} [0:00:13] /b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps has 2 .pack files, re-bootstrapping if >50 [0:00:13] running "git config gc.autodetach 0" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps" [0:00:13] running "git config gc.autopacklimit 0" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps" [0:00:13] running "git config core.deltaBaseCacheLimit 2g" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps" [0:00:13] running "git config remote.origin.url https://chrome-internal.googlesource.com/a/chrome/tools/build/internal.DEPS.git" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps" [0:00:13] running "git config --replace-all remote.origin.fetch +refs/heads/*:refs/heads/* \+refs/heads/\*:.*" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps" [0:00:13] Fetching +refs/heads/*:refs/heads/* [0:00:13] running "git fetch origin +refs/heads/*:refs/heads/*" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps" [0:00:14] From https://chrome-internal.googlesource.com/a/chrome/tools/build/internal.DEPS [0:00:14] * [new branch] chromium/3668 -> chromium/3668 [0:00:14] fetch +refs/heads/*:refs/heads/* took 0.0 minutes [0:00:14] Fetching +refs/tags/*:refs/tags/* [0:00:14] running "git fetch origin +refs/tags/*:refs/tags/*" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps" [0:00:14] fetch +refs/tags/*:refs/tags/* took 0.0 minutes [0:00:14] Fetching +refs/branch-heads/*:refs/branch-heads/* [0:00:14] running "git fetch origin +refs/branch-heads/*:refs/branch-heads/*" in "/b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps" [0:00:15] fetch +refs/branch-heads/*:refs/branch-heads/* took 0.0 minutes [0:00:15] From /b/git-cache/chrome--internal.googlesource.com-a-chrome-tools-build-internal.deps [0:00:15] * [new branch] chromium/3668 -> origin/chromium/3668 [0:00:15] _____ tools_internal : Attempting rebase onto 7f3f19539cdd5bfc2ccdabdac38ef721d408ffce... [0:00:15] [0:00:15] Rebase produced error output: fatal: Needed a single revision Does not point to a valid commit: 7f3f19539cdd5bfc2ccdabdac38ef721d408ffce ---------------------------------------- Error: 7> Unrecognized error, please merge or rebase manually. .... Error: Command 'git checkout --quiet 7f3f19539cdd5bfc2ccdabdac38ef721d408ffce' returned non-zero exit status 128 in /b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal/tools_internal fatal: reference is not a tree: 7f3f19539cdd5bfc2ccdabdac38ef721d408ffce Warnings: Conflicting directory /b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal/tools_internal deleted. Conflicting directory /b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal/src-internal deleted. sync_chrome: Unhandled exception: Traceback (most recent call last): File "/b/c/cbuild/repository/chromite/bin/sync_chrome", line 169, in <module> DoMain() File "/b/c/cbuild/repository/chromite/bin/sync_chrome", line 165, in DoMain commandline.ScriptWrapperMain(FindTarget) File "/b/c/cbuild/repository/chromite/lib/commandline.py", line 912, in ScriptWrapperMain ret = target(argv[1:]) File "/b/c/cbuild/repository/chromite/scripts/sync_chrome.py", line 76, in main SyncChrome(gclient_path, options) File "/b/c/cbuild/repository/chromite/scripts/sync_chrome.py", line 52, in SyncChrome ignore_locks=options.ignore_locks) File "/b/c/cbuild/repository/chromite/lib/gclient.py", line 262, in Sync return cros_build_lib.RunCommand(cmd, cwd=cwd, **run_args) File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 646, in RunCommand raise RunCommandError(msg, cmd_result) chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: /b/c/cbuild/repository/chromium/tools/depot_tools/gclient sync --with_branch_heads --with_tags --reset --force --delete_unversioned_trees --nohooks --verbose --ignore_locks cmd=['/b/c/cbuild/repository/chromium/tools/depot_tools/gclient', 'sync', '--with_branch_heads', '--with_tags', '--reset', '--force', '--delete_unversioned_trees', '--nohooks', '--verbose', '--ignore_locks'], cwd=/b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal [1;33m21:23:58: WARNING: Command failed with retriable error. return code: 1; command: /b/c/cbuild/repository/chromite/bin/sync_chrome --reset --ignore_locks --internal --tag 73.0.3654.0 /b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal cmd=['/b/c/cbuild/repository/chromite/bin/sync_chrome', '--reset', '--ignore_locks', '--internal', '--tag', '73.0.3654.0', '/b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal'], cwd=/b/c/cbuild/repository[0m 21:23:58: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/sync_chrome --reset --ignore_locks --internal --tag 73.0.3654.0 /b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal in /b/c/cbuild/repository 21:23:58: INFO: RunCommand: /b/c/cbuild/repository/chromium/tools/depot_tools/gclient revert --nohooks in /b/c/cbuild/repository/.cache/distfiles/target-factory-octopus-11512.B/chrome-src-internal
,
Jan 11
,
Jan 11
The failures in the builders seem odd. They are reproducible but the same errors were not showing up prior to the builders having to be rebuilt. The manual intervention to get them to the point of failing is also questionable. The best approach now is to abandon our physical machines, since we no longer need the contiguous IP block, and move our machines back to GCE. This will ensure our puppet config is the same and reduce some outliers. -- Mike
,
Jan 11
What do you mean by "The manual intervention to get them to the point of failing"? Any idea on an ETA for factory builders on GCE? Octopus plans on having factory builds working shortly.
,
Jan 11
Discussed offline. The plan will be to move the factory builders on Monday. The actual move is rather quick once we get the machines drained and available. I'll update the bug with additional status once the move process begins. -- Mike
,
Jan 14
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/97c9b9494fc7a80fad83365b6b92177eb2d3a0bc commit 97c9b9494fc7a80fad83365b6b92177eb2d3a0bc Author: Mike Nichols <mikenichols@chromium.org> Date: Mon Jan 14 15:51:34 2019
,
Jan 14
The following revision refers to this bug: https://chrome-internal.googlesource.com/infra/infra_internal/+/7b84b4509bb2d7691251398f34396251f55121bb commit 7b84b4509bb2d7691251398f34396251f55121bb Author: Mike Nichols <mikenichols@chromium.org> Date: Mon Jan 14 17:42:13 2019
,
Jan 14
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build/+/5d08e816344458f065a5ec38154f9778e65ebe6b commit 5d08e816344458f065a5ec38154f9778e65ebe6b Author: Mike Nichols <mikenichols@chromium.org> Date: Mon Jan 14 18:11:18 2019
,
Jan 14
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build/+/3111ed7bd2fe42bbe174e36ffc853da14145c74e commit 3111ed7bd2fe42bbe174e36ffc853da14145c74e Author: Mike Nichols <mikenichols@chromium.org> Date: Mon Jan 14 18:45:21 2019
,
Jan 14
Hi Mike, Thanks for the support and I see there is a force build for octopus factory branch. So I will also track the status.
,
Jan 15
Hi Marco, I have finished the migration (took a bit longer than expected as buildbot did not support our newer kernel images) and kicked off two builds for Octopus: Factory pre-flight (finished successfully): https://uberchromegw.corp.google.com/i/chromeos.branch/builders/octopus%20pre-flight%20factory-octopus-11512.B/builds/3 Factory octopus (still running): https://uberchromegw.corp.google.com/i/chromeos.branch/builders/octopus%20factory%20factory-octopus-11512.B/builds/2 I'll keep an eye on them and we'll adjust accordingly. -- Mike
,
Jan 15
,
Jan 15
Hi Mike, Thanks very much for supporting this issue and octopus-factory build is passed now. But there is a question around image version number - I can make sure I bumped the CHROMEOS_BRANCH from 0 to 1 [1] in factory-octopus-11512.B so I expect the generated image version would be like 11512.X.0 . But it turns out that the version is 11512.6.3 . In the stage of ManifestVersionedSync, it shows logs about [2]. May I know should I file an new issue for it and who can help or this issue can also try to handle it? Thanks. On the other hand, there is really a stabilize-atlas-11512.6.B which is the same 11512 version. But I thought it should increment version by 11512.6.X and not impact 11512.X.0 from octopus factory branch? [1] https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/df5ddcd2445a8d728e9c607b069dbccb3d1c3836%5E%21/#F0 [2] RELEASETAG: 11512.6.3 BUILDROOT: /b/c/cbuild/repository TRACKING BRANCH: factory-octopus-11512.B NEXT MANIFEST: /b/c/cbuild/repository/manifest-versions-internal/buildspecs/73/11512.6.3.xml
,
Jan 15
Hi Marco, Please open another bug related to the version number, especially if you are seeing the same behavior in multiple builders. The factory builders appear to be working again therefore investigating the build behavior should be a separate initiative. Thanks, Mike
,
Jan 15
,
Jan 15
Please cc me on the new bug, I have a little insight. |
||||
►
Sign in to add a comment |
||||
Comment 1 by marcochen@chromium.org
, Jan 11