New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 614793 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: May 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

R52 Physical builders failing to sync Chrome.

Project Member Reported by dgarr...@chromium.org, May 25 2016

Issue description

Most (all?) physical builders on R52 are failing to sync Chrome code.

We made a change recently in TOT in how we manage Chrome syncs, and these builders were syncing from an older branch. Perhaps there is some problem with the transition from the older branch to TOT.

 
Cc: steve...@chromium.org
Aviv did NOT land his changes around this, so he's not to blame.

An example failure (I hope this isn't normal):

https://uberchromegw.corp.google.com/i/chromeos_release/builders/peach_pit-release%20release-R52-8350.B/builds/2/steps/SyncChrome/logs/stdio

[0:02:42] Fetching +refs/tags/*:refs/tags/*
[0:02:42] running "git fetch -v --progress origin +refs/tags/*:refs/tags/*" in "/b/git-cache/chromium.googlesource.com-external-github.com-catapult--project-catapult"
[0:02:42] Fetching +refs/branch-heads/*:refs/branch-heads/*
[0:02:42] running "git fetch -v --progress origin +refs/branch-heads/*:refs/branch-heads/*" in "/b/git-cache/chromium.googlesource.com-external-github.com-catapult--project-catapult"

________ running 'git -c core.deltaBaseCacheLimit=2g fetch origin --verbose' in '/b/cbuild/shared_internal/.cache/distfiles/target/chrome-src-internal/src/third_party/catapult'
[0:02:42] error: refs/remotes/origin/mac_symbols_fix does not point to a valid object!
[0:02:42] error: refs/remotes/origin/mappersv22 does not point to a valid object!
[0:02:42] error: refs/remotes/origin/mac_symbols_fix does not point to a valid object!
[0:02:42] error: refs/remotes/origin/mappersv22 does not point to a valid object!
[0:02:42] error: refs/remotes/origin/mac_symbols_fix does not point to a valid object!
[0:02:42] error: refs/remotes/origin/mappersv22 does not point to a valid object!
[0:02:42] From /b/git-cache/chromium.googlesource.com-external-github.com-catapult--project-catapult
[0:02:42]  = [up to date]      infra/config -> origin/infra/config
[0:02:42]  = [up to date]      mappersv2  -> origin/mappersv2
[0:02:42]  = [up to date]      mappersv2_merge -> origin/mappersv2_merge
[0:02:42] error: refs/remotes/origin/mac_symbols_fix does not point to a valid object!
[0:02:42] error: refs/remotes/origin/mappersv22 does not point to a valid object!
[0:02:42]    0728abf..d01ff3b  master     -> origin/master
[0:02:42]  = [up to date]      md-pages   -> origin/md-pages
[0:02:42]  = [up to date]      polymer10  -> origin/polymer10
[0:02:42]  = [up to date]      polymer10-dashboard -> origin/polymer10-dashboard
[0:02:42]    a0dc5f8..b948445  polymer10-migration -> origin/polymer10-migration
[0:02:42] error: refs/remotes/origin/mac_symbols_fix does not point to a valid object!
[0:02:42] error: refs/remotes/origin/mappersv22 does not point to a valid object!

Steven, do you have any feedback here?
Cc: levarum@chromium.org
None of that makes any sense to me.

We are investigating issue 591814, but have not landed anything yet.



Cc: d...@chromium.org bhthompson@chromium.org
We can probably fix this by clobbering the affected builders, but I don't want to do that until after we finish getting GCE builders to sync successfully.
I don't think this have to do with issue 591814.

Comment#2 - looks like some stuck remote branches, that didn't exit in release branches. Maybe it's ok.

This one is more scaring:

Error: Command 'git merge-base HEAD origin' returned non-zero exit status 1 in /b/cbuild/shared_internal/.cache/distfiles/target/chrome-src-internal/src/third_party/libjingle/source/talk

Cc: josa...@chromium.org
Looks like local git corruption -- per discussion with dgarrett@ can we just clobber the builder and proceed?

Comment 9 by d...@chromium.org, May 27 2016

(If you're waiting on me, go for it).

Git corruption isn't unheard of during milestone transitions. The next "beta" inherits the previous "beta"'s checkouts, so not only do they jump pretty far ahead abruptly, but any change in how Chromite handles checkouts may hit a case suddenly dealing with a (from its POV) really old checkout from the previous milestone's Chromite.
Owner: dgarr...@chromium.org
Status: Started (was: Untriaged)
Starting the clean up, semi-manually.

PS: Not every builder was affected, only most of them.
All of the problematic builders have been clobbers. 

There are some in-progress builds that should be killed, preferably after they finish the SyncChrome stages.
Status: Fixed (was: Started)
They should all have been clobbered, and re-sync'd.
Closing... please feel free to reopen if its not fixed.
Status: Verified (was: Fixed)

Sign in to add a comment