New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 627124 link

Starred by 2 users

Issue metadata

Status: Verified
Owner:
Last visit > 30 days ago
Closed: Jul 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

M52 pfq failing in SyncChrome

Project Member Reported by gkihumba@chromium.org, Jul 11 2016

Issue description

Looks like a timeout issue:

@@@BUILD_STEP@MasterSlaveSyncCompletion@@@
************************************************************
** Start Stage MasterSlaveSyncCompletion - Sun, 10 Jul 2016 23:37:02 -0700 (PDT)
** 
** Stage that records whether we passed or failed to build/test manifest.
************************************************************
23:37:02: INFO: Recording status fail for ['lumpy-pre-flight-branch']
23:37:02: INFO: RunCommand: /b/cbuild/shared_internal/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- /tmp/cbuildbot-tmpKGHctP/cbuildbot-tmp7WJKRS/tmpNPdWHw gs://chromeos-manifest-versions/builder-status/8350.52.0-rc4/lumpy-pre-flight-branch
23:37:03: INFO: Running cidb query on pid 11949, repr(query) starts with 'SELECT deadline >= NOW(), TIMEDIFF(deadline, NOW()) from buildTable where id = 943672'
23:37:03: INFO: Running cidb query on pid 11949, repr(query) starts with <sqlalchemy.sql.expression.Select at 0x6224150; Select object>
23:37:03: ERROR: No status found for build config lumpy-pre-flight-branch.
23:37:03: INFO: Still waiting for the following builds to complete: ['lumpy-pre-flight-branch']
23:37:03: INFO: 0:00:00 until timeout...
23:37:03: INFO: RunCommand: /b/cbuild/shared_internal/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/8350.52.0-rc4/lumpy-pre-flight-branch
23:37:04: ERROR: Not all builds finished before timeout (0 minutes) reached.

@@@STEP_LINK@lumpy-pre-flight-branch: The SyncChrome stage failed: <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: 1; command: /b/cbuild/shared_internal/chromite/bin/sync_chrome --reset --internal --tag 52.0.2743.73 /b/cbuild/shared_internal/.cach@https://uberchromegw.corp.google.com/i/chromiumos.release/builders/lumpy-pre-flight-branch%20release-R52-8350.B/builds/143@@@

@@@STEP_WARNINGS@@@
23:37:04: WARNING: The following builders failed with this manifest:
lumpy-pre-flight-branch
Please check the logs of the failing builders for details.

@@@STEP_FAILURE@@@
23:37:04: ERROR: 
23:37:04: INFO: Running cidb query on pid 11949, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x622ef50>
23:37:04: INFO: Running cidb query on pid 11949, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x6224710>
************************************************************
** Finished Stage MasterSlaveSyncCompletion - Sun, 10 Jul 2016 23:37:04 -0700 (PDT)
************************************************************
23:37:04: INFO: Running cidb query on pid 11949, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x62245d0>
23:37:04: INFO: Running cidb query on pid 11949, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x6224b90>
23:37:05: INFO: Running cidb query on pid 11949, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x6224c90>

Build link:
https://uberchromegw.corp.google.com/i/chromeos_release/builders/lumpy-pre-flight-branch%20release-R52-8350.B/builds/143

Logs link:
https://uberchromegw.corp.google.com/i/chromeos_release/builders/lumpy-pre-flight-branch%20release-R52-8350.B/builds/143/steps/MasterSlaveSyncCompletion/logs/stdio

 
Cc: akes...@chromium.org dgarr...@chromium.org vapier@chromium.org h...@chromium.org
Lumpy-pre-flight is failing on SyncChrome stage in a row. Mike, can you take a look?
Summary: M52 pfq failing in SyncChrome (was: M52 pfq failing)
This is failing in SyncChrome. Looks like some kind of git error (perhaps local git corruption).

"
error: refs/remotes/origin/test_tags1 does not point to a valid object!
error: refs/tags/__experimental.1 does not point to a valid object!
"
Find these errors too:
________ running 'git -c core.deltaBaseCacheLimit=2g fetch origin --verbose' in '/b/cbuild/shared_internal/.cache/distfiles/target/chrome-src-internal/src/v8'
error: refs/remotes/origin/test_tags1 does not point to a valid object!
error: refs/tags/__experimental.1 does not point to a valid object!
error: refs/remotes/origin/test_tags1 does not point to a valid object!
error: refs/tags/__experimental.1 does not point to a valid object!
error: refs/remotes/origin/test_tags1 does not point to a valid object!
error: refs/tags/__experimental.1 does not point to a valid object!

How to fix the local git corruption. Do we need to log into the server, and fix the git manually?
Yes, that is one option (by deleting the broken repos). A more aggressive option is to do force build with clobber. The drawback is we may run out of gerrit quota if not careful.

Don should weigh in on this question.
Based on the team discussion with Don, I will do a force build with clobber.
Clobbering a single builder is always safe. Its when you wipe out 10 or more that things get to be problematic.

As a note, newer branches will auto-clobber Chrome if it fails to sync.
Looks like chrome synched ok in current build...thanks!
Status: Fixed (was: Assigned)
Labels: VerifyIn-54
Status: Verified (was: Fixed)
bulk verified

Sign in to add a comment