Issue 624464 link

Starred by 1 user

Issue metadata

Status:	Archived
Owner:	█ jrbarnette@chromium.org Last visit > 30 days ago
Closed:	Jun 2016
Cc:	█ gkihumba@chromium.org steve...@chromium.org
EstimatedDays:	----
NextAction:	----
OS:	Chrome
Pri:	1
Type:	Bug
M-52 rcissue VerifyIn-55 VerifyIn-56 VerifyIn-57 VerifyIn-58 VerifyIn-59 VerifyIn-60 VerifyIn-61

Sync Chrome stage fails due to timeout

Project Member Reported by gkihumba@chromium.org, Jun 29 2016

Issue description

M52 8350.38.0, 52.0.2743.49

07:02:28: INFO: Recording status fail for ['lumpy-pre-flight-branch']
07:02:28: INFO: RunCommand: /b/cbuild/shared_internal/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cp -v -- /tmp/cbuildbot-tmpsj3HIi/cbuildbot-tmpLwZozH/tmpSfmoo_ gs://chromeos-manifest-versions/builder-status/8350.46.0-rc1/lumpy-pre-flight-branch
07:02:29: INFO: Running cidb query on pid 22091, repr(query) starts with 'SELECT deadline >= NOW(), TIMEDIFF(deadline, NOW()) from buildTable where id = 924779'
07:02:29: INFO: Running cidb query on pid 22091, repr(query) starts with <sqlalchemy.sql.expression.Select at 0x44de450; Select object>
07:02:30: ERROR: No status found for build config lumpy-pre-flight-branch.
07:02:30: INFO: Still waiting for the following builds to complete: ['lumpy-pre-flight-branch']
07:02:30: INFO: 0:02:22.895881 until timeout...
07:03:30: INFO: Running cidb query on pid 22091, repr(query) starts with <sqlalchemy.sql.expression.Select at 0x44de610; Select object>
07:03:34: INFO: cidb query succeeded after 1 retries
07:03:34: ERROR: No status found for build config lumpy-pre-flight-branch.
07:03:34: INFO: Still waiting for the following builds to complete: ['lumpy-pre-flight-branch']
07:03:34: INFO: 0:01:17.970476 until timeout...
07:04:35: INFO: Running cidb query on pid 22091, repr(query) starts with <sqlalchemy.sql.expression.Select at 0x4066490; Select object>
07:04:39: INFO: cidb query succeeded after 1 retries
07:04:40: ERROR: No status found for build config lumpy-pre-flight-branch.
07:04:40: INFO: Still waiting for the following builds to complete: ['lumpy-pre-flight-branch']
07:04:40: INFO: 0:00:12.935405 until timeout...
07:04:40: INFO: RunCommand: /b/cbuild/shared_internal/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/8350.46.0-rc1/lumpy-pre-flight-branch
07:04:40: ERROR: Not all builds finished before timeout (2 minutes) reached.

@@@STEP_LINK@lumpy-pre-flight-branch: The SyncChrome stage failed: <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: 1; command: /b/cbuild/shared_internal/chromite/bin/sync_chrome --reset --internal --tag 52.0.2743.58 /b/cbuild/shared_internal/.cach@https://uberchromegw.corp.google.com/i/chromiumos.release/builders/lumpy-pre-flight-branch%20release-R52-8350.B/builds/117@@@

@@@STEP_WARNINGS@@@
07:04:40: WARNING: The following builders failed with this manifest:
lumpy-pre-flight-branch
Please check the logs of the failing builders for details.


Link to build:
https://uberchromegw.corp.google.com/i/chromeos_release/builders/lumpy-pre-flight-branch%20release-R52-8350.B/builds/117

Link to full report:
https://uberchromegw.corp.google.com/i/chromeos_release/builders/lumpy-pre-flight-branch%20release-R52-8350.B/builds/117/steps/MasterSlaveSyncCompletion/logs/stdio

Comment 1 by jrbarnette@chromium.org, Jun 30 2016

Cc: steve...@chromium.org

All right.  The worst of our crises are over...

Looking, I see that this failure has occurred 6 times,
starting with last night's 17:10 build.

I've half a mind to say this is just a variant on  bug 624177 .
If the builder is using an up-to-date gclient, it would need
the two changes we applied to chromite to be cherry-picked into
M52.

This problem likely would also affect M51 builders.

I'll ask around about my theory, to see if it holds any water.

Comment 2 by jrbarnette@chromium.org, Jun 30 2016

I've checked timing.  The trouble should have started after
this CL went in:
    https://codereview.chromium.org/2099153003

Looking at that change, it went in around 13:04 Pacific
time, yesterday.  The failing builder did have a successful
run starting at 14:00, but if there were a lag between the
Chrome CL committing and its visibility to the Chrome OS builder,
that's not unreasonable.  So, timing-wise, the theory holds up.

Comment 3 by steve...@chromium.org, Jun 30 2016

Yes, since the gclient change that broke us is part of depot_tools, anything using depot_tools (which iiuc auto-updates itself) would be affected.

We should merge the chromite changes to 52 and 51.

Project Member

Comment 4 by sheriffbot@chromium.org, Jun 30 2016

Labels: -M-52 M-53 MovedFrom-52

Moving this nonessential bug to the next milestone.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 5 by gkihumba@chromium.org, Jun 30 2016

Labels: -M-53 -MovedFrom-52 M-52

Comment 6 by jrbarnette@chromium.org, Jun 30 2016

Status: Fixed (was: Assigned)

These CLs are now in:
    https://chromium-review.googlesource.com/#/c/357570/
    https://chromium-review.googlesource.com/#/c/357571/

... and now the builder is now green, so I'm declaring victory.