New issue
Advanced search Search tips
Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

CanaryCompletion failing on multiple builders

Project Member Reported by mnissler@chromium.org, Apr 5 Back to list

Issue description

As previously observed in issue 800949, I'm seeing CanaryCompletion failures like this:

00:48:25: INFO: Updating manifest-versions checkout.
00:50:20: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:50:20: ERROR: Retrying to update the status:  Retry 1/20
00:50:20: INFO: Updating manifest-versions checkout.
00:50:58: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:50:58: ERROR: Retrying to update the status:  Retry 2/20
00:50:58: INFO: Updating manifest-versions checkout.
00:51:36: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:51:36: ERROR: Retrying to update the status:  Retry 3/20
00:51:36: INFO: Updating manifest-versions checkout.
00:52:14: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:52:14: ERROR: Retrying to update the status:  Retry 4/20
00:52:14: INFO: Updating manifest-versions checkout.
00:52:52: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:52:52: ERROR: Retrying to update the status:  Retry 5/20
00:52:52: INFO: Updating manifest-versions checkout.
00:53:30: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:53:30: ERROR: Retrying to update the status:  Retry 6/20
00:53:30: INFO: Updating manifest-versions checkout.
00:54:07: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:54:07: ERROR: Retrying to update the status:  Retry 7/20
00:54:07: INFO: Updating manifest-versions checkout.
00:54:45: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:54:45: ERROR: Retrying to update the status:  Retry 8/20
00:54:45: INFO: Updating manifest-versions checkout.
00:55:23: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:55:23: ERROR: Retrying to update the status:  Retry 9/20
00:55:23: INFO: Updating manifest-versions checkout.
00:56:01: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:56:01: ERROR: Retrying to update the status:  Retry 10/20
00:56:01: INFO: Updating manifest-versions checkout.
00:56:40: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:56:40: ERROR: Retrying to update the status:  Retry 11/20
00:56:40: INFO: Updating manifest-versions checkout.
00:57:19: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:57:19: ERROR: Retrying to update the status:  Retry 12/20
00:57:19: INFO: Updating manifest-versions checkout.
00:57:56: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:57:56: ERROR: Retrying to update the status:  Retry 13/20
00:57:57: INFO: Updating manifest-versions checkout.
00:58:34: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:58:34: ERROR: Retrying to update the status:  Retry 14/20
00:58:34: INFO: Updating manifest-versions checkout.
00:59:12: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:59:12: ERROR: Retrying to update the status:  Retry 15/20
00:59:12: INFO: Updating manifest-versions checkout.
00:59:49: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
00:59:49: ERROR: Retrying to update the status:  Retry 16/20
00:59:49: INFO: Updating manifest-versions checkout.
01:00:27: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
01:00:27: ERROR: Retrying to update the status:  Retry 17/20
01:00:27: INFO: Updating manifest-versions checkout.
01:01:04: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
01:01:04: ERROR: Retrying to update the status:  Retry 18/20
01:01:04: INFO: Updating manifest-versions checkout.
01:01:42: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
01:01:42: ERROR: Retrying to update the status:  Retry 19/20
01:01:42: INFO: Updating manifest-versions checkout.
01:02:19: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
01:02:19: ERROR: Retrying to update the status:  Retry 20/20
01:02:19: INFO: Updating manifest-versions checkout.
01:02:57: ERROR: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal
01:02:57: ERROR: Retrying to update the status:  Retry 21/20
01:02:57: INFO: Updating manifest-versions checkout.
01:03:18: ERROR: Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 701, in Run
    self.PerformStage()
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/completion_stages.py", line 264, in PerformStage
    super(MasterSlaveSyncCompletionStage, self).PerformStage()
  File "/b/c/cbuild/repository/chromite/cbuildbot/stages/completion_stages.py", line 84, in PerformStage
    message=self.message)
  File "/b/c/cbuild/repository/chromite/cbuildbot/manifest_version.py", line 780, in UpdateStatus
    raise StatusUpdateException(last_error)
StatusUpdateException: Failed to update the status for chell-release during remote command: cmd=['git', 'push', 'origin', 'temp_auto_checkin_branch:refs/heads/master'], cwd=/b/c/cbuild/repository/manifest-versions-internal



Note that these errors don't look like the expected flakes due to concurrent commit attempts from multiple builders, but something else. Too bad that we don't see any diagnostics / error codes from git.

This affects multiple builders, here are the runs I could find from the red blobs I'm seeing on the waterfall:

https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fauron_yuna-release%2F2068%2F%2B%2Frecipes%2Fsteps%2FCanaryCompletion%2F0%2Fstdout
https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fchell-release%2F2048%2F%2B%2Frecipes%2Fsteps%2FCanaryCompletion%2F0%2Fstdout
https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Felm-release%2F2083%2F%2B%2Frecipes%2Fsteps%2FCanaryCompletion%2F0%2Fstdout
https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fenguarde-release%2F2213%2F%2B%2Frecipes%2Fsteps%2FCanaryCompletion%2F0%2Fstdout
https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fnami-release%2F345%2F%2B%2Frecipes%2Fsteps%2FCanaryCompletion%2F0%2Fstdout
https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Forco-release%2F2086%2F%2B%2Frecipes%2Fsteps%2FCanaryCompletion%2F0%2Fstdout
https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fscarlet-release%2F1237%2F%2B%2Frecipes%2Fsteps%2FCanaryCompletion%2F0%2Fstdout
https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fveyron_minnie-release%2F2085%2F%2B%2Frecipes%2Fsteps%2FCanaryCompletion%2F0%2Fstdout

Adding people on previous bug to CC.

I'll watch the situation to see whether this persists.
 
Cc: -akes...@chromium.org -jrbarnette@chromium.org -pprabhu@chromium.org jclinton@chromium.org
Components: -Infra>Client>ChromeOS Infra>Client>ChromeOS>CI

Comment 3 by jclinton@chromium.org, Apr 18 (3 days ago)

Cc: mikenichols@chromium.org
Labels: -Pri-3 Pri-2
Owner: mikenichols@chromium.org
Status: Assigned
This looks like another example of Git-on-Borg not meeting its SLO's. We should probably have better logging in this section of the CBuildBot code. Mike recently touched this code; Mike can you improve the error logging here a little bit so we have something to go to the GoB team with?

Comment 4 by mikenichols@chromium.org, Apr 18 (3 days ago)

I can look to add some timing to have a better idea of what is going on.  The challenge may be that we recently removed the git retry logic, as it is being handled by the git wrapper itself, which may be the better option for logging the timings.  I look into both options for the best approach.

-- Mike

Sign in to add a comment