New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 661296 link

Starred by 2 users
Status: Archived
Owner:
█ nxia@chromium.org
Last visit > 30 days ago
Closed: Feb 2017
Cc:
akes...@chromium.org
█ nxia@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

PreCQ Launcher Crashes because of CL conflict.

Project Member Reported by dgarr...@chromium.org, Nov 1 2016 
This PreCQ launcher run:
https://uberchromegw.corp.google.com/i/chromeos/builders/pre-cq-launcher/builds/7997/

Crashed:
https://uberchromegw.corp.google.com/i/chromeos/builders/pre-cq-launcher/builds/7997/steps/PreCQLauncher/logs/stdio


11:10:51: INFO: Attempting to create transaction for kelinlee:*294659:*fd6fa3a1
11:10:52: INFO: Change kelinlee:*294659:*fd6fa3a1 will be submitted
11:10:57: WARNING: A transient error occured while querying chrome-internal-review.googlesource.com:

POST /a/changes/294659/revisions/fd6fa3a16178d9625729e2aa0f5112f61c4e38d9/submit HTTP/1.1
HTTP/1.1 409 Conflict
Response body: 'Error submitting change\n'
X-ErrorId: 409, EFERW-298,EFERW-284,RAS-1118,RAS-428,RAF-125,RIUF-55,RCF-75,GSMSGAFPF-179,GQMGQF-79,RSD-268,SSMHSSF-237,HMF-197,AF-223,SSMSLRF-146,SSPF-71,GFIF-30,GFMMF-235,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-238,CAF-127,SCZFI-172,SCMDRF-215,EF-233,GUFUAF-202,GBLHCF-141,TF-62,TF-54,TF-54,TF-46,GCHSR-63,STSF-84;MO-432,S-223,S-81,RAS-374,RAF-125,RIUF-55,RCF-75,GSMSGAFPF-179,GQMGQF-79,RSD-268,SSMHSSF-237,HMF-197,AF-223,SSMSLRF-146,SSPF-71,GFIF-30,GFMMF-235,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-238,CAF-127,SCZFI-172,SCMDRF-215,EF-233,GUFUAF-202,GBLHCF-141,TF-62,TF-54,TF-54,TF-46,GCHSR-63,STSF-84;MO-486,MO-429,S-223,S-81,RAS-374,RAF-125,RIUF-55,RCF-75,GSMSGAFPF-179,GQMGQF-79,RSD-268,SSMHSSF-237,HMF-197,AF-223,SSMSLRF-146,SSPF-71,GFIF-30,GFMMF-235,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-238,CAF-127,SCZFI-172,SCMDRF-215,EF-233,GUFUAF-202,GBLHCF-141,TF-62,TF-54,TF-54,TF-46,GCHSR-63,STSF-84;BU-494,MO-467,MO-429,S-223,S-81,RAS-374,RAF-125,RIUF-55,RCF-75,GSMSGAFPF-179,GQMGQF-79,RSD-268,SSMHSSF-237,HMF-197,AF-223,SSMSLRF-146,SSPF-71,GFIF-30,GFMMF-235,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-238,CAF-127,SCZFI-172,SCMDRF-215,EF-233,GUFUAF-202,GBLHCF-141,TF-62,TF-54,TF-54,TF-46,GCHSR-63,STSF-84;BU-457,MO-467,MO-429,S-223,S-81,RAS-374,RAF-125,RIUF-55,RCF-75,GSMSGAFPF-179,GQMGQF-79,RSD-268,SSMHSSF-237,HMF-197,AF-223,SSMSLRF-146,SSPF-71,GFIF-30,GFMMF-235,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-238,CAF-127,SCZFI-172,SCMDRF-215,EF-233,GUFUAF-202,GBLHCF-141,TF-62,TF-54,TF-54,TF-46,GCHSR-63,STSF-84;WI-228,WI-44,WI-136,WI-133,WI-149,CI-183,CIIT-304,CIIT-263
11:10:57: WARNING: conn.sock.getpeername(): ('108.177.11.82', 443)
11:10:57: ERROR: CL:*294659 could not be submitted because Gerrit reported a conflict. Did you modify your patch during the CQ run? Or do you just need to rebase?
11:10:57: ERROR: Could not submit kelinlee:*294659:*fd6fa3a1
11:11:00: WARNING: A transient error occured while querying chrome-internal-review.googlesource.com:

POST /a/changes/294659/revisions/4a9cf1e581d4fd952ebb5f97714e8ff1475c9d43/review HTTP/1.1
HTTP/1.1 409 Conflict
Response body: 'Cannot reduce vote on labels for closed change: Commit-Queue\n'
X-ErrorId: 409, EFERW-298,EFERW-284,RAS-1118,RAS-428,RAF-125,RIUF-55,RCF-75,GSMSGAFPF-179,GQMGQF-79,RSD-268,SSMHSSF-237,HMF-197,AF-223,SSMSLRF-146,SSPF-71,GFIF-30,GFMMF-235,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-238,CAF-127,SCZFI-172,SCMDRF-215,EF-233,GUFUAF-202,GBLHCF-141,TF-46,TF-46,GCHSR-63,STSF-84;PRO-859,PRO-805,PRO-480,BUCT-948,BUCT-918,BUCT-892,BU-766,BU-430,BU-673,BU-668,PR-233,PR-161,PR-108,RAS-374,RAF-125,RIUF-55,RCF-75,GSMSGAFPF-179,GQMGQF-79,RSD-268,SSMHSSF-237,HMF-197,AF-223,SSMSLRF-146,SSPF-71,GFIF-30,GFMMF-235,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-238,CAF-127,SCZFI-172,SCMDRF-215,EF-233,GUFUAF-202,GBLHCF-141,TF-46,TF-46,GCHSR-63,STSF-84
11:11:00: WARNING: conn.sock.getpeername(): ('108.177.11.82', 443)

@@@STEP_FAILURE@@@
11:11:00: ERROR: <class 'chromite.lib.gob_util.GOBError'>: Conflict
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/failures_lib.py", line 172, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 1931, in PerformStage
    builder_run=self._run)
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 1460, in AcquirePool
    if pool.AcquireChanges(gerrit_query, ready_fn, change_filter):
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 1360, in AcquireChanges
    self, self.candidates, self.non_manifest_changes)
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 1874, in ProcessChanges
    reason=constants.STRATEGY_NONMANIFEST)
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 2387, in SubmitNonManifestChanges
    check_tree_open=check_tree_open)
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 2000, in SubmitChanges
    self._HandleCouldNotSubmit(patch, error)
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 2635, in _HandleCouldNotSubmit
    self.RemoveReady(change)
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 2336, in RemoveReady
    self._helper_pool.ForChange(change).RemoveReady(change, dryrun=self.dryrun)
  File "/b/cbuild/internal_master/chromite/lib/gerrit.py", line 375, in RemoveReady
    label='Commit-Queue', notify='OWNER')
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 513, in ResetReviewLabels
    response = FetchUrlJson(host, path, reqtype='POST', body=body)
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 232, in FetchUrlJson
    fh = FetchUrl(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 224, in FetchUrl
    _FetchUrlHelper, sleep=SLEEP)
  File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 155, in RetryException
    return GenericRetry(exc_retry, max_retry, functor, *args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 105, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 221, in _FetchUrlHelper
    raise GOBError(response.status, response.reason)
GOBError: Conflict

Comment 1 by dgarr...@chromium.org, Nov 1 2016 

This CL was the problem:

https://chrome-internal-review.googlesource.com/#/c/294659/

It was successfully submitted by the requests above, AND had a failure message appended.

Comment 2 by dgarr...@chromium.org, Nov 1 2016

Owner: dgarr...@chromium.org

Comment 3 by autumn@chromium.org, Nov 1 2016

Labels: -current-issue

Comment 4 by dgarr...@chromium.org, Nov 16 2016

Owner: nxia@chromium.org
Is this still an issue?

Comment 6 by nxia@chromium.org, Feb 2 2017 

Error snippet from #5

[1;33m14:32:10: WARNING: A transient error occured while querying chromium-review.googlesource.com:

POST /a/changes/412327/revisions/a947622cafd7322c192038884ad91a0bd4df038a/review HTTP/1.1
HTTP/1.1 409 Conflict
Response body: 'change is closed\n'
X-ErrorId: 409, EFERW-298,EFERW-284,RAS-1118,RAS-428,RAF-125,RIUF-55,RCF-75,GSMSGAFPF-179,GQMGQF-79,RSD-268,SSMHSSF-237,HMF-197,AF-223,SSMSLRF-146,SSPF-71,GFIF-30,GFMMF-235,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-238,CAF-127,SCZFI-172,SCMDRF-215,EF-233,GUFUAF-202,GBLHCF-141,TF-46,TF-46,GCHSR-63,STSF-84,PLRHF-156,TTLF-35,GPLF-113;PRO-895,PRO-868,PRO-543,BUCT-955,BUCT-918,BUCT-892,BU-766,BU-430,BU-673,BU-668,PR-280,PR-168,PR-115,RAS-374,RAF-125,RIUF-55,RCF-75,GSMSGAFPF-179,GQMGQF-79,RSD-268,SSMHSSF-237,HMF-197,AF-223,SSMSLRF-146,SSPF-71,GFIF-30,GFMMF-235,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-233,GFMMF-238,CAF-127,SCZFI-172,SCMDRF-215,EF-233,GUFUAF-202,GBLHCF-141,TF-46,TF-46,GCHSR-63,STSF-84,PLRHF-156,TTLF-35,GPLF-113
14:32:10: WARNING: conn.sock.getpeername(): ('74.125.139.82', 443)

14:32:10: ERROR: <class 'chromite.lib.gob_util.GOBError'>: Conflict
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/failures_lib.py", line 172, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 1955, in PerformStage
    builder_run=self._run)
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 1460, in AcquirePool
    if pool.AcquireChanges(gerrit_query, ready_fn, change_filter):
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 1360, in AcquireChanges
    self, self.candidates, self.non_manifest_changes)
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 1863, in ProcessChanges
    pool, launchable_progress_map):
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 1559, in GetDisjointTransactionsToTest
    max_txn_length=self.MAX_PATCHES_PER_TRYBOT_RUN)
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 2851, in CreateDisjointTransactions
    self._HandleApplyFailure(failed)
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 2471, in _HandleApplyFailure
    self._HandleCouldNotApply(failure)
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 2485, in _HandleCouldNotApply
    self.RemoveReady(failure.patch)
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 2339, in RemoveReady
    self._helper_pool.ForChange(change).RemoveReady(change, dryrun=self.dryrun)
  File "/b/cbuild/internal_master/chromite/lib/gerrit.py", line 377, in RemoveReady
    label='Trybot-Ready', notify='OWNER')
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 513, in ResetReviewLabels
    response = FetchUrlJson(host, path, reqtype='POST', body=body)
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 232, in FetchUrlJson
    fh = FetchUrl(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 224, in FetchUrl
    _FetchUrlHelper, sleep=SLEEP)
  File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 172, in RetryException
    return GenericRetry(exc_retry, max_retry, functor, *args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 122, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 221, in _FetchUrlHelper
    raise GOBError(response.status, response.reason)
GOBError: Conflict


Comment 7 by nxia@chromium.org, Feb 2 2017

Labels: Hotlist-Fixit

Comment 8 by nxia@chromium.org, Feb 2 2017

Cc: akes...@chromium.org nxia@chromium.org
 Issue 676155  has been merged into this issue.
Project Member

Comment 9 by bugdroid1@chromium.org, Feb 3 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/2b7dd13e6f9fb0719aef48facaa4c115d83b4def

commit 2b7dd13e6f9fb0719aef48facaa4c115d83b4def
Author: Ningning Xia <nxia@chromium.org>
Date: Fri Feb 03 13:15:11 2017

Do not raise certain 409 GOBErrors on RemoveReady.

Do not raise exceptions when RemoveReady trys to remove ready labels on
CLs which have been closed.

BUG= chromium:661296 
TEST=unit_tests

Change-Id: I6fcb2ff15ea5e8d8b5f14c76d8b858078220c1ca
Reviewed-on: https://chromium-review.googlesource.com/435651
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/2b7dd13e6f9fb0719aef48facaa4c115d83b4def/cbuildbot/validation_pool.py
[modify] https://crrev.com/2b7dd13e6f9fb0719aef48facaa4c115d83b4def/lib/gob_util_unittest.py
[modify] https://crrev.com/2b7dd13e6f9fb0719aef48facaa4c115d83b4def/cbuildbot/validation_pool_unittest.py
[modify] https://crrev.com/2b7dd13e6f9fb0719aef48facaa4c115d83b4def/lib/gob_util.py

Comment 10 by nxia@chromium.org, Feb 3 2017

Labels: cros-infra-fixedit-q117
Status: Fixed (was: Untriaged)

Comment 11 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 12 by dchan@google.com, May 30 2017

Labels: VerifyIn-60

Comment 13 by dchan@chromium.org, Aug 1 2017

Labels: VerifyIn-61

Comment 14 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)
►

Sign in to add a comment