New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 677335 link

Starred by 0 users

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Mar 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

pre-cq launcher fails when a dependent cl isn't in the changes set to generate transactions for.

Project Member Reported by kevcheng@chromium.org, Dec 28 2016

Issue description

Is there a bug somewhere? I wouldn't think this would cause the pre-cq launcher to die.

https://luci-milo.appspot.com/buildbot/chromeos/pre-cq-launcher/8347
https://luci-milo.appspot.com/buildbot/chromeos/pre-cq-launcher/8346

10:56:04: INFO: Failed creating transaction for ayatane:419831:8e0c85ae: CL:419831 depends on CL:419830, which depends on CL:419829, which is not marked Code-Review=+2.
10:56:04: INFO: Attempting to create transaction for ihf:407581:a389378b
10:56:05: INFO: Attempting to create transaction for mnissler:420988:dbc9df96

10:56:05: ERROR: <type 'exceptions.KeyError'>: <chromite.lib.patch.GerritPatch object at 0x7f1285e34d10>
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/failures_lib.py", line 172, in wrapped_functor
    return functor(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 1842, in PerformStage
    builder_run=self._run)
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 497, in AcquirePool
    if pool.AcquireChanges(gerrit_query, ready_fn, change_filter):
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 397, in AcquireChanges
    self, self.candidates, self.non_manifest_changes)
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 1750, in ProcessChanges
    pool, launchable_progress_map):
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/sync_stages.py", line 1446, in GetDisjointTransactionsToTest
    max_txn_length=self.MAX_PATCHES_PER_TRYBOT_RUN)
  File "/b/cbuild/internal_master/chromite/cbuildbot/validation_pool.py", line 1796, in CreateDisjointTransactions
    changes, max_txn_length=max_txn_length)
  File "/b/cbuild/internal_master/chromite/cbuildbot/patch_series.py", line 531, in CreateDisjointTransactions
    new_changes = list(dep_change for dep_change in deps[change]
KeyError: <chromite.lib.patch.GerritPatch object at 0x7f1285e34d10>
 
Cc: ayatane@chromium.org
All CLs in the transaction have to be at least CR+2 or TR+1.

If that wasn't the case here, then this is WAI.
Status: WontFix (was: Untriaged)
ah, all righty, closing out as wontfix.  It would be nice if the launcher wouldn't die but instead just ignore the CL or emit a clear message for the waterfall to consume.
The launcher should not die, if that's the case then that is indeed a bug.
Status: Available (was: WontFix)
Agree, the launcher shouldn't die from uncaught exception even if the pre-cq run cannot start.
Labels: Hotlist-Fixit
Labels: -Pri-3 Pri-2
Owner: nxia@chromium.org
This should be a small fix. Since it happened again, upping to P2.

nxia@ can you try to squash this?
Status: Assigned (was: Available)

Comment 9 by nxia@chromium.org, Feb 21 2017

Cc: jrbarnette@chromium.org
The recent pre-cq-launcher failure looks to be a separate issue:


https://uberchromegw.corp.google.com/i/chromeos/builders/pre-cq-launcher/builds/8675/steps/PreCQLauncher/logs/stdio


  File "/b/cbuild/internal_master/chromite/lib/gerrit.py", line 109, in GetChangeDetail
    self.host, change_num, o_params=('CURRENT_REVISION', 'CURRENT_COMMIT'))
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 375, in GetChangeDetail
    return FetchUrlJson(host, path)
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 260, in FetchUrlJson
    fh = FetchUrl(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 252, in FetchUrl
    _FetchUrlHelper, sleep=SLEEP)
  File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 171, in RetryException
    return GenericRetry(exc_retry, max_retry, functor, *args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 122, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 183, in _FetchUrlHelper
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1089, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 400, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 341, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 260, in read
    return self._sslobj.read(len)


Deputy can file a separate bug for the recent pre-cq-launcher failure.

Comment 10 by nxia@chromium.org, Feb 22 2017

Summary: pre-cq launcher fails when a dependent cl isn't in the changes set to generate transactions for. (was: pre-cq launcher fails when a dependent cl isn't marked +2)
Project Member

Comment 11 by bugdroid1@chromium.org, Feb 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/a4e0b251e3955411a716d3e9ca6342ebb05f0ed0

commit a4e0b251e3955411a716d3e9ca6342ebb05f0ed0
Author: Ningning Xia <nxia@chromium.org>
Date: Fri Feb 24 13:42:54 2017

PatchSeries: fix possible KeyError in CreateDisjointTransactions.

It's possible a change was filtered for creating a disjoint transaction
while the other changes it depends on were not picked up in the same
changes set. This can cause a KeyError Exception.
With out the fix, the new unit test TestCreateDisjointTransactions will
fail with KeyError.

BUG= chromium:677335 
TEST=unit_tests

Change-Id: I2860c26cc9c4d15453325efc5b2558e568d63635
Reviewed-on: https://chromium-review.googlesource.com/446019
Commit-Ready: Ningning Xia <nxia@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/a4e0b251e3955411a716d3e9ca6342ebb05f0ed0/cbuildbot/patch_series.py
[modify] https://crrev.com/a4e0b251e3955411a716d3e9ca6342ebb05f0ed0/cbuildbot/patch_series_unittest.py

Comment 12 by nxia@chromium.org, Mar 13 2017

Status: Fixed (was: Assigned)

Sign in to add a comment