M63 CanaryCompletion - Failing to get SlaveBuildbucketInfo |
||||||||
Issue description
M63 release builders failing to GetAllSlaveBuildbucketInfo
Log snippet:
@@@STEP_FAILURE@@@
11:57:43: ERROR: Traceback (most recent call last):
File "/b/c/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 696, in Run
self.PerformStage()
File "/b/c/cbuild/repository/chromite/cbuildbot/stages/completion_stages.py", line 247, in PerformStage
statuses = self._FetchSlaveStatuses()
File "/b/c/cbuild/repository/chromite/cbuildbot/stages/completion_stages.py", line 170, in _FetchSlaveStatuses
manager, build_id, db, builder_names, timeout))
File "/b/c/cbuild/repository/chromite/cbuildbot/stages/completion_stages.py", line 127, in _GetSlaveBuildStatus
timeout=timeout)
File "/b/c/cbuild/repository/chromite/cbuildbot/manifest_version.py", line 649, in GetBuildersStatus
side_effect_func=_PrintRemainingTime)
File "/b/c/cbuild/repository/chromite/lib/timeout_util.py", line 317, in WaitForSuccess
return retry()
File "/b/c/cbuild/repository/chromite/lib/timeout_util.py", line 291, in retry
if not retry_check(value):
File "/b/c/cbuild/repository/chromite/cbuildbot/manifest_version.py", line 645, in <lambda>
lambda x: slave_status.ShouldWait(),
File "/b/c/cbuild/repository/chromite/cbuildbot/build_status.py", line 535, in ShouldWait
retval, slaves_remain, long_pole = self._ShouldWait()
File "/b/c/cbuild/repository/chromite/cbuildbot/build_status.py", line 562, in _ShouldWait
self._GetUncompletedExperimentalBuildbucketIDs())
File "/b/c/cbuild/repository/chromite/cbuildbot/build_status.py", line 397, in _GetUncompletedExperimentalBuildbucketIDs
self.dry_run
File "/b/c/cbuild/repository/chromite/lib/builder_status_lib.py", line 418, in GetAllSlaveBuildbucketInfo
assert buildbucket_client is not None, 'buildbucket_client is None'
AssertionError: buildbucket_client is None
11:57:43: INFO: Translating result buildbucket_client is None to fail.
Full log:
https://uberchromegw.corp.google.com/i/chromeos_release/builders/master-release%20release-R63-10032.B/builds/11/steps/CanaryCompletion/logs/stdio
,
Oct 23 2017
+nxia any idea what's going on here?
,
Oct 23 2017
Introduced by https://chromium-review.googlesource.com/c/chromiumos/chromite/+/660881/ we should only query buildbucket if the master scheduled slaves using buildbucket. Currently the code uses "if self.new_buildbucket_info_dict is not None:" to differentiate the master using buildbucket from the other not using buildbucket.
,
Oct 23 2017
This is a release blocker on M63, so please prioritize a fix. Land it in ToT, then use chromite/bin/cros_merge_to_branch to merge it to M63 (or get josafat's assistance with that).
,
Oct 23 2017
yes, please update with any CLs and please consider auto approved (or ping me) once validated in ToT
,
Oct 24 2017
CL up. I'm not really sure why we even had that bare assert.
,
Oct 24 2017
^ link to CL?
,
Oct 24 2017
crrev.com/c/735523, and everyone here is CC'ed on it
,
Oct 25 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/955a804f23160550b3e7a924f64dbd8fa57309f4 commit 955a804f23160550b3e7a924f64dbd8fa57309f4 Author: Jacob Kopczynski <jkop@google.com> Date: Wed Oct 25 20:51:47 2017 Replace buildbucket assert with short-circuit. Instead of asserting the buildbucket client is present while getting info, short-circuit the method and return an empty dict if it is not present. TEST=tryjob BUG= chromium:777588 Change-Id: Ie27e3cb85e9e0b68c2ce53ef819ef79e25b126b4 Reviewed-on: https://chromium-review.googlesource.com/735523 Commit-Ready: Jacob Kopczynski <jkop@chromium.org> Tested-by: Jacob Kopczynski <jkop@chromium.org> Reviewed-by: Ningning Xia <nxia@chromium.org> [modify] https://crrev.com/955a804f23160550b3e7a924f64dbd8fa57309f4/lib/builder_status_lib.py
,
Oct 25 2017
Anything else pending to resolve this one (e.g. merge to m63), if not please go ahead and mark as fixed.
,
Oct 25 2017
Merge to M63 has not happened. Will do tomorrow morning.
,
Oct 26 2017
ok, marking this as approved for merge. Please confirm when merge is done
,
Oct 26 2017
Ran `cros_merge_to_branch 955a804f23160550b3e7a924f64dbd8fa57309f4 release-R63-10032.B` successfully. Not sure if there's more to do on my end; I found a relevant playbook but it's using some deprecated commands: g3doc/cloud/containers/image/g3doc/oncall/playbook#cherry-pick-a-change-to-branch
,
Oct 26 2017
That just creates a CL that you still need to land: https://chromium-review.googlesource.com/c/chromiumos/chromite/+/739955
,
Oct 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/7c632d0a71b0ff08310ce30c27734fb55ba8965d commit 7c632d0a71b0ff08310ce30c27734fb55ba8965d Author: Jacob Kopczynski <jkop@google.com> Date: Thu Oct 26 18:38:15 2017 Replace buildbucket assert with short-circuit. Instead of asserting the buildbucket client is present while getting info, short-circuit the method and return an empty dict if it is not present. TEST=tryjob BUG= chromium:777588 Change-Id: Ie27e3cb85e9e0b68c2ce53ef819ef79e25b126b4 Previous-Reviewed-on: https://chromium-review.googlesource.com/735523 (cherry picked from commit 9b95adfbf2f89a7c1fa00ae36a54ae28bcfec422) Reviewed-on: https://chromium-review.googlesource.com/739955 Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/7c632d0a71b0ff08310ce30c27734fb55ba8965d/lib/builder_status_lib.py
,
Oct 26 2017
,
Oct 30 2017
This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible! If all merges have been completed, please remove any remaining Merge-Approved labels from this issue. Thanks for your time! To disable nags, add the Disable-Nags label. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Oct 30 2017
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by akes...@chromium.org
, Oct 23 2017