New issue
Advanced search Search tips

Issue 855801 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocked on:
issue 795316



Sign in to add a comment

CancelObsoleteSlaveBuilds isn't working.

Project Member Reported by dgarr...@chromium.org, Jun 23 2018

Issue description

I did a manual test with master-full, and found that cancelling the master does NOT cancel the slave builds from the previous build.

On swarming, this means that we end up with multiple instances of the slaves in question running in parallel, which can be bad.

 
Project Member

Comment 1 by bugdroid1@chromium.org, Jun 26 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/29cb3db927ce81db7fac7fdf47bb3d145d2b4eb2

commit 29cb3db927ce81db7fac7fdf47bb3d145d2b4eb2
Author: Don Garrett <dgarrett@google.com>
Date: Tue Jun 26 00:20:24 2018

build_stages: Add logging to CancelObsoleteSlaveBuilds.

CancelObsoleteSlaveBuilds isn't always working in production, so add
logging to help with debugging.

BUG= chromium:855801 
TEST=run_tests

Change-Id: I7d7a4db8d49f79610b46fcc8b62be64d4f543dc3
Reviewed-on: https://chromium-review.googlesource.com/1112855
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Mike Nichols <mikenichols@chromium.org>

[modify] https://crrev.com/29cb3db927ce81db7fac7fdf47bb3d145d2b4eb2/cbuildbot/stages/build_stages.py

I cancelled this master-full builder:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8942710896101467568

Then forced this one to run right away:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8942700342395253280

During it's cleanup stage, it logged this:

17:47:03: INFO: Found Previous Master builds: 8942728797411534576, 8942740539238796304, 8942752430009638336


This shows that the buildbucket ID of the previous build wasn't part of the discovered list:

Current:
  8942700342395253280

Previous:
  8942710896101467568

Discovered:
  8942728797411534576
  8942740539238796304
  8942752430009638336

History found via Legoland:
  8942700342395253280
  8942710896101467568
  8942728797411534576
  8942740539238796304
  8942752430009638336

Project Member

Comment 3 by bugdroid1@chromium.org, Jun 26 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4ed6cd004572a366d718c85f95d9516661dcab1b

commit 4ed6cd004572a366d718c85f95d9516661dcab1b
Author: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Tue Jun 26 08:58:15 2018

Roll src/third_party/chromite c8a626f49c09..29cb3db927ce (2 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/c8a626f49c09..29cb3db927ce


git log c8a626f49c09..29cb3db927ce --date=short --no-merges --format='%ad %ae %s'
2018-06-26 dgarrett@google.com build_stages: Add logging to CancelObsoleteSlaveBuilds.
2018-06-26 metzman@chromium.org [fuzzing] Make env script handle new builds better.


Created with:
  gclient setdep -r src/third_party/chromite@29cb3db927ce

The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG= chromium:855801 ,chromium:648042
TBR=chrome-os-gardeners@chromium.org

Change-Id: I43554c0983bc43f97ae4f021b113baafd720556b
Reviewed-on: https://chromium-review.googlesource.com/1114047
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#570351}
[modify] https://crrev.com/4ed6cd004572a366d718c85f95d9516661dcab1b/DEPS

Blockedon: 795316
The issue was that abort from LUCI Scheduler doesn't work, NOT that the search was wrong.

Status: Fixed (was: Started)

Sign in to add a comment