Preliminary investigation leads me to think that the best way to solve this is to define multiple LUCI Builders (we only have "Generic" today), and give them priorities.

This requires that buildbucket requests specify the right builder for the desired priority.

The same mechanism can be used to select builder pools, or experimental build infrastructure.

I have a mechanism in place (see  crbug.com/832747 ) and it's being used by tryjobs. However, since we don't share code properly, it's not being used for slave builds.

Keeping this bug open to address slave builds.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/5816f4eacbcabef94a60a5a14a56615f2d890892

commit 5816f4eacbcabef94a60a5a14a56615f2d890892
Author: Don Garrett <dgarrett@google.com>
Date: Fri Apr 27 19:22:36 2018

config_lib: Define LUCI_BUILDER constants.

A previous CL added the concept of LUCI Builders to build configs, but
defined them as free form strings. This CL added fixed constants, and
defines a unittest to check for valid values.

BUG= chromium:824550 
TEST=run_tests && seeing no update to config_dump.json.

Change-Id: I0c525d0cbf9754147cdf74f03ebd670ebb57b886
Reviewed-on: https://chromium-review.googlesource.com/1028978
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/5816f4eacbcabef94a60a5a14a56615f2d890892/lib/remote_try.py
[modify] https://crrev.com/5816f4eacbcabef94a60a5a14a56615f2d890892/lib/config_lib.py
[modify] https://crrev.com/5816f4eacbcabef94a60a5a14a56615f2d890892/cbuildbot/chromeos_config_unittest.py
[modify] https://crrev.com/5816f4eacbcabef94a60a5a14a56615f2d890892/cbuildbot/chromeos_config.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9f7781e9bb60bd6a58e02911d1c35d6031c7a520

commit 9f7781e9bb60bd6a58e02911d1c35d6031c7a520
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Sat Apr 28 00:05:06 2018

Roll src/third_party/chromite/ 6b3c651c4..7c0757b09 (7 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/6b3c651c4ed2..7c0757b099e5

$ git log 6b3c651c4..7c0757b09 --date=short --no-merges --format='%ad %ae %s'
2018-04-27 levarum Revert "Copy ndk-translation artifacts when uprevving android"
2018-04-24 dgarrett som: Remove Gardener SOM tree.
2018-04-13 bmgordon cbuildbot: Use previous build state for chroot reuse
2018-04-25 dgarrett config_lib: Define LUCI_BUILDER constants.
2018-04-25 jclinton appengine/cq_stats: Set the Python SSL version to latest
2018-04-26 jclinton appengine: Update the SDK version to support new SSL
2018-04-26 jclinton lib/clactions*: Refactor into separate clactions_metrics for ts_mon

Created with:
  roll-dep src/third_party/chromite
BUG=chromium:None,chromium:829665,chromium:824550,chromium:826372,chromium:826372,chromium:826372


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: Ic622485c6d32417602f5aff7bc2cc02047e49f9a
Reviewed-on: https://chromium-review.googlesource.com/1033960
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#554572}
[modify] https://crrev.com/9f7781e9bb60bd6a58e02911d1c35d6031c7a520/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/7f3db4da85075e03a3372b146eca539697f71031

commit 7f3db4da85075e03a3372b146eca539697f71031
Author: Don Garrett <dgarrett@google.com>
Date: Wed May 16 23:42:41 2018

cros_tryjob: Move tryjob specific logic into cros tryjob.

I want to refactor lib/remote_try.py into a general library for
submitting ChromeOS build requests that is shared between a tryjob and
master builder logic.

This is so we can have consistent usage of builders, tags, and
properties between all sources of builds.

This CL migrates most tryjob specific logic out of the library, and
into the cros_tryjob command.

BUG= chromium:824550 
TEST=run_tests
     With manual buildbucket examination:
       cros tryjob success-build
       cros tryjob --committer-email dgarrett@chromium.org success-build
       cros tryjob -p chromiumos/chromite success-build

Change-Id: I4544d9eff17a56a25cbc150dbdf49016f5ae0195
Reviewed-on: https://chromium-review.googlesource.com/1031851
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Mike Nichols <mikenichols@chromium.org>

[modify] https://crrev.com/7f3db4da85075e03a3372b146eca539697f71031/lib/remote_try.py
[modify] https://crrev.com/7f3db4da85075e03a3372b146eca539697f71031/lib/remote_try_unittest.py
[modify] https://crrev.com/7f3db4da85075e03a3372b146eca539697f71031/cli/cros/cros_tryjob.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/7df6118ecdd71734d885b616484e6717d6936b1b

commit 7df6118ecdd71734d885b616484e6717d6936b1b
Author: Don Garrett <dgarrett@google.com>
Date: Wed May 16 23:42:42 2018

remote_try: Rearrange a little.

Move most decisions about auto-detected values into the
constructor. Remove remaining autodetection from cros tryjob.

BUG= chromium:824550 
TEST=run_tests + cros tryjob success-build

Change-Id: I3963df483447f54bbd872476da79c0da7dc17000
Reviewed-on: https://chromium-review.googlesource.com/1033923
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Mike Nichols <mikenichols@chromium.org>

[modify] https://crrev.com/7df6118ecdd71734d885b616484e6717d6936b1b/lib/remote_try.py
[modify] https://crrev.com/7df6118ecdd71734d885b616484e6717d6936b1b/lib/remote_try_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/c9e3b0029b92126340034662f35fb8d3da163359

commit c9e3b0029b92126340034662f35fb8d3da163359
Author: Don Garrett <dgarrett@google.com>
Date: Wed May 16 23:42:42 2018

remote_try: Rename to request_build.

This library is being turned into a generic helper for requesting
buildbucket builds.

BUG= chromium:824550 
TEST=run_tests

Change-Id: I547d433a6186fed8ab2d41123a20486d288d7a70
Reviewed-on: https://chromium-review.googlesource.com/1033924
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Mike Nichols <mikenichols@chromium.org>

[rename] https://crrev.com/c9e3b0029b92126340034662f35fb8d3da163359/lib/request_build.py
[rename] https://crrev.com/c9e3b0029b92126340034662f35fb8d3da163359/lib/request_build_unittest.py
[rename] https://crrev.com/c9e3b0029b92126340034662f35fb8d3da163359/lib/request_build_unittest
[modify] https://crrev.com/c9e3b0029b92126340034662f35fb8d3da163359/cli/cros/cros_tryjob.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b6e9f9e54e907eb474b62cc926eb40744bb01ae5

commit b6e9f9e54e907eb474b62cc926eb40744bb01ae5
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Thu May 17 00:53:03 2018

Roll src/third_party/chromite/ 4c1562dfd..966fe6267 (6 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/4c1562dfdeef..966fe62677b1

$ git log 4c1562dfd..966fe6267 --date=short --no-merges --format='%ad %ae %s'
2018-05-14 ayatane sysmon: Add ignored metric to prod manifest
2018-05-16 bmgordon build_stages: Handle exceptions in snapshot commands
2018-05-16 manojgupta cros_fuzz_test_env: Fix cleanup option.
2018-04-27 dgarrett remote_try: Rename to request_build.
2018-04-27 dgarrett remote_try: Rearrange a little.
2018-04-26 dgarrett cros_tryjob: Move tryjob specific logic into cros tryjob.

Created with:
  roll-dep src/third_party/chromite
BUG=chromium:None,chromium:843441,chromium:843627,chromium:824550,chromium:824550,chromium:824550


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: Ibd98b2152ec46624b2c80e06bb60514e17ca9e2a
Reviewed-on: https://chromium-review.googlesource.com/1063030
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#559372}
[modify] https://crrev.com/b6e9f9e54e907eb474b62cc926eb40744bb01ae5/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/26a0c825c51cdae43387ea8386939ff902ea53a6

commit 26a0c825c51cdae43387ea8386939ff902ea53a6
Author: Don Garrett <dgarrett@google.com>
Date: Sat May 19 00:19:50 2018

request_build: Add features needed by master/slave builds.

Compare to PostSlaveBuildToBuildbucket in scheduler_stages.py.

* Pass along both CIDB and buildbucket id's of the master.
* Set buildset: with a unique id for each run.
* Allow the buildbucket to be redefined.
* Set "master:False" for slave builds, but not otherwise.
* Allow luci_builder to be set, so we can handle waterfall columns.
* Return created_ts from buildbucket in scheduling result.

There are some other tags also set by master builders that are no
longer used, and so not added to this library.

BUG= chromium:824550 
TEST=run_tests

Change-Id: I27d22742b4fc0153da52d68535d29c92d75394da
Reviewed-on: https://chromium-review.googlesource.com/1062203
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Mike Nichols <mikenichols@chromium.org>

[modify] https://crrev.com/26a0c825c51cdae43387ea8386939ff902ea53a6/lib/request_build.py
[modify] https://crrev.com/26a0c825c51cdae43387ea8386939ff902ea53a6/lib/request_build_unittest.py
[modify] https://crrev.com/26a0c825c51cdae43387ea8386939ff902ea53a6/cli/cros/cros_tryjob.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/353830374be545c248ba5251ed4e83f2eb1ab37c

commit 353830374be545c248ba5251ed4e83f2eb1ab37c
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Sat May 19 02:14:36 2018

Roll src/third_party/chromite/ 17530e94d..752413029 (3 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/17530e94d1c5..752413029ca9

$ git log 17530e94d..752413029 --date=short --no-merges --format='%ad %ae %s'
2018-05-16 dgarrett CleanUpStage: Revamp build slave cancellation.
2018-05-16 dgarrett ScheduleSlavesStage: Fix master buildbucket id.
2018-05-15 dgarrett request_build: Add features needed by master/slave builds.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:834076 ,chromium:None,chromium:824550


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I68e7f2ad0d0e706785667b5bc74c072491b61a4e
Reviewed-on: https://chromium-review.googlesource.com/1066928
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#560145}
[modify] https://crrev.com/353830374be545c248ba5251ed4e83f2eb1ab37c/DEPS