During operations like GPU driver or graphics card upgrades, the bots in the Swarming pool will generally be in one of two configurations: the previous config and the next one.

The differences between these might include the PCI ID of the graphics card; the driver version; the exact OS version (i.e., an upgrade from Win7 to Win10, or macOS 10.12 to 10.13), etc. It also might involve changes to more than one of these dimensions.

In https://cs.chromium.org/chromium/src/content/test/gpu/generate_buildbot_json.py we prefer to be precise in the Swarming dimensions specified when we trigger tests, to avoid problems we've seen in the past where a one-off machine added to the Swarming pool for a particular purpose inadvertently has jobs triggered against it that weren't supposed to run on it.

In Issue 756295 we requested an "or" operator for Swarming dimensions, to be evaluated on the server. While this would be convenient for specification in the JSON files, the semantics would become messy pretty quickly. Which one would be preferred and why? What would the fallback criteria be?

martiniss@ pointed out in https://bugs.chromium.org/p/chromium/issues/detail?id=756295#c7 that he's implemented a new mechanism called trigger scripts, which have the opportunity to run code between the time the recipe runs a Swarming step, and when the shards are actually triggered on the bots. From that comment:

""" https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/swarming/api.py?q=trigger_script&sq=package:chromium&dr=C&l=1205 are the docs on this feature. You can add a "trigger_script" key in a //testing/buildbot/*.json file, and it should use this logic. https://cs.chromium.org/chromium/src/tools/perf/perf_device_trigger.py?q=perf_device&sq=package:chromium&l=1 is the trigger script we're using, if you want to see some sample code for a trigger script.
"""

After discussion with martiniss@ I think we should use this mechanism rather than asking for the Swarming feature server-side. The semantics of the server side would be complicated, and with a bit of work, all the flexibility, and more, can be implemented on the client side.

The rough sketch would be:

1) We write a GPU-test-specific trigger script somewhere. This script will receive as input all of the usual arguments to "swarming.py trigger". If a bot is in the process of being migrated, it will pass down via the trigger script's "args" argument (somehow, TBD) the two or more dimension sets it expects to see in the Swarming pool.

2) The trigger script makes multiple queries against the Swarming pool to enumerate the bots available for each dimension set.

3) Based on these results, the trigger script makes a determination of which dimension set it should actually use for each shard. It might consider factors like:
  - The number of available bots for that dimension set
  - If no available bots, the number of busy bots for each dimension set, and assign shards probabilistically

4) The trigger script writes out a JSON file describing how each shard is to be triggered, which is picked up by Swarming.

Then generate_buildbot_json.py can be changed so that for a given bot, like "Win7 Release (NVIDIA)", multiple swarming_dimensions can be set. If this is the case, a trigger_script property would be specified in the JSON file it outputs, passing all of the swarming_dimensions down to be evaluated as above.

There are some downsides to this approach, like there being a sort of race condition with other clients triggering jobs, and the job not being able to migrate between configurations once specified. However, these are probably relatively minor.

We should implement this before the next big upgrade in the fleet. It can't be done in a rush and so can't be done for the GeForce GT 610 -> Quadro P400 upgrade currently underway.