Forking this from  issue 741001 .

Background:
Previously on Chrome Infra, we installed <some version> of python with <some version> of <some pip packages>, which build tasks then relied on.

We'd like to get to the point where a given task (T) completely specifies its dependencies for things it depends on.

Constraints:
This needs to work roughly the same way for all the platforms we support (so, mac, linux, windows). This unfortunately precludes the use of docker as a primary solution (:sob:, though I think we should ALSO pursue docker for linux builds, it just can't be a "one-and-done" solution to this problem, because it only solves linux).

Approach so far:
We're currently building a hermetic python-in-a-cipd package for all platforms (see third-party-packages builders here: https://build.chromium.org/p/chromium.infra.cron/builders). We also have built a tool called vpython which allows python scripts to explicitly list their dependencies and have a hermetic Python VirtualEnv manufactured for them just-in-time.

Goal:
Ultimately, every swarming task should fully specify it's python version (via the python cipd bundle), and the precise versions of any python packages that it relies on (via a vpython spec), and swarming should do its best* to remove the system python from PATH when executing the task, and we can de-provision all extra packages from the system python (i.e. don't globally install anything with pip).

Doing this will allow clients to do things like CQ upgrades to the version of Python and/or the version(s) of python packages. It will also allow clients with conflicting package version requirements (i.e. requests v1 vs. requests v2) to both operate on the same fleet of bots without having to split and maintain multiple bot pools.

It will also decouple upgrades of the system python and packages (i.e. for the swarming bot's own internal dependencies, or the dependencies for things like the infra-python package) from the version requirements of clients' code/tests.



* Unfortunately scripts which have a shebang like `#!/usr/bin/python` will probably always get the system python. But if we de-provision all extra packages then these will probably switch to using `env` instead to find python, which will work.