I don't think this is pri-1, moving to pri-2. Dirk, please correct me if I'm wrong.

How much work is this going to be to deploy everywhere? Who has signed off on the design?

I circulated the design doc, and anyone who expressed an interest seemed to be happy with it. P2 is fine, don't really care.

Do we have enough spare cycles to actually deploy it completely, everywhere? I don't want us to end up in a situation where we just have +1 (weird-ish) environment to support, in additional to existing ones.

IMHO if we don't have enough stamina to push it to the completion, better not start at all.

I don't think that I'll have cycles by myself. However, this is designed to be iteratively rolled out. I suspect we'll want to distribute the work between Platform, SREs, and our clients (Chromium, ChromeOS, etc.).

That said, I'm okay getting it to the point where the binary is deployed on all systems.

I may have set the wrong expectations earlier ...

I think the idea sounds like a good one, and it's a problem worth solving. It's not yet clear to me how much work things will be or what the rollout plan is and I agree that we shouldn't spend much time working on this until we have a better sense of things.

It would be good, for example, to decide if life is better if we roll this out but don't *also* make sure all of the other uses are stomped out.

Thoughts on how to get a better sense of things? This can live un-reviewed in a development branch indefinitely, or we can land it and start experimenting for real. I don't think we can experiment in earnest without landing it first (i.e., moving binaries and messing with local configs is bad).

Maybe push to "infra/experimental/..." as a halfway point? The only problem there is that it will be more difficult to review if we decide to commit, since it's bulk-landed.

I don't really have a good feel for the dynamic of the decision. This is designed to be iterative and complementary; however, it will definitely run out of steam if we don't have strategic support. What are the next steps, if not putting the code up for review? If getting approval to proceed is too much work, that does not bode well.

I think getting the code landed is a separate question from deciding whether or not to roll things out. I.e., it makes sense to do the former even if we then decide to put it on the back burner for a quarter or two.

Okay, I think that's fine. I do agree with Vadim in #4; however, I don't think it needs to be all-at-once or not-at-all. I see a few stages:

- Getting our services onto it, and, in doing so, dogfooding the experience.
  - Probably warrants an exploratory foray followed by a focused effort.
  - End goal is "all Infra runs on vpython".
- Moving Recipe Engine onto it, and, through that, encroaching on our build stack.
- Moving individual client tools onto it. This would require us to reach out to identify tools and reach out to some specific tool owners.

I think this is worthy of a design document in and of itself, and I doubt that everything will ultimately be converted. We'll be left with a mix of:
1) Fully-converted services (vpython-aware and explicit)
2) Services that use vpython on virtue of inheriting it from a launcher such as Recipe Engine, Kitchen, Swarming, etc.
3) Services that explicitly invoke via "/usr/bin/python" or the like.

End goal would be to have as many of (1) as possible, with (2) as a fine alternative and (3) something worth minimizing.

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/49cc23e49067b4303d84e86a81e85d25bb27f3ee

commit 49cc23e49067b4303d84e86a81e85d25bb27f3ee
Author: dnj <dnj@chromium.org>
Date: Wed Feb 22 06:47:45 2017

vpython: Add README and environment protobuf.

Add the vpython tool README and environment protobuf. The README details
the tool and its layout and usage.

BUG= chromium:691818 
TEST=None

Review-Url: https://codereview.chromium.org/2703463006

[add] https://crrev.com/49cc23e49067b4303d84e86a81e85d25bb27f3ee/vpython/README.md
[add] https://crrev.com/49cc23e49067b4303d84e86a81e85d25bb27f3ee/vpython/api/env/gen.go
[add] https://crrev.com/49cc23e49067b4303d84e86a81e85d25bb27f3ee/vpython/api/env/spec.pb.go
[add] https://crrev.com/49cc23e49067b4303d84e86a81e85d25bb27f3ee/vpython/api/env/spec.proto

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/091cb1591f7a0516f730228126c04e5574dc7db0

commit 091cb1591f7a0516f730228126c04e5574dc7db0
Author: dnj <dnj@chromium.org>
Date: Wed Feb 22 07:21:24 2017

vpython: Add filesystem operation / test packages.

Adds the "filesystem" vpython package, which includes the filesystem
operations used by the "vpython" tool.

Also includes "filesystem/testfs", which are testing-oriented filesystem
operations.

Both of these will be heavily leveraged in future patch sets.

BUG= chromium:691818 
TEST=None

Review-Url: https://codereview.chromium.org/2703793003

[add] https://crrev.com/091cb1591f7a0516f730228126c04e5574dc7db0/vpython/filesystem/filesystem.go
[add] https://crrev.com/091cb1591f7a0516f730228126c04e5574dc7db0/vpython/filesystem/filesystem_test.go
[add] https://crrev.com/091cb1591f7a0516f730228126c04e5574dc7db0/vpython/filesystem/remove_all.go
[add] https://crrev.com/091cb1591f7a0516f730228126c04e5574dc7db0/vpython/filesystem/testfs/build.go
[add] https://crrev.com/091cb1591f7a0516f730228126c04e5574dc7db0/vpython/filesystem/testfs/tempdir.go

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/d225f829fb168a0fe57b96206a6210b4f728c9e9

commit d225f829fb168a0fe57b96206a6210b4f728c9e9
Author: dnj <dnj@chromium.org>
Date: Wed Feb 22 07:28:11 2017

vpython: Add wheel parsing/management package.

Add the "wheel" package to "vpython". This package allows "vpython" to
examine and understand the Python wheel file naming conventions, and
manipulate them on the filesystem.

BUG= chromium:691818 
TEST=unit

Review-Url: https://codereview.chromium.org/2700273002

[add] https://crrev.com/d225f829fb168a0fe57b96206a6210b4f728c9e9/vpython/wheel/wheel.go
[add] https://crrev.com/d225f829fb168a0fe57b96206a6210b4f728c9e9/vpython/wheel/wheel_test.go

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/fa008f62168a77f43978a9da9dc6aa22de82bbc5

commit fa008f62168a77f43978a9da9dc6aa22de82bbc5
Author: dnj <dnj@chromium.org>
Date: Wed Feb 22 07:43:52 2017

vpython: Add environment spec package.

This package is used by "vpython" to locate, load, and manipulate the
environment specification.

The specification can be loaded in numerous ways (detailed in README):
- From an arbitrary external file on the disk.
- From a file on disk alongside a script entry point (test.py /
  test.py.vpython).
- Inline, in an entry point script.

BUG= chromium:691818 
TEST=unit

Review-Url: https://codereview.chromium.org/2705623003

[rename] https://crrev.com/fa008f62168a77f43978a9da9dc6aa22de82bbc5/vpython/api/vpython/gen.go
[rename] https://crrev.com/fa008f62168a77f43978a9da9dc6aa22de82bbc5/vpython/api/vpython/spec.pb.go
[rename] https://crrev.com/fa008f62168a77f43978a9da9dc6aa22de82bbc5/vpython/api/vpython/spec.proto
[add] https://crrev.com/fa008f62168a77f43978a9da9dc6aa22de82bbc5/vpython/spec/load.go
[add] https://crrev.com/fa008f62168a77f43978a9da9dc6aa22de82bbc5/vpython/spec/load_test.go
[add] https://crrev.com/fa008f62168a77f43978a9da9dc6aa22de82bbc5/vpython/spec/spec.go
[add] https://crrev.com/fa008f62168a77f43978a9da9dc6aa22de82bbc5/vpython/spec/spec_test.go

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/7e123ef96d5a134b8864271e2db68bcbad3e0036

commit 7e123ef96d5a134b8864271e2db68bcbad3e0036
Author: dnj <dnj@chromium.org>
Date: Wed Feb 22 07:49:38 2017

vpython: Add Python interpreter handling package.

Add a package to "vpython" to locate and manipulate the system Python
interpreter. This includes location, version resolution, and invocation
functionality.

This package includes opportunistic tests, which will run and assert
based on which interpreter(s) are currently installed on the system.

BUG= chromium:691818 
TEST=unit

Review-Url: https://codereview.chromium.org/2701073002

[add] https://crrev.com/7e123ef96d5a134b8864271e2db68bcbad3e0036/vpython/python/find.go
[add] https://crrev.com/7e123ef96d5a134b8864271e2db68bcbad3e0036/vpython/python/interpreter.go
[add] https://crrev.com/7e123ef96d5a134b8864271e2db68bcbad3e0036/vpython/python/python.go
[add] https://crrev.com/7e123ef96d5a134b8864271e2db68bcbad3e0036/vpython/python/python_test.go
[add] https://crrev.com/7e123ef96d5a134b8864271e2db68bcbad3e0036/vpython/python/version.go
[add] https://crrev.com/7e123ef96d5a134b8864271e2db68bcbad3e0036/vpython/python/version_test.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/7533dc35e3a265fbee9fa5fc4c3552a0de29e59a

commit 7533dc35e3a265fbee9fa5fc4c3552a0de29e59a
Author: Dan Jacques <dnj@chromium.org>
Date: Sat Mar 11 03:16:55 2017

Add "gofslock", bump Go deps.

Add "github.com/danjacques/gofslock" filesystem locking package for
"vpython". Also bump Go deps.

TBR=iannucci@chromium.org
BUG= chromium:691818 
TEST=None

Change-Id: Icfb0080aac1745b7cf687e25ecf8bdabcd9a30f8
Reviewed-on: https://chromium-review.googlesource.com/452866
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/7533dc35e3a265fbee9fa5fc4c3552a0de29e59a/go/deps.lock
[modify] https://crrev.com/7533dc35e3a265fbee9fa5fc4c3552a0de29e59a/go/deps.yaml
[modify] https://crrev.com/7533dc35e3a265fbee9fa5fc4c3552a0de29e59a/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/af075cbfb85dba33b2fc91881e530700e3d74e46

commit af075cbfb85dba33b2fc91881e530700e3d74e46
Author: dnj <dnj@chromium.org>
Date: Sat Mar 11 07:07:44 2017

vpython: Add VirtualEnv creation package.

Add the "venv" package, which is responsible for creating and managing
collections of specified VirtualEnv environments.

This includes the ability to resolve, create, load, and manage (pruning)
an enviornment.

This package includes unit tests which assert the correctness of the
environment. These, in turn, leverage a few binaries (sadface). However,
the binaries are small, and this seems like a reasonable thing to do
given the subject under test.

BUG= chromium:691818 
TEST=unit

Review-Url: https://codereview.chromium.org/2699063004

[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/.gitignore
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/config.go
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/prune.go
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/system_posix.go
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/system_windows.go
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/test_data/README.md
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/test_data/pants.src/.gitignore
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/test_data/pants.src/pants/__init__.py
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/test_data/pants.src/pants/pants.py
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/test_data/pants.src/setup.cfg
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/test_data/pants.src/setup.py
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/test_data/setup_check.py
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/test_data/shirt.src/.gitignore
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/test_data/shirt.src/setup.cfg
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/test_data/shirt.src/setup.py
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/test_data/shirt.src/shirt/__init__.py
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/test_data/shirt.src/shirt/pants.py
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/venv.go
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/venv_resources_test.go
[add] https://crrev.com/af075cbfb85dba33b2fc91881e530700e3d74e46/vpython/venv/venv_test.go

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/ddf90529577a925f0c6720bc71238f18a0184611

commit ddf90529577a925f0c6720bc71238f18a0184611
Author: dnj <dnj@chromium.org>
Date: Sat Mar 11 17:00:20 2017

vpython: Add CIPD support.

Add a VirtualEnv PackageLoader implementation built on CIPD. This will
be installed by the main "vpython" executable and used exclusively in
the Chrome Operations build of "vpython".

The CIPD installation is baked into "vpython", and uses (in-memory)
ensure file format to resolve packages by name.

BUG= chromium:691818 
TEST=None

Review-Url: https://codereview.chromium.org/2699223002

[modify] https://crrev.com/ddf90529577a925f0c6720bc71238f18a0184611/vpython/README.md
[modify] https://crrev.com/ddf90529577a925f0c6720bc71238f18a0184611/vpython/api/vpython/spec.pb.go
[modify] https://crrev.com/ddf90529577a925f0c6720bc71238f18a0184611/vpython/api/vpython/spec.proto
[add] https://crrev.com/ddf90529577a925f0c6720bc71238f18a0184611/vpython/cipd/cipd.go
[modify] https://crrev.com/ddf90529577a925f0c6720bc71238f18a0184611/vpython/spec/load_test.go
[modify] https://crrev.com/ddf90529577a925f0c6720bc71238f18a0184611/vpython/spec/spec.go
[modify] https://crrev.com/ddf90529577a925f0c6720bc71238f18a0184611/vpython/spec/spec_test.go
[modify] https://crrev.com/ddf90529577a925f0c6720bc71238f18a0184611/vpython/venv/venv_resources_test.go
[modify] https://crrev.com/ddf90529577a925f0c6720bc71238f18a0184611/vpython/venv/venv_test.go

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/e8e42efd50792c3df5e499cd6d1859f8943b6c16

commit e8e42efd50792c3df5e499cd6d1859f8943b6c16
Author: dnj <dnj@chromium.org>
Date: Sat Mar 11 18:46:04 2017

Disable flaky test.

vpython/venv's TestVirtualEnv seems to be consistently flaky on builders.
Disable it until this can be resolved.

TBR=iannucci@chromium.org
BUG= chromium:691818 
TEST=None

Review-Url: https://codereview.chromium.org/2743523006

[modify] https://crrev.com/e8e42efd50792c3df5e499cd6d1859f8943b6c16/vpython/venv/venv_test.go

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/d98a0c6fa24f79922e5695de8905bd0f238c2948

commit d98a0c6fa24f79922e5695de8905bd0f238c2948
Author: dnj <dnj@chromium.org>
Date: Sat Mar 11 18:58:27 2017

vpython: Add primary execution package.

Add the primary "vpython" execution package, which exports a set of
Options and a Run method. This implements the primary "vpython"
Python bootstrapping behavior.

BUG= chromium:691818 
TEST=None

Review-Url: https://codereview.chromium.org/2702873002

[add] https://crrev.com/d98a0c6fa24f79922e5695de8905bd0f238c2948/vpython/doc.go
[add] https://crrev.com/d98a0c6fa24f79922e5695de8905bd0f238c2948/vpython/options.go
[modify] https://crrev.com/d98a0c6fa24f79922e5695de8905bd0f238c2948/vpython/python/interpreter.go
[modify] https://crrev.com/d98a0c6fa24f79922e5695de8905bd0f238c2948/vpython/python/python.go
[modify] https://crrev.com/d98a0c6fa24f79922e5695de8905bd0f238c2948/vpython/python/python_test.go
[add] https://crrev.com/d98a0c6fa24f79922e5695de8905bd0f238c2948/vpython/run.go
[add] https://crrev.com/d98a0c6fa24f79922e5695de8905bd0f238c2948/vpython/system_posix.go
[add] https://crrev.com/d98a0c6fa24f79922e5695de8905bd0f238c2948/vpython/system_windows.go

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/59ecc3c27fc9d5ef1e92c6eb845f928350e68d18

commit 59ecc3c27fc9d5ef1e92c6eb845f928350e68d18
Author: dnj <dnj@chromium.org>
Date: Sat Mar 11 21:35:32 2017

vpython: Add application entry point.

Add the "vpython" application entry point. This pulls all of the various
vpython packages together into a single functional binary, which will be
the main Chrome Operations "vpython" interface.

BUG= chromium:691818 
TEST=None

Review-Url: https://codereview.chromium.org/2705593003

[add] https://crrev.com/59ecc3c27fc9d5ef1e92c6eb845f928350e68d18/vpython/application/application.go
[add] https://crrev.com/59ecc3c27fc9d5ef1e92c6eb845f928350e68d18/vpython/application/subcommand_install.go
[add] https://crrev.com/59ecc3c27fc9d5ef1e92c6eb845f928350e68d18/vpython/application/support.go

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/6a55c1dc81825688ac63b65b2b14a9370de0a0b0

commit 6a55c1dc81825688ac63b65b2b14a9370de0a0b0
Author: dnj <dnj@chromium.org>
Date: Sat Mar 11 22:14:47 2017

Re-enable vpython/venv's TestVirtualEnv.

Re-enable the previously-flaky test, TestVirtualEnv. The flake is
believed to be due to the test re-using the output directory to build
testing wheel files. Now, the test builds each wheel in its own
temporary, isolating them from each other.

TBR=iannucci@chromium.org
BUG= chromium:691818 
TEST=None

Review-Url: https://codereview.chromium.org/2744933003

[modify] https://crrev.com/6a55c1dc81825688ac63b65b2b14a9370de0a0b0/vpython/venv/venv_resources_test.go
[modify] https://crrev.com/6a55c1dc81825688ac63b65b2b14a9370de0a0b0/vpython/venv/venv_test.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/d67aea3b2c2821b637534b712b53e32a79bd821c

commit d67aea3b2c2821b637534b712b53e32a79bd821c
Author: Dan Jacques <dnj@chromium.org>
Date: Sun Mar 12 00:22:46 2017

Add vpython main entry point, roll luci-go.

Add the vpython main entry point for Infra. This
configures a generic "vpython" executable for Infra
usage.

Roll "luci-go" to pick up the "vpython" patches.

Also, add a Go build script for "vpython", so it
will be built as part of Infra's continuous builder.

TBR=iannucci@chromium.org
BUG= chromium:691818 
TEST=None

Change-Id: I50d028778f201c24a19b5de7f8e24138c99b1fe1
Reviewed-on: https://chromium-review.googlesource.com/444040
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[add] https://crrev.com/d67aea3b2c2821b637534b712b53e32a79bd821c/go/src/infra/tools/vpython/main.go
[add] https://crrev.com/d67aea3b2c2821b637534b712b53e32a79bd821c/build/packages/vpython.yaml
[modify] https://crrev.com/d67aea3b2c2821b637534b712b53e32a79bd821c/DEPS
[add] https://crrev.com/d67aea3b2c2821b637534b712b53e32a79bd821c/go/src/infra/tools/vpython/vpython.infra_testing

Full CL set landed and available via CIPD! Marking this as Fixed.