So this is not a vpython issue. "vpython" isn't invoked here at all AFAICT; it's just one of the environment variables that is dumped.

I've seen similar things with infra/ENV when ENV was built with one version of Python (/bundle) and another version is rolled out, but ENV is not rebuilt? Otherwise, this is pretty bizarre.

"vpython" solves this by hashing the Python interpreter and its path and storing those alongside the VirtualEnv, then rebuilding it if those change. Should probably add this to "runhooks" too.

Tryjobs are supposed to rebuild infra/ENV from scratch in runhooks stage :-/

"json" is also a Python module, not a builtin, so an import failure here suggests ENV corruption.

Or sys.path corruption I suppose...

I have a theory. I think we get broken runs when we hit a warm builder cache directory on Swarming. Looks like virtualenv somehow is not compatible with our caching implementation.

Here's log of "runhooks" of a healthy run: https://luci-logdog.appspot.com/v/?s=infra%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8967746586697383296%2F%2B%2Fsteps%2Fgclient_runhooks%2F0%2Fstdout

(Notice how it has a lot of stuff)

Here's log of "runhooks" of a broken run: https://luci-logdog.appspot.com/v/?s=infra%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8967784217932228464%2F%2B%2Fsteps%2Fgclient_runhooks%2F0%2Fstdout

(Notice how it skips everything).

Hmm, makes me think it's #1/2 then.

Propose: https://chromium-review.googlesource.com/#/c/infra/infra/+/679055

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/e499b5005fa92f18f2ab1e3ffbafdbfd8d3f81f7

commit e499b5005fa92f18f2ab1e3ffbafdbfd8d3f81f7
Author: Dan Jacques <dnj@chromium.org>
Date: Fri Sep 22 20:29:33 2017

Incorporate interpreter into manifest.

The "bootstrap.py" VirtualEnv bootstrap script generates a manifest of
its contents derived solely from the input packages. However,
VirtualEnv are additionally sensitive to the Python interpreter that
they are installed from.

Expand the manifest to include the path (hard-coded by VirtualEnv) and
hash of the Python interpreter into the manifest. This will cause a
VirtualEnv generated by a different interpreter to be properly
regenerated.

This should also allow us to remove a hack to hard-code Windows Python
location versatility into the VirtualEnv. If Windows Python path
changes, it will now regenerate.

BUG= chromium:767750 
TEST=local

Change-Id: I96800136589244e828af1c1ffae1bef72a3ca1fe
Reviewed-on: https://chromium-review.googlesource.com/679055
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/e499b5005fa92f18f2ab1e3ffbafdbfd8d3f81f7/bootstrap/util.py
[modify] https://crrev.com/e499b5005fa92f18f2ab1e3ffbafdbfd8d3f81f7/bootstrap/bootstrap.py

This started working again, good sign?
 https://uberchromegw.corp.google.com/i/chromium.infra/builders/luci-go-precise32/

So far several builders that were consistently failing aren't now. I think this might be fixed?