I mentioned this up above, but to make vpython a noop, set 

  $VPYTHON_BYPASS="manually managed python not supported by chrome operations"

Otherwise you can nuke ~/.vpython-root (or whatever you have set $VPYTHON_VIRTUALENV_ROOT to) to have it regenerate the virtualenv.

We'll have to investigate how to make the hashing more reliable; ideally virtualenv itself would be able to identify this and report, but as usual the tools in the python community tend to be, uh... "best effort".

> if the venv caching logic is this fragile

VirtualEnv *itself* is that fragile; it copies/mutates the seed Python binary when it creates the VirtualEnv, and references (by path) that Python's associated libraries.

vpython tries to offset some of this fragility by fingerprinting things that it can efficiently identify/fingerprint, but identifying/fingerprinting the full system Python library set is (AFAIK) non-trivial. vpython uses a combination of the Python binary's (a) path, (b) content hash, and (c) reported version string.

When Gentoo (or anyone else) changes a Python distribution, it should change its version string, right? If it does this, then vpython will identify it as a different Python distribution and generate a different VirtualEnv for it.

> is there a command we can add to the builders to wipe the venv cache at every run ?

From the logs, it looks like the environment is being generated here: 
/build/elm/tmp/portage/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1/homedir/.vpython-root

You can ensure that a clean environment is used each run by removing this directory (note that many files are 0400, so use something that stomps write permission bits). Alternatively, you can use a temporary directory as the vpython root by setting the "VPYTHON_VIRTUALENV_ROOT" environment variable.

(to be clear, normal virtualenv doesn't do any sort of interpreter verification, it just breaks. We tried to prevent this in vpython by hashing the interpreter which seems to work on mac and debian, but is apparently insufficient here. The other libraries should be able to change without harm; the problem is that virtualenv copypasta's the python interpreter binary into the virtualenv folder)

Looks like we have a solution for the interpreter hash on gentoo btw, working on that in parallel.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/043220934aed46e9514844a3a1332792ad8b5254

commit 043220934aed46e9514844a3a1332792ad8b5254
Author: Dan Jacques <dnj@chromium.org>
Date: Tue Feb 06 02:09:36 2018

[vpython] probe and resolve Python runtime

Have Python self-report its executable and prefix when being
fingerprinted. This supports a case where a binary wrapper launches
Python, as it does on Gentoo. It also generates a fingerprint that is
more closely aligned with the parameters used by VirtualEnv.

BUG= chromium:808434 
TEST=unit,local
  - Ran locally, seems to work.

Change-Id: Ie17612bb794dc102abff11f9ff84d59b3e069e01
Reviewed-on: https://chromium-review.googlesource.com/903285
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/043220934aed46e9514844a3a1332792ad8b5254/vpython/api/vpython/env.pb.go
[modify] https://crrev.com/043220934aed46e9514844a3a1332792ad8b5254/vpython/api/vpython/env.proto
[modify] https://crrev.com/043220934aed46e9514844a3a1332792ad8b5254/vpython/python/interpreter.go
[modify] https://crrev.com/043220934aed46e9514844a3a1332792ad8b5254/vpython/python/interpreter_test.go
[modify] https://crrev.com/043220934aed46e9514844a3a1332792ad8b5254/vpython/venv/config.go
[modify] https://crrev.com/043220934aed46e9514844a3a1332792ad8b5254/vpython/venv/venv_test.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/5a87eb15986b6714d3d79c847ae48a2086998756

commit 5a87eb15986b6714d3d79c847ae48a2086998756
Author: Dan Jacques <dnj@chromium.org>
Date: Tue Feb 06 02:28:09 2018

Roll luci-go and luci/gae.

infra/go/src/go.chromium.org/luci:
04322093 [vpython] probe and resolve Python runtime

infra/go/src/go.chromium.org/gae:
e5d16d0 Add OWNERS and tweak contributing instructions.

TBR=iannucci@chromium.org
BUG= chromium:808434 
TEST=None

Change-Id: I6c94caebbf503bffdda19314c1bf2e7a221538a3
Reviewed-on: https://chromium-review.googlesource.com/903463
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/5a87eb15986b6714d3d79c847ae48a2086998756/DEPS

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/14d2e818fd337b488e3c95a1035a201d6bc7d9a5

commit 14d2e818fd337b488e3c95a1035a201d6bc7d9a5
Author: Robert Iannucci <iannucci@chromium.org>
Date: Tue Feb 06 19:23:14 2018

it doesn't look like a stale cache is the issue.  if the paths are all under e.g. /build/elm/tmp/portage/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1/homedir/.vpython-root, then those are recreated on each build.

Vincent pointed out the interesting note that only arm based builds are failing.  which makes me wonder whether there's some conflict with the re-rooting of the python interp and the cross-compiling logic we have.

in the mean time, i've posted a CL to disable the venv logic entirely:
  https://chromium-review.googlesource.com/905105
i think all the python modules depot_tools tries to install we already offer, so there shouldn't really be a need to venv everything here.

that lines up with my understanding; if 'python' changes meaning, but the binary thunk doesn't change, then virtualenv is going to fail. The vpython change above identifies sys.executable for the python that actually gets loaded, then hashes that. dnj verified that this can see through the `python` executable on $PATH.

virtualenv copypasta's "sys.executable" into the folder that it creates and (I think) does some modification to it to adjust where it finds the python 'prefix'. If that executable mismatches with the system libraries that are linked into it, then it'll lead to missing builtin modules.

with the change above, vpython should identify that 'python' means something different between the two environments, and create a second environment that works with the currently-active python.

(*verified using a gentoo live CD)

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/3f886b0c376829076c348e2274a2a8aada746e77

commit 3f886b0c376829076c348e2274a2a8aada746e77
Author: Robert Iannucci <iannucci@chromium.org>
Date: Tue Feb 06 22:36:36 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/08419509d88a19016c3bab06ca2537a3aa648367

commit 08419509d88a19016c3bab06ca2537a3aa648367
Author: Robert Iannucci <iannucci@chromium.org>
Date: Tue Feb 06 22:52:04 2018

[vpython] Bump vpython in staging.

a2eb2e90f [Findit] Flake Analyzer - Run more try jobs.
9a3bd613e Roll recipe dependencies (trivial).
42f912526 Roll recipe dependencies (trivial).
66acb1cd1 Update path in example input in GitFileIsolator test
18b65029f Add a note to the GitRef comment in Tricium AnalyzeRequest
78bd9ceaf [Tricium] Proofread and reformat comments
11d51e0f0 Remove temporary code used to add https to hosts
b6876f848 Fix two minor spelling issues in isolate.go
9aeb8d71f Roll recipe dependencies (trivial).
fffcbaf25 Roll recipe dependencies (trivial).
2d91b6863 [Findit] Instrument bug filing to use CustomFields instead of labels
5a87eb159 Roll luci-go and luci/gae.
37175a98c Roll recipe dependencies (trivial).
965cb361b Revert "Use git.exe and python.exe from %PATH% on Windows"
3ca5b0adb Roll luci-go DEPS
0ae3fcdc2 [test-results] Parse build_id form parameters from uploads.
dd5056295 Roll recipe dependencies (trivial).
896ada564 Roll recipe dependencies (trivial).
71ed7ee94 [som] Add monitoring metrics for alert *groups*
549b91906 Make submodules respect 'condition' for inclusion.
f39b717ce Roll recipe dependencies (trivial).
19eb6271e Roll recipe dependencies (trivial).
75aa8fc6a [Findit] Move swarming related logic only for flake analyzer to a separate module.
6429566ed [Findit] Translate models to BQ schema for test flake & compile failure
8d1731533 Roll recipe dependencies (trivial).
7c8adc760 Roll recipe dependencies (trivial).
a1844f202 [Monorail] Refine the order that we peel off autolinking trailing delimeters.
5409e9f46 Roll recipe dependencies (trivial).
c69c8d781 bot_setup: do not initialize swarming client on multibots
66388a79d Roll recipe dependencies (trivial).
61163aa41 Roll recipe dependencies (trivial).
4c5009d2a Roll recipe dependencies (trivial).
03f5cac69 Use system python for preallocated Swarming bots.
3d0fe245d Roll recipe dependencies (trivial).
0c35aa4a3 [Findit] miscellaneous bug fixes
1cb856336 Roll recipe dependencies (trivial).
f1e5e0076 [doc] Document permissions for cross-project joins
bb4241f99 [Monorail] Fix typo in SQL declarations.
b8e22ed36 Roll recipe dependencies (trivial).
7f8d1f1ab [buildbucket] make timing metrics cumulative
1d7d9ffa0 Roll recipe dependencies (trivial).
6afb9a8c8 Roll recipe dependencies (trivial).
bd4151e59 Roll recipe dependencies (trivial).
d450c3faf Roll recipe dependencies (trivial).
783bc9322 Roll recipe dependencies (trivial).
f2770a571 Roll recipe dependencies (trivial).
e9b8e600b Roll recipe dependencies (trivial).
03db6cde2 [Findit] move all gtest specific logic to services/gtest.py
c4884b963 Use GCE service account when fetching Swarming bot code.
a4f61f59e Roll recipe dependencies (trivial).
1ebe658ce Gen some ssh keys on cros container startup.
382263cf8 Roll recipe dependencies (trivial).
26b8b693e Point swarm*-c7 to chrome-swarming.appspot.com
d0b138cd0 [Predator] Strip parentheses in function name when matching with crash_state.
b35020f73 Allow force-version and migrate-to to accept versions greater than 9
2f0a1df32 Roll recipe dependencies (trivial).
5c77e779e Roll recipe dependencies (trivial).
6cb457523 [Monorail] Add instructions for running one unit test
64291222f Roll recipe dependencies (trivial).
e50870d52 [Monorail] Fix presubmit JS incompatibility
7d8342ee3 Roll luci-go DEPS.
afaeeb0fc [Tricium] Include notify: NONE when posting to Gerrit
a2985c216 [Findit] Refactoring BigQuery schema for analysis events
f8272181b [Monorail] Make autocomplete dropdown accessible for ChromeVox users.
d31763ca4 [Findit] Add bigquery helper that interacts with BQ REST API
d4dc0c29a mac_toolchain: add support for legacy Xcode 8.*
a304231ca Roll recipe dependencies (trivial).
6ec369965 Roll recipe dependencies (trivial).
4d60ec8f8 Roll recipe dependencies (trivial).
07e3fd5ed Roll recipe dependencies (trivial).
390212979 [Findit] Move gtest related logic to gtest.py (1/2)
d5e83d77d Roll recipe dependencies (trivial).
5df250555 Generalize android_docker image building for CrOS as well.
ed978bda5 [buildbucket] use swarming's timestamps
a14f4bfb9 Roll recipe dependencies (trivial).
68f043ea7 Roll recipe dependencies (trivial).
a926b8f75 [Monorail] Autolink text in bigger issues.
63a39573b [som] Update RELNOTES.md
3ae74f454 Roll recipe dependencies (trivial).
4d5af48ff Roll recipe dependencies (trivial).
9d0c9b8b3 Roll recipe dependencies (trivial).
cd4db1329 Add approval_id to FieldDef table.
3ad142a2c [som] Increase client RPC timeouts to 1 minute
eaaa18b94 Roll recipe dependencies (trivial).
09b3ab3ff Roll recipe dependencies (trivial).
64bcfd437 Roll recipe dependencies (trivial).
2faa1f2ac Roll recipe dependencies (trivial).
2e0feb96a Roll recipe dependencies (trivial).
86e40f0c0 Bootstrap gn with clang
9c305f17f [Findit] Ignore IntelliJ project files
17bd7606d [bbroll] Fix other link :)
94c06b37a Add IssueSnapshot table SQL schema and join tables
b0f8ad63f [Findit] Flake Analyzer - New DetermineApproximatePassRatePipeline
89dfd42af Roll recipe dependencies (trivial).
00ac3227d Roll recipe dependencies (trivial).
aab58f01d Roll recipe dependencies (trivial).
c643aedfe [bbroll] Fix canary links.
52f211da7 Monorail: Allow specifying one label in an email address for ticket alerts.
dd7df86a2 [bq events] Update docs with comments about SoM and test-results tables.

R=nodir@chromium.org, vadimsh@chromium.org

Bug:  808434 
Change-Id: I1d99f296d2b518a92ab58e19fd0556b2c8a5f0f6
Reviewed-on: https://chromium-review.googlesource.com/905467
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/08419509d88a19016c3bab06ca2537a3aa648367/scripts/slave/cipd_bootstrap_v2.py

unfortunately disabling venv didn't work.  it just produced a weirder error message:
________ running 'vpython -vpython-spec src/.vpython -vpython-tool install' in '/home/chrome-bot/chrome_root'
/usr/bin/python2.7: can't open file 'install': [Errno 2] No such file or directory
Error: Command 'vpython -vpython-spec src/.vpython -vpython-tool install' returned non-zero exit status 2 in /home/chrome-bot/chrome_root

this is from running `gclient runhooks --force`.

this one i can reproduce locally.  annoying the original failure this bug raises doesn't reproduce locally w/arm boards.

Ick... -vpython-tool means "this is a vpython specific subcommand". Looks like we still "pass that through" to python when vpython is disabled. This should be a noop instead.

That said, I'm about to roll out vpython with the better interpreter detection logic, so that might fix it.

I'll also fix vpython to turn these sorts of commands into noops.

is there a know "next step" for this problem?

This is already a P0, but I have to say that this is blocking us from delivering 2 compilers with needed security patches. 

The next step is I'm going to roll out the new version of vpython in the next couple minutes, which I think should fix this.

Bummed that we didn't get to try disabling vpython until right now (as I suggested in #26).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/1bb1f69174ea70835525a1ec5caea66d23501274

commit 1bb1f69174ea70835525a1ec5caea66d23501274
Author: Robert Iannucci <iannucci@chromium.org>
Date: Tue Feb 06 23:53:45 2018

[vpython] Bump vpython in prod.

a2eb2e90f [Findit] Flake Analyzer - Run more try jobs.
9a3bd613e Roll recipe dependencies (trivial).
42f912526 Roll recipe dependencies (trivial).
66acb1cd1 Update path in example input in GitFileIsolator test
18b65029f Add a note to the GitRef comment in Tricium AnalyzeRequest
78bd9ceaf [Tricium] Proofread and reformat comments
11d51e0f0 Remove temporary code used to add https to hosts
b6876f848 Fix two minor spelling issues in isolate.go
9aeb8d71f Roll recipe dependencies (trivial).
fffcbaf25 Roll recipe dependencies (trivial).
2d91b6863 [Findit] Instrument bug filing to use CustomFields instead of labels
5a87eb159 Roll luci-go and luci/gae.
37175a98c Roll recipe dependencies (trivial).
965cb361b Revert "Use git.exe and python.exe from %PATH% on Windows"
3ca5b0adb Roll luci-go DEPS
0ae3fcdc2 [test-results] Parse build_id form parameters from uploads.
dd5056295 Roll recipe dependencies (trivial).
896ada564 Roll recipe dependencies (trivial).
71ed7ee94 [som] Add monitoring metrics for alert *groups*
549b91906 Make submodules respect 'condition' for inclusion.
f39b717ce Roll recipe dependencies (trivial).
19eb6271e Roll recipe dependencies (trivial).
75aa8fc6a [Findit] Move swarming related logic only for flake analyzer to a separate module.
6429566ed [Findit] Translate models to BQ schema for test flake & compile failure
8d1731533 Roll recipe dependencies (trivial).
7c8adc760 Roll recipe dependencies (trivial).
a1844f202 [Monorail] Refine the order that we peel off autolinking trailing delimeters.
5409e9f46 Roll recipe dependencies (trivial).
c69c8d781 bot_setup: do not initialize swarming client on multibots
66388a79d Roll recipe dependencies (trivial).
61163aa41 Roll recipe dependencies (trivial).
4c5009d2a Roll recipe dependencies (trivial).
03f5cac69 Use system python for preallocated Swarming bots.
3d0fe245d Roll recipe dependencies (trivial).
0c35aa4a3 [Findit] miscellaneous bug fixes
1cb856336 Roll recipe dependencies (trivial).
f1e5e0076 [doc] Document permissions for cross-project joins
bb4241f99 [Monorail] Fix typo in SQL declarations.
b8e22ed36 Roll recipe dependencies (trivial).
7f8d1f1ab [buildbucket] make timing metrics cumulative
1d7d9ffa0 Roll recipe dependencies (trivial).
6afb9a8c8 Roll recipe dependencies (trivial).
bd4151e59 Roll recipe dependencies (trivial).
d450c3faf Roll recipe dependencies (trivial).
783bc9322 Roll recipe dependencies (trivial).
f2770a571 Roll recipe dependencies (trivial).
e9b8e600b Roll recipe dependencies (trivial).
03db6cde2 [Findit] move all gtest specific logic to services/gtest.py
c4884b963 Use GCE service account when fetching Swarming bot code.
a4f61f59e Roll recipe dependencies (trivial).
1ebe658ce Gen some ssh keys on cros container startup.
382263cf8 Roll recipe dependencies (trivial).
26b8b693e Point swarm*-c7 to chrome-swarming.appspot.com
d0b138cd0 [Predator] Strip parentheses in function name when matching with crash_state.
b35020f73 Allow force-version and migrate-to to accept versions greater than 9
2f0a1df32 Roll recipe dependencies (trivial).
5c77e779e Roll recipe dependencies (trivial).
6cb457523 [Monorail] Add instructions for running one unit test
64291222f Roll recipe dependencies (trivial).
e50870d52 [Monorail] Fix presubmit JS incompatibility
7d8342ee3 Roll luci-go DEPS.
afaeeb0fc [Tricium] Include notify: NONE when posting to Gerrit
a2985c216 [Findit] Refactoring BigQuery schema for analysis events
f8272181b [Monorail] Make autocomplete dropdown accessible for ChromeVox users.
d31763ca4 [Findit] Add bigquery helper that interacts with BQ REST API
d4dc0c29a mac_toolchain: add support for legacy Xcode 8.*
a304231ca Roll recipe dependencies (trivial).
6ec369965 Roll recipe dependencies (trivial).
4d60ec8f8 Roll recipe dependencies (trivial).
07e3fd5ed Roll recipe dependencies (trivial).
390212979 [Findit] Move gtest related logic to gtest.py (1/2)
d5e83d77d Roll recipe dependencies (trivial).
5df250555 Generalize android_docker image building for CrOS as well.
ed978bda5 [buildbucket] use swarming's timestamps
a14f4bfb9 Roll recipe dependencies (trivial).
68f043ea7 Roll recipe dependencies (trivial).
a926b8f75 [Monorail] Autolink text in bigger issues.
63a39573b [som] Update RELNOTES.md
3ae74f454 Roll recipe dependencies (trivial).
4d5af48ff Roll recipe dependencies (trivial).
9d0c9b8b3 Roll recipe dependencies (trivial).
cd4db1329 Add approval_id to FieldDef table.
3ad142a2c [som] Increase client RPC timeouts to 1 minute
eaaa18b94 Roll recipe dependencies (trivial).
09b3ab3ff Roll recipe dependencies (trivial).
64bcfd437 Roll recipe dependencies (trivial).
2faa1f2ac Roll recipe dependencies (trivial).
2e0feb96a Roll recipe dependencies (trivial).
86e40f0c0 Bootstrap gn with clang
9c305f17f [Findit] Ignore IntelliJ project files
17bd7606d [bbroll] Fix other link :)
94c06b37a Add IssueSnapshot table SQL schema and join tables
b0f8ad63f [Findit] Flake Analyzer - New DetermineApproximatePassRatePipeline
89dfd42af Roll recipe dependencies (trivial).
00ac3227d Roll recipe dependencies (trivial).
aab58f01d Roll recipe dependencies (trivial).
c643aedfe [bbroll] Fix canary links.
52f211da7 Monorail: Allow specifying one label in an email address for ticket alerts.
dd7df86a2 [bq events] Update docs with comments about SoM and test-results tables.

R=nodir@chromium.org, vadimsh@chromium.org

Bug:  808434 
Change-Id: I012f13335f1573e8ae60ea1ae2e79dbc94f45f05
Reviewed-on: https://chromium-review.googlesource.com/905630
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/1bb1f69174ea70835525a1ec5caea66d23501274/scripts/slave/cipd_bootstrap_v2.py
[modify] https://crrev.com/1bb1f69174ea70835525a1ec5caea66d23501274/scripts/slave/recipe_modules/swarming/api.py

Please try this build again

(although... maybe we need to revert the envvar change first now, assuming it landed)

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/521e1f1754bcb7ec60ee86cf8cb703eac62d82ab

commit 521e1f1754bcb7ec60ee86cf8cb703eac62d82ab
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Feb 07 00:03:57 2018

[vpython] Make vpython subcommands noops in bypass mode.

R=nodir@chromium.org, tandrii@chromium.org, vadimsh@chromium.org

Bug:  808434 
Change-Id: Id4ab9aa694b50ade629aea966ab9eb9bdb92451c
Reviewed-on: https://chromium-review.googlesource.com/905632
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/521e1f1754bcb7ec60ee86cf8cb703eac62d82ab/vpython/application/application.go

(I'm also going to roll out the change to make vpython subcommands correctly bypass too, if you'd rather keep the envvar and wait for that)

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/58cd44f906b35a11d9c516047f28fb17c1de4e11

commit 58cd44f906b35a11d9c516047f28fb17c1de4e11
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Feb 07 00:26:14 2018

Roll luci-go.

infra/go/src/go.chromium.org/luci:
521e1f17 [vpython] Make vpython subcommands noops in bypass mode.
e2c2e70d [Machine Database] Validate MAC and IP addresses

R=tandrii@chromium.org, vadimsh@chromium.org

Bug:  808434 
Change-Id: I2f14cc6558969f555e5205e28d3989398fcce3c6
Reviewed-on: https://chromium-review.googlesource.com/905823
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/58cd44f906b35a11d9c516047f28fb17c1de4e11/DEPS

Rolling a new vpython to fix the issue observed in #52 (I still haven't seen a new build on elm-release so I have no idea if my original change fixed it... I'm just doin' stuff since I can't trigger any builds :)).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/8b4a4d738b40e86d6b694ada09d82e6856b643fd

commit 8b4a4d738b40e86d6b694ada09d82e6856b643fd
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Feb 07 01:38:57 2018

Roll vpython in staging + prod for small bugfix.

This only actually picks up
https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/905632

f88d8d9aa [buildbucket] cancel swarming tasks when deleting builds
0c7d29eb5 Roll recipe dependencies (trivial).
58cd44f90 Roll luci-go.
b3dbb75ba Roll recipe dependencies (trivial).
57743d010 Revert "Remove temporary code used to add https to hosts"
4592a0a20 [buildbucket] refactor task cancelation
76f45a19b Roll recipe dependencies (trivial).
b2fb6be8e [som] Fix some FE unit tests.
57f45c589 [som] Update RELNOTES.md
c7bc0b942 Roll recipe dependencies (trivial).
f91ebe64a SoM: Add autosuggested labels back.
b2de6ae83 [Findit] Translate test failure to event schema
52adb6e67 Roll recipe dependencies (trivial).
efa035c1d [Monorail] Follow-up fix for presubmit browser compat fix
bf5820217 [prpc,luci-auth] Set builds to use static linking.
959f0f9bc SoM: Update ChOpsUI version.
91e92e1e6 Roll luci-go.
ebc1baca9 Roll recipe dependencies (trivial).
47b35eab2 In set review request, use patch set number
886cc525e [buildbucket] no kitchen output file => Infra failure
d73861ca3 [Predator] Increase soft memory for uma sampling profiler.
cd84bd1c5 [buildbucket] refactor build_run_result loading
011e82c92 [Monorail] Fix two issues introduced by presubmit Safari/IE fix
a004798d1 [luci-migration] Surface Infra Failures information.
3a20d2ded Roll recipe dependencies (trivial).
7d82e1b8b Roll recipe dependencies (trivial).

BUG= 808434 
R=tandrii@chromium.org, vadimsh@chromium.org

Change-Id: I372022a0c0b4a9854f013e2e935a294a94dcc5e2
Reviewed-on: https://chromium-review.googlesource.com/905867
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/8b4a4d738b40e86d6b694ada09d82e6856b643fd/scripts/slave/cipd_bootstrap_v2.py
[modify] https://crrev.com/8b4a4d738b40e86d6b694ada09d82e6856b643fd/scripts/slave/recipe_modules/swarming/api.py

Ok, so at this point we need to either: 
  * Trigger a build without $VPYTHON_BYPASS (to see if the improved sys.executable detection fixes the issue)
  * Trigger a build with $VPYTHON_BYPASS (to pick up the fix I just rolled out in #63 that should enable bypass mode to work correctly)

I just ran a tryjob, not fully sure if it picked up the changes above.

Still failing:
https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromiumos.tryserver%2Frelease%2F19801%2F%2B%2Frecipes%2Fsteps%2FBuildPackages__afdo_use_%2F0%2Fstdout

Yep that build had the latest version. Still not sure what's wrong (but the bypass thing should work correctly now)

 Issue 809866  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/4a92cc9a1f7ced74f01473c7040992a97c1a4079

commit 4a92cc9a1f7ced74f01473c7040992a97c1a4079
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Feb 07 08:15:48 2018

[vpython] Roll vpython to unblock CrOS.

This is already rolled to all buildbot machines; was waiting to roll
vpython in depot_tools last, but it appears that their builds may
actually depend on the version here.

f88d8d9aa [buildbucket] cancel swarming tasks when deleting builds
0c7d29eb5 Roll recipe dependencies (trivial).
58cd44f90 Roll luci-go.
b3dbb75ba Roll recipe dependencies (trivial).
57743d010 Revert "Remove temporary code used to add https to hosts"
4592a0a20 [buildbucket] refactor task cancelation
76f45a19b Roll recipe dependencies (trivial).
b2fb6be8e [som] Fix some FE unit tests.
57f45c589 [som] Update RELNOTES.md
c7bc0b942 Roll recipe dependencies (trivial).
f91ebe64a SoM: Add autosuggested labels back.
b2de6ae83 [Findit] Translate test failure to event schema
52adb6e67 Roll recipe dependencies (trivial).
efa035c1d [Monorail] Follow-up fix for presubmit browser compat fix
bf5820217 [prpc,luci-auth] Set builds to use static linking.
959f0f9bc SoM: Update ChOpsUI version.
91e92e1e6 Roll luci-go.
ebc1baca9 Roll recipe dependencies (trivial).
47b35eab2 In set review request, use patch set number
886cc525e [buildbucket] no kitchen output file => Infra failure
d73861ca3 [Predator] Increase soft memory for uma sampling profiler.
cd84bd1c5 [buildbucket] refactor build_run_result loading
011e82c92 [Monorail] Fix two issues introduced by presubmit Safari/IE fix
a004798d1 [luci-migration] Surface Infra Failures information.
3a20d2ded Roll recipe dependencies (trivial).
7d82e1b8b Roll recipe dependencies (trivial).
a2eb2e90f [Findit] Flake Analyzer - Run more try jobs.
9a3bd613e Roll recipe dependencies (trivial).
42f912526 Roll recipe dependencies (trivial).
66acb1cd1 Update path in example input in GitFileIsolator test
18b65029f Add a note to the GitRef comment in Tricium AnalyzeRequest
78bd9ceaf [Tricium] Proofread and reformat comments
11d51e0f0 Remove temporary code used to add https to hosts
b6876f848 Fix two minor spelling issues in isolate.go
9aeb8d71f Roll recipe dependencies (trivial).
fffcbaf25 Roll recipe dependencies (trivial).
2d91b6863 [Findit] Instrument bug filing to use CustomFields instead of labels
5a87eb159 Roll luci-go and luci/gae.
37175a98c Roll recipe dependencies (trivial).
965cb361b Revert "Use git.exe and python.exe from %PATH% on Windows"
3ca5b0adb Roll luci-go DEPS
0ae3fcdc2 [test-results] Parse build_id form parameters from uploads.
dd5056295 Roll recipe dependencies (trivial).
896ada564 Roll recipe dependencies (trivial).
71ed7ee94 [som] Add monitoring metrics for alert *groups*
549b91906 Make submodules respect 'condition' for inclusion.
f39b717ce Roll recipe dependencies (trivial).
19eb6271e Roll recipe dependencies (trivial).
75aa8fc6a [Findit] Move swarming related logic only for flake analyzer to a separate module.
6429566ed [Findit] Translate models to BQ schema for test flake & compile failure
8d1731533 Roll recipe dependencies (trivial).
7c8adc760 Roll recipe dependencies (trivial).
a1844f202 [Monorail] Refine the order that we peel off autolinking trailing delimeters.
5409e9f46 Roll recipe dependencies (trivial).
c69c8d781 bot_setup: do not initialize swarming client on multibots
66388a79d Roll recipe dependencies (trivial).
61163aa41 Roll recipe dependencies (trivial).
4c5009d2a Roll recipe dependencies (trivial).
03f5cac69 Use system python for preallocated Swarming bots.
3d0fe245d Roll recipe dependencies (trivial).
0c35aa4a3 [Findit] miscellaneous bug fixes
1cb856336 Roll recipe dependencies (trivial).
f1e5e0076 [doc] Document permissions for cross-project joins
bb4241f99 [Monorail] Fix typo in SQL declarations.
b8e22ed36 Roll recipe dependencies (trivial).
7f8d1f1ab [buildbucket] make timing metrics cumulative
1d7d9ffa0 Roll recipe dependencies (trivial).
6afb9a8c8 Roll recipe dependencies (trivial).
bd4151e59 Roll recipe dependencies (trivial).
d450c3faf Roll recipe dependencies (trivial).
783bc9322 Roll recipe dependencies (trivial).
f2770a571 Roll recipe dependencies (trivial).
e9b8e600b Roll recipe dependencies (trivial).
03db6cde2 [Findit] move all gtest specific logic to services/gtest.py
c4884b963 Use GCE service account when fetching Swarming bot code.
a4f61f59e Roll recipe dependencies (trivial).
1ebe658ce Gen some ssh keys on cros container startup.
382263cf8 Roll recipe dependencies (trivial).
26b8b693e Point swarm*-c7 to chrome-swarming.appspot.com
d0b138cd0 [Predator] Strip parentheses in function name when matching with crash_state.
b35020f73 Allow force-version and migrate-to to accept versions greater than 9
2f0a1df32 Roll recipe dependencies (trivial).
5c77e779e Roll recipe dependencies (trivial).
6cb457523 [Monorail] Add instructions for running one unit test
64291222f Roll recipe dependencies (trivial).
e50870d52 [Monorail] Fix presubmit JS incompatibility
7d8342ee3 Roll luci-go DEPS.
afaeeb0fc [Tricium] Include notify: NONE when posting to Gerrit
a2985c216 [Findit] Refactoring BigQuery schema for analysis events
f8272181b [Monorail] Make autocomplete dropdown accessible for ChromeVox users.
d31763ca4 [Findit] Add bigquery helper that interacts with BQ REST API
d4dc0c29a mac_toolchain: add support for legacy Xcode 8.*
a304231ca Roll recipe dependencies (trivial).
6ec369965 Roll recipe dependencies (trivial).
4d60ec8f8 Roll recipe dependencies (trivial).
07e3fd5ed Roll recipe dependencies (trivial).
390212979 [Findit] Move gtest related logic to gtest.py (1/2)
d5e83d77d Roll recipe dependencies (trivial).
5df250555 Generalize android_docker image building for CrOS as well.
ed978bda5 [buildbucket] use swarming's timestamps
a14f4bfb9 Roll recipe dependencies (trivial).
68f043ea7 Roll recipe dependencies (trivial).
a926b8f75 [Monorail] Autolink text in bigger issues.
63a39573b [som] Update RELNOTES.md
3ae74f454 Roll recipe dependencies (trivial).
4d5af48ff Roll recipe dependencies (trivial).
9d0c9b8b3 Roll recipe dependencies (trivial).
cd4db1329 Add approval_id to FieldDef table.
3ad142a2c [som] Increase client RPC timeouts to 1 minute
eaaa18b94 Roll recipe dependencies (trivial).
09b3ab3ff Roll recipe dependencies (trivial).
64bcfd437 Roll recipe dependencies (trivial).
2faa1f2ac Roll recipe dependencies (trivial).
2e0feb96a Roll recipe dependencies (trivial).
86e40f0c0 Bootstrap gn with clang
9c305f17f [Findit] Ignore IntelliJ project files
17bd7606d [bbroll] Fix other link :)
94c06b37a Add IssueSnapshot table SQL schema and join tables
b0f8ad63f [Findit] Flake Analyzer - New DetermineApproximatePassRatePipeline
89dfd42af Roll recipe dependencies (trivial).
00ac3227d Roll recipe dependencies (trivial).
aab58f01d Roll recipe dependencies (trivial).
c643aedfe [bbroll] Fix canary links.
52f211da7 Monorail: Allow specifying one label in an email address for ticket alerts.
dd7df86a2 [bq events] Update docs with comments about SoM and test-results tables.


TBR=vadimsh@chromium.org

Bug:  808434 
Change-Id: Id16a8758257ad0c531b0a590e923b3b4d3442397
Reviewed-on: https://chromium-review.googlesource.com/905830
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/4a92cc9a1f7ced74f01473c7040992a97c1a4079/cipd_manifest.txt

Current theory is that CrOS actually uses the pin in depot_tools to get vpython... I had been neglecting to update that pin because we (until now) assumed that it was only used by devs and so was lowest priority to roll. I was waiting until the buildbot + LUCI pins were working well to update devs last. However it appears that these builds rely on depot_tools instead of the buildbot pins! eep!

I launched a tryjob after #69 and it still failed.

Please review it is using the correct version of Chrome

https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromiumos.tryserver%2Frelease%2F19820%2F%2B%2Frecipes%2Fsteps%2FBuildPackages__afdo_use_%2F0%2Fstdout

Tried a build without bypass:
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/19817
Failed with this usual "ImportError: No module named _io"

And with bypass (https://chromium-review.googlesource.com/#/c/906171/):
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/19819
Failed with:
chromeos-chrome-66.0.3342.0_rc-r2: /usr/bin/python2.7: can't open file 'install': [Errno 2] No such file or directory
chromeos-chrome-66.0.3342.0_rc-r2: Error: Command 'vpython -vpython-spec src/.vpython -vpython-tool install' returned non-zero exit status 2 in /home/chrome-bot/chrome_root/.

vapier@ points out in #47 that that ".vpython-root" directory is clobbered each run. If this is accurate, any amount of fingerprinting logic updates, while useful, should not impact this bug.

From the error in #71:
goroutine 1:
#0 go.chromium.org/luci/vpython/venv/venv.go:577 - venv.(*Env).installVirtualEnv()
 reason: failed to create VirtualEnv
#1 go.chromium.org/luci/vpython/venv/venv.go:498 - venv.(*Env).createLocked.func1()
  reason: failed to install VirtualEnv
#2 go.chromium.org/luci/common/system/filesystem/tempdir.go:55 - filesystem.(*TempDir).With()
#3 go.chromium.org/luci/vpython/venv/venv.go:75 - venv.withTempDir()
#4 go.chromium.org/luci/vpython/venv/venv.go:474 - venv.(*Env).createLocked()
#5 go.chromium.org/luci/vpython/venv/venv.go:243 - venv.(*Env).ensure.func1()
  reason: failed to create new VirtualEnv

The version of vpython that matches those line numbers appears to be: https://chromium.googlesource.com/infra/luci/luci-go/+/b1e7bc36178785f38cbd82bef3bf536629865153/vpython
This is 4 months old, and some potentially relevant commits have landed since then. It's certainly the reason why any changes from this bug haven't had any impact!

The dates (mid-October 2017) correspond to the depot_tools pin in "chromite" recipe module, so maybe that's where it's pulling "vpython" from? https://chromium.googlesource.com/chromium/tools/build/+/96971f3d12aa121ba553e54caf54884c5709ee3b/scripts/slave/recipe_modules/chromite/api.py#16

At this point, it would be worthwhile for someone to log onto one of these bots, repro the problem, and probe. There is no *good* reason why the "_io" module should be causing problems. Making things even weirder, xiyuan@ notes in #8 that this is only happening on ARM canary bots. Perhaps their chroot's Python layout is different or broken?

From the stack trace:
chromeos-chrome-66.0.3343.0_alpha-r1:   Complete output from command /build/elm/tmp/porta...50d5a2/bin/python2.7 - setuptools pip wheel:
chromeos-chrome-66.0.3343.0_alpha-r1:   Traceback (most recent call last):
chromeos-chrome-66.0.3343.0_alpha-r1:   File "<stdin>", line 4, in <module>
chromeos-chrome-66.0.3343.0_alpha-r1:   File "/usr/lib64/python2.7/tempfile.py", line 32, in <module>
chromeos-chrome-66.0.3343.0_alpha-r1:     import io as _io
chromeos-chrome-66.0.3343.0_alpha-r1:   File "/usr/lib64/python2.7/io.py", line 51, in <module>
chromeos-chrome-66.0.3343.0_alpha-r1:     import _io
chromeos-chrome-66.0.3343.0_alpha-r1: ImportError: No module named _io

I've logged onto the bot in question, entered the chroot, and observed the "_io.so" resides in "/usr/lib64/python2.7/lib-dynload/_io.so". I've also tried running vanilla VirtualEnv, the "latest" vpython, and a "vpython" from October 15th (when this bug was happening) in the chroot and all of them seem to be working; I'm not able to repro this.

I've run Python with the vpython isolation flags and it can import "_io":
$ python -B -E -s -c 'import _io; print _io'
<module '_io' from '/usr/lib64/python2.7/lib-dynload/_io.so'>

I've run the above command from within the "cros_sdk" command at the header of the build in #71 (replacing ./build_packages), and it behaves identically.

At this point, someone should:
1) Trigger a failing build on the bot.
2) Freeze the bot so it doesn't build anything else.
3) Log onto that bot and repro manually.
4) Figure out WTF is up with "_io" :)

Oh RE #72, if vapier@ is not correct about "/build/elm/tmp/portage/chromeos-base/chromeos-chrome-66.0.3336.3_rc-r1/homedir/.vpython-root/6005b1/bin/python2.7" being deleted, then the fingerprinting changes might work.

I pointed out in #40 that "vpython" protects its VirutalEnv directories by removing the "write" permission bit from all of its files; is it possible that portage or Chromite are not correctly cleaning up because of this?

Issue 809874 has been merged into this issue.

I'm exceedingly confused as well; I believe, at the very least, I've fixed the error observed in #71... I can no longer repro it locally using the latest version of vpython (i.e. the one deployed in depot_tools here: https://chromium.googlesource.com/chromium/tools/depot_tools/+/master/cipd_manifest.txt#11 . This version is also deployed to all buildbot machines: https://chromium.googlesource.com/chromium/tools/build/+/master/scripts/slave/cipd_bootstrap_v2.py#45

That version of depot_tools was also rolled into chromium/src.git: https://chromium.googlesource.com/chromium/src/+/8f42d098e508b6ff34986976b4f30c2f476434cd

I'm very confused about how the error in comment 71 could possibly show up any more... is there some other pin of depot_tools or vpython that these bots use?

OK, new theory. Looks like cros has 2 additional pins of depot_tools (lol):

https://cs.corp.google.com/chromeos_public/manifest/full.xml?l=400
https://chromium.googlesource.com/chromium/tools/build/+/96971f3d12aa121ba553e54caf54884c5709ee3b/scripts/slave/recipe_modules/chromite/api.py#16

I'll try bumping those, though I'm not sure what order I'm supposed to bump them in because they're in different repos......

Unless someone (dgarrett? vapier?) is more familiar with how to bump these? 

They should be rolled to https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/4a92cc9a1f7ced74f01473c7040992a97c1a4079

There are two more pins of depot_tools. 

In our build recipe api, which cbuildbot uses outside of the chroot:

https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromite/api.py?rcl=909127168ec40e0e311d01c4bcdcee0f3143ef06&l=12

In our source manifest, which is the only version inside of the chroot:

https://cs.corp.google.com/chromeos_public/manifest/full.xml

They should be kept roughly in sync. I can bump them both if that's needed here.

i sent you guys some CLs.  i tested w/latest depot_tools and it's passing with & w/out the vpython bypass for me (but it was passing w/out before locally too).  so lets do this depot_tools roll, and if it's still failing, i'll send my earlier CL to another trybot to see if it helps.

i'd rather not yank vpython stuff out from under you guys as CrOS (even if we provided up-to-date deps) is already enough of a special snowflake.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/987b8691f1ed9de1e1133e23f62b29f487910e7c

commit 987b8691f1ed9de1e1133e23f62b29f487910e7c
Author: vapier@chromium.org <vapier@chromium.org>
Date: Wed Feb 07 20:12:55 2018

chromite: depot_tools roll

Recipe-Nontrivial-Roll: build_limited_scripts_slave
Bug:  808434 
Change-Id: I420ad1622e4ab9eb3666e0b901b311db23bdb70d
Reviewed-on: https://chromium-review.googlesource.com/907075
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/basic_compressed.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/swarming.expected/swarming_builder.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/swarming.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/pre_git_cache_release.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/unknown_config.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot.expected/swarming_builder.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipe_modules/chromite/api.py
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot.expected/chromiumos_paladin_buildbucket.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot.expected/chromiumos_paladin.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot.expected/chromiumos_coverage.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot.expected/chromiumos_chromium_builder.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/release_branch_one_param.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/pre_cq_buildbucket_config.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/release.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/external.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/ndk/ndk_buildbot.expected/basic.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/release_branch_two_params.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot.expected/chromiumos_coverage_variant.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/internal.json
[modify] https://crrev.com/987b8691f1ed9de1e1133e23f62b29f487910e7c/scripts/slave/recipes/cros/cbuildbot.expected/chromiumos_paladin_manifest_failure.json

awesome, looks like it's all landed now

I launched a tryjob and it failed. Not sure how to verify it is picking the right pieces.

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/19843

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/manifest/+/66526607895efc0160371689e6860e99c4184959

commit 66526607895efc0160371689e6860e99c4184959
Author: Mike Frysinger <vapier@chromium.org>
Date: Wed Feb 07 21:54:39 2018

depot_tools roll

BUG= chromium:808434 
TEST=precq passes

Change-Id: I420ad1622e4ab9eb3666e0b901b311db23bdb70d
Reviewed-on: https://chromium-review.googlesource.com/907074
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/66526607895efc0160371689e6860e99c4184959/full.xml

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/manifest-internal/+/d32ba41d7b6a5c47772216321d418c3360e55441

commit d32ba41d7b6a5c47772216321d418c3360e55441
Author: Mike Frysinger <vapier@chromium.org>
Date: Wed Feb 07 21:54:45 2018

i've kicked off chrome builds with & w/out the venv disable patch

The stack trace line numbers in #84 match those in #72, suggesting that it is not using a different version.

Retry the tryjob. The other parts of the pin bump just landed at 1:54 PM (MTV, center of the world, TZ)

i'm cautiously optimistic that ToT is fixed now.  my trybots passed BuildPackages for elm.

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/chrome_pfq_informational/builds/94
[ebuild  N    *] chromeos-base/chromeos-chrome-66.0.3343.0_alpha-r1::chromiumos to /build/elm/ ...
Started chromeos-base/chromeos-chrome-66.0.3343.0_alpha-r1 (logged in /tmp/chromeos-chrome-66.0.3343.0_alpha-r1-dNuyhj)
Completed chromeos-base/chromeos-chrome-66.0.3343.0_alpha-r1 (in 18m30.7s)

so i won't land my venv disable hack.  lets see if the chrome pfq & full & sdk bots also clear up.

Issue 809399 has been merged into this issue.

woot for cautious optimism :)

TIL: we have waaay too many depot_tools' on bots (I think at least 5 in this build?)

I launched a full build (elm-release) and it failed. 

I think it should have the appropriate commits. please review.

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/19847

my tryjob command:

cros tryjob  --pass-through=--chrome_rev=tot elm-release-tryjob

Looks like the failure in #93 matches TOT vpython, so bumping "depot_tools" did work, but the latest code did not end up fixing the problem.

I think it would be best to submit the bypass change until we understand this better

dnj and I have a deterministic repro on the bot (it looks like something is setting some $VIRTUALENV variables which leak through to vpython, $VIRTUALENV_ALWAYS_COPY=1 in particular).  Working on a fix for this as well.

iannucci@ found a bug in VirtualEnv where "VIRTUALENV_ALWAYS_COPY" produces the corrupt vpython environment. We're going to roll a vpython that clears those environment variables.

Probably: https://github.com/pypa/virtualenv/issues/565

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/62edf1462165dff3babf56517ad73140458adc13

commit 62edf1462165dff3babf56517ad73140458adc13
Author: Dan Jacques <dnj@chromium.org>
Date: Thu Feb 08 03:26:33 2018

[vpython] clear VIRTUALENV_ environment

Clear the VIRTUALENV_ environment variables when setting up VirtualEnv.
This ensures that external influences don't corrupt the VirtualEnv setup
routine.

BUG= chromium:808434 
TEST=local

Change-Id: I05a4a4b3f8e42cb2c792d6f11e42d38043563313
Reviewed-on: https://chromium-review.googlesource.com/907929
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/62edf1462165dff3babf56517ad73140458adc13/common/system/environ/environment.go
[modify] https://crrev.com/62edf1462165dff3babf56517ad73140458adc13/common/system/environ/environment_test.go
[modify] https://crrev.com/62edf1462165dff3babf56517ad73140458adc13/vpython/venv/venv.go

I launched a tryjob with bypass enabled on both 9999 and stable Chrome ebuilds, and the build seems to be fine (Chrome is building right now, which is a good sign):
https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/905105/2
https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/906171/2

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/19849

Let's wait for the tryjob to complete, but suggest we land the bypass to turn the builds green ASAP.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/dd1d2f648cf0e78e8fda8012d7952528a0c9e6ea

commit dd1d2f648cf0e78e8fda8012d7952528a0c9e6ea
Author: Robert Iannucci <iannucci@chromium.org>
Date: Thu Feb 08 03:41:46 2018

Roll luci-go

infra/go/src/go.chromium.org/luci:
62edf146 [vpython] clear VIRTUALENV_ environment

R=tandrii@chromium.org, vadimsh@chromium.org

Bug:  808434 
Change-Id: I0c85c72a2a8065787b4095bf654cb8782a000520
Reviewed-on: https://chromium-review.googlesource.com/908110
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/dd1d2f648cf0e78e8fda8012d7952528a0c9e6ea/DEPS

Bypass worked for BuildPackages/chromeos-chrome, but now we are failing UnitTest with similar error:
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/19849

ci-results-archiver-0.0.1-r1: >>> Test phase: chromeos-base/ci-results-archiver-0.0.1-r1
ci-results-archiver-0.0.1-r1: Traceback (most recent call last):
ci-results-archiver-0.0.1-r1:   File "/usr/lib/python-exec/python2.7/virtualenv", line 9, in <module>
ci-results-archiver-0.0.1-r1:     load_entry_point('virtualenv==13.1.2', 'console_scripts', 'virtualenv')()
ci-results-archiver-0.0.1-r1:   File "/usr/lib64/python2.7/site-packages/virtualenv.py", line 832, in main
ci-results-archiver-0.0.1-r1:     symlink=options.symlink)
ci-results-archiver-0.0.1-r1:   File "/usr/lib64/python2.7/site-packages/virtualenv.py", line 1004, in create_environment
ci-results-archiver-0.0.1-r1:     install_wheel(to_install, py_executable, search_dirs)
ci-results-archiver-0.0.1-r1:   File "/usr/lib64/python2.7/site-packages/virtualenv.py", line 969, in install_wheel
ci-results-archiver-0.0.1-r1:     'PIP_NO_INDEX': '1'
ci-results-archiver-0.0.1-r1:   File "/usr/lib64/python2.7/site-packages/virtualenv.py", line 910, in call_subprocess
ci-results-archiver-0.0.1-r1:     % (cmd_desc, proc.returncode))
ci-results-archiver-0.0.1-r1: OSError: Command /build/elm/tmp/porta...7d6ff7/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip wheel failed with error code 1
ci-results-archiver-0.0.1-r1: Traceback (most recent call last):
ci-results-archiver-0.0.1-r1:   File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
ci-results-archiver-0.0.1-r1:     "__main__", fname, loader, pkg_name)
ci-results-archiver-0.0.1-r1:   File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
ci-results-archiver-0.0.1-r1:     exec code in run_globals
ci-results-archiver-0.0.1-r1:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/scripts/create_venv.py", line 46, in <module>
ci-results-archiver-0.0.1-r1:     main()
ci-results-archiver-0.0.1-r1:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/scripts/create_venv.py", line 35, in main
ci-results-archiver-0.0.1-r1:     print(venv.ensure())
ci-results-archiver-0.0.1-r1:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/venvlib.py", line 100, in ensure
ci-results-archiver-0.0.1-r1:     self._check_or_create()
ci-results-archiver-0.0.1-r1:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/venvlib.py", line 108, in _check_or_create
ci-results-archiver-0.0.1-r1:     self._create()
ci-results-archiver-0.0.1-r1:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/venvlib.py", line 117, in _create
ci-results-archiver-0.0.1-r1:     logfile=logfile)
ci-results-archiver-0.0.1-r1:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/venvlib.py", line 209, in _create_venv
ci-results-archiver-0.0.1-r1:     _log_check_call(command, logfile=logfile)
ci-results-archiver-0.0.1-r1:   File "/mnt/host/source/infra_virtualenv/venv/cros_venv/venvlib.py", line 254, in wrapped_command
ci-results-archiver-0.0.1-r1:     call_func(args, stdout=logfile, **kwargs)
ci-results-archiver-0.0.1-r1:   File "/usr/lib64/python2.7/subprocess.py", line 540, in check_call
ci-results-archiver-0.0.1-r1:     raise CalledProcessError(retcode, cmd)
ci-results-archiver-0.0.1-r1: subprocess.CalledProcessError: Command '[u'virtualenv', u'/build/elm/tmp/portage/chromeos-base/ci-results-archiver-0.0.1-r1/homedir/.cache/cros_venv/venv-2.7.10-9b73359f13f2cf7b6ed8f86d2c7d6ff7', u'-p', u'/usr/bin/python2.7', u'--extra-search-dir', u'/mnt/host/source/infra_virtualenv/pip_packages', u'--setuptools', u'--clear']' returned non-zero exit status 1
ci-results-archiver-0.0.1-r1: ERROR: Failed to set up a virtualenv.

So to be clear, this is going to affect all virtualenv invocations, not just vpython. I'm not sure where 'infra_virtualenv' comes from, but it looks like a CrOS usage of virtualenv.

Still not sure what the root source/cause of this is.

I can reproduce the issue in #103 by running:
VIRTUALENV_ALWAYS_COPY=1 FEATURES=test emerge-elm -av ci-results-archiver

Suggested patch:
https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/908175/3

Tryjob running here:
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/19851

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/28ef4e7b0851c5d33e42c57de6fdfd3edf6f8e46

commit 28ef4e7b0851c5d33e42c57de6fdfd3edf6f8e46
Author: Robert Iannucci <iannucci@chromium.org>
Date: Thu Feb 08 16:02:31 2018

Roll vpython.

This only includes the change to remove all VIRTUALENV envvars when
vpython preps environments. See
  https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/907929

R=dnj@chromium.org, vadimsh@chromium.org

Bug:  808434 
Change-Id: I2f6f1b2cec342997081322884b9dde06557a0c60
Reviewed-on: https://chromium-review.googlesource.com/908058
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/28ef4e7b0851c5d33e42c57de6fdfd3edf6f8e46/cipd_manifest.txt

So the fixed version of vpython is in depot_tools (to be rolled again), but that won't fix the cros_venv error observed in #103 (which is not using vpython)

The version of depot_tools to roll to is 28ef4e7b0851c5d33e42c57de6fdfd3edf6f8e46

RE #103, I did engage the CrOS team during early vpython development to make them aware of the tool. I'll take #103 as an opportunity to re-engage: https://bugs.chromium.org/p/chromium/issues/detail?id=810460

who is rolling depot_tools? 

can we just do as suggested in #96? 
we can keep trying things to fix this "offline" but we need to get unblocked. 

RE #110, there is no "bypass" CL for the problem in #103, since bypass is a vpython capability. The vpython solution is rolling, and the CrOS-side VirtualEnv one will require something like #105.

I made a CL in the vein of the vpython fix for CrOS infra's virtualenv https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/907929

https://chromium-review.googlesource.com/c/chromiumos/infra_virtualenv/+/909532

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/4ccaa1f9565beee9675cdcc4fffabc31f45769e2

commit 4ccaa1f9565beee9675cdcc4fffabc31f45769e2
Author: Don Garrett <dgarrett@google.com>
Date: Thu Feb 08 22:23:52 2018

chromite/api: Bump depot_tools pin.

BUG= chromium:808434 

Recipe-Nontrivial-Roll: build_limited_scripts_slave
Change-Id: I5bde58e01d4fae8ec2238e1149e0b21735aa49b0
Reviewed-on: https://chromium-review.googlesource.com/909853
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot.expected/swarming_builder.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipe_modules/chromite/api.py
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/swarming.expected/tryjob_simple.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/swarming.expected/master_builder.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot.expected/chromiumos_goma_canary.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot.expected/chromiumos_paladin_buildbucket.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/pre_cq_buildbucket_config.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/release.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/external.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/release_branch_two_params.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/swarming.expected/tryjob_complex.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/swarming.expected/complex_slave_builder.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/basic_compressed.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/swarming.expected/swarming_builder.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot.expected/chromiumos_chromium_builder.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/swarming.expected/tuple_args.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot.expected/chromiumos_coverage_variant.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot.expected/chromiumos_paladin_manifest_failure.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/pre_git_cache_release.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/swarming.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/unknown_config.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot.expected/chromiumos_paladin.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/README.recipes.md
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot.expected/chromiumos_coverage.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/release_branch_one_param.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/ndk/ndk_buildbot.expected/basic.json
[modify] https://crrev.com/4ccaa1f9565beee9675cdcc4fffabc31f45769e2/scripts/slave/recipes/cros/cbuildbot_tryjob.expected/internal.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/manifest/+/5d19cea31debf4e9cb93f3c21a8d788a0611b898

commit 5d19cea31debf4e9cb93f3c21a8d788a0611b898
Author: Don Garrett <dgarrett@google.com>
Date: Thu Feb 08 22:52:46 2018

manifest: Bump depot_tools again.

CQ-DEPEND=CL:*566119
BUG= chromium:808434 
TEST=None

Change-Id: I1be0a7b0f4de5dc5fa9438fd4dacfaef7de24aec
Reviewed-on: https://chromium-review.googlesource.com/909848
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/5d19cea31debf4e9cb93f3c21a8d788a0611b898/full.xml

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/manifest-internal/+/959b405c6d2a8738744aa9cc49ccb4a5731e5168

commit 959b405c6d2a8738744aa9cc49ccb4a5731e5168
Author: Don Garrett <dgarrett@google.com>
Date: Thu Feb 08 22:52:52 2018

after the depot_tools roll this is still failing: 
(unless the builder is not picking the right depot_tools)

Please verify 

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/19878

The "bypass" solution will not fix the problem in #103 but will unblock the chromiumos_sdk. 

So, we need the bypass to allow delivery of security fixes to the kernel.

and regarding #103, we should consider disabling the unit test to get things rolling until a real fix is found.

CL to disable unit tests: https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/910610

Hi folks,

Luis' concern here is *not* unfounded. The toolchain team is blocked on this issue to complete a Spectre (might have heard of it) related fix. I'd appreciate if his team could be unblocked as expediently as possible.

Thanks for the help =)

elm-release tryjob running with bypass and ci-results-archiver unit test disabled: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/19879

Small mistake in unit test CL, forgot to ammend before pushing. Take 2: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/19880

+1 for bypassing as much as we can to get a build. My other posts on this bug weren't meant to imply otherwise.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/82d5ad2f79ece97e309791ded66ab494d443b51a

commit 82d5ad2f79ece97e309791ded66ab494d443b51a
Author: Mike Frysinger <vapier@chromium.org>
Date: Fri Feb 09 04:50:29 2018

chromeos-chrome: disable venv usage

The venv logic in Chromium is breaking when cross-compiling and it's
not entirely clear why.  But since our SDK should include all the
python modules that Chromium needs (or we can add new ones), lets
disable the logic entirely.

BUG= chromium:808434 
TEST=`cros tryjob elm-tot-chrome-pfq-informational` passes
TEST=`cros tryjob elm-release-tryjob` passes

Change-Id: Ia7fb3852868d4741e9bdb0d20b40750524c0827f
Reviewed-on: https://chromium-review.googlesource.com/905105
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>
Commit-Queue: Nicolas Boichat <drinkcat@chromium.org>
Tested-by: Nicolas Boichat <drinkcat@chromium.org>

[modify] https://crrev.com/82d5ad2f79ece97e309791ded66ab494d443b51a/chromeos-base/chromeos-chrome/chromeos-chrome-66.0.3343.0_rc-r1.ebuild
[modify] https://crrev.com/82d5ad2f79ece97e309791ded66ab494d443b51a/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/81cba41ca275424fdfd57bd8de23437e6fcdf1f0

commit 81cba41ca275424fdfd57bd8de23437e6fcdf1f0
Author: Nicolas Boichat <drinkcat@google.com>
Date: Fri Feb 09 04:54:38 2018

ci-results-archiver: Disable unit tests

Until we can figure out the root cause, disable the unit tests.

BUG= chromium:808434 
TEST=VIRTUALENV_ALWAYS_COPY=1 FEATURES=test emerge-elm -av ci-results-archiver
TEST=cros tryjob elm-release-tryjob

Change-Id: Ieccc8c082cbcfe3b31f29e1c8a179d5127d7dd63
Reviewed-on: https://chromium-review.googlesource.com/910610
Reviewed-by: Nicolas Boichat <drinkcat@chromium.org>
Commit-Queue: Nicolas Boichat <drinkcat@chromium.org>
Tested-by: Nicolas Boichat <drinkcat@chromium.org>

[modify] https://crrev.com/81cba41ca275424fdfd57bd8de23437e6fcdf1f0/chromeos-base/ci-results-archiver/ci-results-archiver-9999.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/7451547f9689263bbdd3dd538455ef64d545ea3d

commit 7451547f9689263bbdd3dd538455ef64d545ea3d
Author: Dan Jacques <dnj@chromium.org>
Date: Fri Feb 09 05:13:24 2018

[vpython] more context in errors

Add more context to error message in order to help debug obscure failure
case on CrOS.

TBR=iannucci@chromium.org
BUG= chromium:808434 
TEST=None

Change-Id: I59974086c6aeb469aee9aa1dda85be35412ba570
Reviewed-on: https://chromium-review.googlesource.com/910197
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/7451547f9689263bbdd3dd538455ef64d545ea3d/vpython/venv/venv.go

We got a successful elm-release ToT build here (still a tryjob):
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/19882

We'll need to revert #127 though, filed  issue 810678  for this purpose.

We'll also need to revert the chromeos-chrome bypass #126 eventually, I guess we can use this bug to track.

We got a green elm-release build, at last:
https://uberchromegw.corp.google.com/i/chromeos/builders/elm-release/builds/1930

Awesome! We'll want to try some builds with new "vpython" versions to see if we can figure out the underlying issue. How would you recommend doing that? Tryjobs like #105, or can we break this builder again now that we have a green?

You can launch a tryjob with the revert:
https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/911448

So cros tryjob -g 911448 elm-release-tryjob

That CL will only work until the next chrome uprev though, we might either need to update it as needed, or modify the -9999 ebuild only and use `--pass-through=--chrome_rev=tot` (see #93).

prob not the right owner for this (anymore?).  going to reassign to the vpython/depot_tools guys.

this bug is quite long/noisy, so might be useful to just mark it as fixed and start up a new one (referencing this) for figuring out why vpython is failing here

thanks for the workarounds!

Closing in favor of crbug.com/810834 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/infra_virtualenv/+/500052dac368dbbef4cdc9b11e824b1f73a924e4

commit 500052dac368dbbef4cdc9b11e824b1f73a924e4
Author: Allen Li <ayatane@google.com>
Date: Sat Feb 10 00:07:17 2018

Fix virtualenv bug when VIRTUALENV_ALWAYS_COPY is set

BUG= chromium:808434 
TEST=Run tests

Change-Id: Iab44fc3a2e18b19d4e3d8486c5ec16b4e382d2cc
Reviewed-on: https://chromium-review.googlesource.com/909532
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/500052dac368dbbef4cdc9b11e824b1f73a924e4/venv/cros_venv/venvlib.py

RE #136, note that #124 suggests that this environment variable, while capable of reproducing the problem, isn't the actual cause of the problem. #124 was run with a "vpython" version that did something similar (removed VIRTUALENV_* and VIRTUAL_ENV_*) and the build still broke identically :(

Issue 809815 has been merged into this issue.