For reference, this is `go.chromium.org/luci/vpython/venv/venv_test.go`

+dnj in case he has any ideas

It's failing to build the VirtualEnv here: https://chromium.googlesource.com/infra/luci/luci-go/+/bdab6a0fb342f52cc9bf7284be93638776910a78/vpython/venv/venv_resources_test.go#236

I can't see anything obviously wrong, and that VirtualEnv construction should be locking within itself to prevent contention. Is this a recurring thing, or just a one-off?

Seemed pretty reproducible on the bots (https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/853354, can't seem to view tryjobs from previous patches tho. all failed with the same error, and also with retries)

Still don't have a local repro.

This is with the vpython native doohickey.

Hm... So I see that these tests are running in parallel. Is it possible that it's getting multiply horked? It's failing in a temp dir though so I dunno why :(

Each parallel test should be running in its own tempdir vpython root, and should not conflict. Even then, the part of the code that's failing should be operating on a vpython root protected by a filesystem lock.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/090904ec7384dd631ec886fa967892f226b8901c

commit 090904ec7384dd631ec886fa967892f226b8901c
Author: Dan Jacques <dnj@chromium.org>
Date: Wed Jan 10 03:43:30 2018

[vpython] Ensure stdlib is user-writable.

Ensure that the VirtualEnv's stdlib directory is user-writable. This
avoids a dependency on VirtualEnv setting vpython-compatible directory
permissions on its generated VirtualEnv.

This is somewhat speculative, as local reproduction of the error has not
yet been successful.

BUG:  chromium:800599 
TEST: None
Change-Id: Ide96c530ed7e5a9d5a13e449a6d22fc269b49860
Reviewed-on: https://chromium-review.googlesource.com/858696
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/090904ec7384dd631ec886fa967892f226b8901c/common/system/filesystem/filesystem.go
[modify] https://crrev.com/090904ec7384dd631ec886fa967892f226b8901c/vpython/venv/venv.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c673482d7b205e7f383a3b7311cb474244613975

commit c673482d7b205e7f383a3b7311cb474244613975
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Jan 10 04:34:50 2018

Roll luci-go

infra/go/src/go.chromium.org/luci:
090904ec [vpython] Ensure stdlib is user-writable.
c84050fe [milo] Bail out of saving master data if we encounter a large build
bdab6a0f [milo] Reduce concurrent requests
1cdaa514 [milo] Remove logs service
08c9bc73 Revert "[milo] Simplify appengine services."
643247bc Revert "[milo] Remove unnecessary dispatch.yaml."
e192bc2e [gitiles] fix logging, formatting args
48025314 Milo: add gerrit patchset numbers to swarmbucket builds.
80b7e192 [doc] update bqschemaupdater docs
145c4e79 [Machine Database] Add command to retrieve vlans
3c39b297 [milo] include experimental builds in buildbucket builder view
f3bdacbc [milo] Add buildbucket access ACL functions.
843f2ab3 [bqschemaupdater] ignore protobuf structs
c8b8e482 [prpc] correct code->http status mapping
13c221aa Add int64 flag list type
1583a56a [gerrit] Extend Change with label information.
62e5ce7b [Machine Database] Create rpc to retrieve vlans
e7e8e68e [Machine Database] Add command to retrieve platforms
0d619549 [Machine Database] Create rpc to retrieve platforms
527ae48e [Machine Database] Add command to retrieve operating systems
db1e023a Revert "Reland "[milo] disable API emulation""
5ab11937 [milo] use git package in buildstore/blame.go
11b58001 [milo] improve git package

TBR=dnj@chromium.org, tandrii@chromium.org

Bug:  800599 
Change-Id: I45b56ba11bc76e06473b19509c1d107776c151b7
Reviewed-on: https://chromium-review.googlesource.com/858739
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/c673482d7b205e7f383a3b7311cb474244613975/DEPS

"ensure stdlib is user-writeable" didn't help :/

https://ci.chromium.org/swarming/task/3af7b729d6b61910?server=chromium-swarm.appspot.com

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/b85306df43764dc7366a96df096ad180630442da

commit b85306df43764dc7366a96df096ad180630442da
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Jan 10 08:52:21 2018

[vpython] Remove sitecustomize.py when creating venv.

This should ensure that even if VirtualEnv copies a sitecustomize.py
(from anywhere), that we actually overwrite it.

Apparently VirtualEnv construction is non-deterministic! Hooray.

R=dnj@chromium.org, tandrii@chromium.org

Bug:  800599 
Change-Id: If14aa87390f3fd1c32f41b57887b7fc0471b4dee
Reviewed-on: https://chromium-review.googlesource.com/859110
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/b85306df43764dc7366a96df096ad180630442da/vpython/venv/venv.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/a52a91ee98ec08c886c3fb4926e7b050ae3e516b

commit a52a91ee98ec08c886c3fb4926e7b050ae3e516b
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Jan 10 09:15:30 2018

Roll luci-go

infra/go/src/go.chromium.org/luci:
b85306df [vpython] Remove sitecustomize.py when creating venv.

TBR=dnj@chromium.org, tandrii@chromium.org

Bug:  800599 
Change-Id: I73cd373c7d17ad076110b3e8d1b92a231bbfeba1
Reviewed-on: https://chromium-review.googlesource.com/859357
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/a52a91ee98ec08c886c3fb4926e7b050ae3e516b/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/a9603b62bacdad9e4ec807bdfc254a95fde8c15d

commit a9603b62bacdad9e4ec807bdfc254a95fde8c15d
Author: Dan Jacques <dnj@chromium.org>
Date: Wed Jan 10 14:53:11 2018

[vpython] Handle write-only cases when creating

Update the "sitecustomize.py" creation logic to handle cases where the
parent directory is not user-writable, or where the file exists but is
not user-writable. These edge cases can occur at the discretion of
VirtualEnv creation, which is out of vpython's immediate control.

TBR: iannucci@chromium.org
BUG:  chromium:800599 
TEST: None
Change-Id: I4e421b571f296ac6403d7a2dc732566b65a05f29
Reviewed-on: https://chromium-review.googlesource.com/859840
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/a9603b62bacdad9e4ec807bdfc254a95fde8c15d/common/system/filesystem/filesystem.go
[modify] https://crrev.com/a9603b62bacdad9e4ec807bdfc254a95fde8c15d/vpython/venv/venv.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/50265a598caed50636ca3d5e8654c1331aaa4d02

commit 50265a598caed50636ca3d5e8654c1331aaa4d02
Author: Dan Jacques <dnj@chromium.org>
Date: Wed Jan 10 15:13:30 2018

Roll luci-go.

infra/go/src/go.chromium.org/luci:
a9603b62 [vpython] Handle write-only cases when creating

TBR=iannucci@chromium.org

BUG:  chromium:800599 
TEST: None
Change-Id: I95785ffdcc6060e171cfd8f3941f4aec9cfc88eb
Reviewed-on: https://chromium-review.googlesource.com/860217
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/50265a598caed50636ca3d5e8654c1331aaa4d02/DEPS

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/ec62800569d6d8fe94543ded3df1867087739c1f

commit ec62800569d6d8fe94543ded3df1867087739c1f
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Jan 10 18:59:57 2018

yeyy fixed: https://ci.chromium.org/swarming/task/3afa8814774cef10?server=chromium-swarm.appspot.com

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/07b3c24bf41a7f268f98424fddf1a9221b6962e1

commit 07b3c24bf41a7f268f98424fddf1a9221b6962e1
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Jan 10 21:01:29 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/8d30b2285801d896b09e01342a41557d6fde32ce

commit 8d30b2285801d896b09e01342a41557d6fde32ce
Author: Robert Iannucci <iannucci@chromium.org>
Date: Thu Jan 11 01:05:09 2018

[vpython] Bump production version of vpython.

This fixes the bug where virtualenvs bootstrapped from vpython virtualenvs
would fail to build.

R=nodir@chromium.org, tandrii@chromium.org, vadimsh@chromium.org

Bug: 784597, 800599 
Change-Id: I83db29cedc52e7c6de8cf02974531e1fe82cbb4c
Reviewed-on: https://chromium-review.googlesource.com/861304
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/8d30b2285801d896b09e01342a41557d6fde32ce/scripts/slave/cipd_bootstrap_v2.py
[modify] https://crrev.com/8d30b2285801d896b09e01342a41557d6fde32ce/scripts/slave/recipe_modules/swarming/api.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/da669d61230a10f71a13703889a97827db35ffd9

commit da669d61230a10f71a13703889a97827db35ffd9
Author: Robert Iannucci <iannucci@chromium.org>
Date: Thu Jan 11 01:08:19 2018

[vpython] Bump vpython version

This fixes the bug where virtualenvs bootstrapped from vpython virtualenvs
would fail to build.

R=nodir@chromium.org, tandrii@chromium.org, vadimsh@chromium.org

Bug: 784597, 800599 
Change-Id: Iddcda5fca5f395f031df7a200c688d55ab74ae95
Reviewed-on: https://chromium-review.googlesource.com/860959
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/da669d61230a10f71a13703889a97827db35ffd9/cipd_manifest.txt