Some CrOS builders seem to be incompatible with python virtualenv |
|||||
Issue descriptionSee the saga in issue 808434 . This is affecting the `vpython` tool, as well as the `cros_venv`. We thought that it was caused by an envvar `VIRTUALENV_ALLOW_COPY=1`, which repros the same crash, but even after scrubbing the environment we're still seeing crashes. Currently vpython is in `bypass` mode on these builders (with CL https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/911448), and it would be good to diagnose and understand what's going wrong so we can fix it for real.
,
Feb 9 2018
Just to ask... Most of our python scripts are launched by a wrapper script which modifies the runtime environment (which is how we handle import paths). This wrapper is linked by the symlink (e.g. 'cbuildbot' vs 'cbuildbot.py') that exists for every chromite script. Allen created a new wrapper script for setting up virtualenvs. This means that nearly any new python script invocation could be re-inserting that ENV. Is that what's happening?
,
Feb 9 2018
This only applies when creating the virtualenv, and the virtualenv creation script removes said envvar, so that should not matter. In any case, there's no references to said envvar in chromite, so I doubt chromite is inserting it.
,
Feb 9 2018
So I landed this last night: https://chromium.googlesource.com/infra/luci/luci-go/+/7451547f9689263bbdd3dd538455ef64d545ea3d Once we roll vpython, errors during VirtualEnv setup should dump a full environment stack and command context for debugging. I'm hoping that will shed some light onto what might be going wrong here.
,
Feb 9 2018
Does that mean another depot_tools pin bump?
,
Feb 9 2018
Yeah, infra/DEPS bump to pick it up, then depot_tools bumps to distribute. Is a long process :(
,
Feb 9 2018
(so don't bump depot_tools yet)
,
Feb 9 2018
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/8ea3c307adedf641c4e00fbcdb408b5cd2c72d48 commit 8ea3c307adedf641c4e00fbcdb408b5cd2c72d48 Author: Robert Iannucci <iannucci@chromium.org> Date: Fri Feb 09 20:33:49 2018 Roll luci-go infra/go/src/go.chromium.org/luci: 7451547f [vpython] more context in errors 3d18e65e [Machine Database] Filter NICs at the database level 3907c8dd [cipd] Implement BeginUpload RPC method. d7af7274 [milo] Fix luci buildersummary 665a380f [milo] use new migration app endpoint to read bucket of a master 16245a7b [milo] add paging to git.Log 5d499c13 [milo] Emergency Bandaid Fix - Make main console not panic 9e5fef48 [luci-auth] Rename default account id. 964973df [cipd] Add API definitions for Storage's BeginUpload/FinishUpload. TBR=dnj@chromium.org, nodir@chromium.org Bug: 810834 Change-Id: I7b3164e1b561b4d075b03e1c575f15c12f640cff Reviewed-on: https://chromium-review.googlesource.com/912310 Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Commit-Queue: Robbie Iannucci <iannucci@chromium.org> [modify] https://crrev.com/8ea3c307adedf641c4e00fbcdb408b5cd2c72d48/DEPS
,
Mar 30 2018
,
Mar 30 2018
,
Jul 20
Don, let's check on if there's anything else to do here.
,
Jul 23
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by iannucci@chromium.org
, Feb 9 2018