Consider replacing "cros_venv" with "vpython"? |
||||||||||||||
Issue descriptionEarly on in vpython development, I engaged dgarrett@ and the CrOS team to offer some detail on the (then-planned) vpython tool. vpython has since been developed, rolled-out, and battle tested, and made an integral part of Chromium toolchain. This bug is an opportuinty to reconsider employing "vpython" by the CrOS team for VirtualEnv management. It's: 1) Availble in CIPD, meaning it's available to CrOS toolchain tools. 2) A consistent picture for how VirtualEnv are defined and shaped. 3) Pretty good and efficient at making VirtualEnv. 4) Offers a single place to make fixes to VirtualEnv building code. 5) Offers an opportunity to build and manage wheels consistently and leverage Chrome Infrastructure tooling for this. 6) Uses CIPD for wheel management, instead of Git or PyPi. (4) in particular would let a single fix apply to all VirtualEnv construction problems, such as https://bugs.chromium.org/p/chromium/issues/detail?id=808434#c103 . Anyway, just wanted to bring that back up for discussion. Feel free to WontFix.
,
Feb 8 2018
More information can be found: - Tool docs (go/vpython): https://chromium.googlesource.com/infra/infra/+/master/doc/users/vpython.md - Infra Implementation Docs: https://chromium.googlesource.com/infra/infra/+/master/go/src/infra/tools/vpython - Generic Tool Docs: https://chromium.googlesource.com/infra/luci/luci-go/+/master/vpython - Current Wheel List: https://chromium.googlesource.com/infra/infra/+/master/infra/tools/dockerbuild/wheels.md
,
Feb 8 2018
I can help with this effort too, if desired :)
,
Feb 9 2018
First blocker: We would either need a way to add custom import paths (e.g., PYTHONPATH) (easy to do the migration), or start building wheels for our Python packages (probably much harder).
,
Feb 9 2018
PYTHONPATH carries over into vpython as of https://chromium.googlesource.com/infra/luci/luci-go/+/cb4b15a5293ccf1ec6fca7968fc611947a30312d ! It is discarded for VirtualEnv setup, but is propagated for the end Python invocation. Wheels are also an option. I tend to prefer them for actual packaged things, but when we already have a mechanism to check out the exact version of software you want to run (git / repo), there's no harm in using it via PYTHONPATH.
,
Feb 9 2018
,
Feb 23 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/2e30c5e417c33187a77d602c2a45922699647581 commit 2e30c5e417c33187a77d602c2a45922699647581 Author: Allen Li <ayatane@chromium.org> Date: Fri Feb 23 07:54:37 2018 Move sysmon to vpython BUG=chromium:810460 TEST=Ran locally for a few ticks Change-Id: I2f9b8d9fcdf4fc6f4b950d88bd215657ca6ca8e9 Reviewed-on: https://chromium-review.googlesource.com/912200 Commit-Ready: Allen Li <ayatane@chromium.org> Tested-by: Allen Li <ayatane@chromium.org> Reviewed-by: Paul Hobbs <phobbs@google.com> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> [delete] https://crrev.com/1b20b7548d6693bfa78f41223b3792a43a7f39b2/bin/find_virtualenv.sh [add] https://crrev.com/2e30c5e417c33187a77d602c2a45922699647581/.vpython [modify] https://crrev.com/2e30c5e417c33187a77d602c2a45922699647581/bin/venv_python [modify] https://crrev.com/2e30c5e417c33187a77d602c2a45922699647581/bin/sysmon
,
Feb 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/29ddab90111e264e6846a7b6763dd58f4af34cb7 commit 29ddab90111e264e6846a7b6763dd58f4af34cb7 Author: Ningning Xia <nxia@chromium.org> Date: Sat Feb 24 00:18:01 2018 Revert "Move sysmon to vpython" This reverts commit 2e30c5e417c33187a77d602c2a45922699647581. Reason for revert: crbug.com/815307 Original change's description: > Move sysmon to vpython > > BUG=chromium:810460 > TEST=Ran locally for a few ticks > > Change-Id: I2f9b8d9fcdf4fc6f4b950d88bd215657ca6ca8e9 > Reviewed-on: https://chromium-review.googlesource.com/912200 > Commit-Ready: Allen Li <ayatane@chromium.org> > Tested-by: Allen Li <ayatane@chromium.org> > Reviewed-by: Paul Hobbs <phobbs@google.com> > Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Bug: chromium:810460 Change-Id: Ib119f850d75f5a5f190708f945f5366d073a1949 Reviewed-on: https://chromium-review.googlesource.com/935802 Reviewed-by: Paul Hobbs <phobbs@google.com> Tested-by: Ningning Xia <nxia@chromium.org> [add] https://crrev.com/29ddab90111e264e6846a7b6763dd58f4af34cb7/bin/find_virtualenv.sh [delete] https://crrev.com/92e338611b65c8b118146ef0f7b29f9cccb608a3/.vpython [modify] https://crrev.com/29ddab90111e264e6846a7b6763dd58f4af34cb7/bin/venv_python [modify] https://crrev.com/29ddab90111e264e6846a7b6763dd58f4af34cb7/bin/sysmon
,
Feb 24 2018
,
Mar 2 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/44ca9f0b9b24705baafee16a48c153c8f165f335 commit 44ca9f0b9b24705baafee16a48c153c8f165f335 Author: Allen Li <ayatane@chromium.org> Date: Fri Mar 02 00:13:39 2018 Reland "Move sysmon to vpython" This is a reland of 2e30c5e417c33187a77d602c2a45922699647581. Original change's description: > Move sysmon to vpython > > BUG=chromium:810460 > TEST=Ran locally for a few ticks > > Change-Id: I2f9b8d9fcdf4fc6f4b950d88bd215657ca6ca8e9 > Reviewed-on: https://chromium-review.googlesource.com/912200 > Commit-Ready: Allen Li <ayatane@chromium.org> > Tested-by: Allen Li <ayatane@chromium.org> > Reviewed-by: Paul Hobbs <phobbs@google.com> > Reviewed-by: Robbie Iannucci <iannucci@chromium.org> BUG=chromium:810460 TEST=Run on staging-master TEST=Run on devserver CQ-DEPEND=CL:*578176 Change-Id: I80a4bdb965a02273f2b301eaf70f0e8502ca98c2 Reviewed-on: https://chromium-review.googlesource.com/938323 Commit-Ready: Allen Li <ayatane@chromium.org> Tested-by: Allen Li <ayatane@chromium.org> Reviewed-by: Paul Hobbs <phobbs@google.com> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> [delete] https://crrev.com/aee36b8a1a2607b1d3e61a896661ad11bb77fb48/bin/find_virtualenv.sh [add] https://crrev.com/44ca9f0b9b24705baafee16a48c153c8f165f335/.vpython [modify] https://crrev.com/44ca9f0b9b24705baafee16a48c153c8f165f335/bin/venv_python [modify] https://crrev.com/44ca9f0b9b24705baafee16a48c153c8f165f335/bin/sysmon
,
Mar 5 2018
,
Mar 9 2018
sysmon works, we can probably move over everything with a bit of care.
,
Mar 29 2018
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/7b6cb919cfbe65f46fd741c3e46cee15d04dd183 commit 7b6cb919cfbe65f46fd741c3e46cee15d04dd183 Author: Allen Li <ayatane@chromium.org> Date: Thu Mar 29 18:56:42 2018 Add chromite wheels These wheels are needed for chromite (Chrome OS tools). BUG=chromium:810460 TEST=None Change-Id: I476f7d070d9a361b552243d167eb58454c7c876e Reviewed-on: https://chromium-review.googlesource.com/985494 Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Commit-Queue: Robbie Iannucci <iannucci@chromium.org> [modify] https://crrev.com/7b6cb919cfbe65f46fd741c3e46cee15d04dd183/infra/tools/dockerbuild/wheel.py
,
Mar 29 2018
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/598f003b656753864fb03a985d44c648992e2cb2 commit 598f003b656753864fb03a985d44c648992e2cb2 Author: Robert Iannucci <iannucci@chromium.org> Date: Thu Mar 29 19:51:52 2018 [dockerbuild] Update wheels.md TBR=nodir@chromium.org Bug: 810460 Change-Id: I93b9d3ecbf6381109b2b40d0a6f2476c39b361fb Reviewed-on: https://chromium-review.googlesource.com/986834 Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Commit-Queue: Robbie Iannucci <iannucci@chromium.org> [modify] https://crrev.com/598f003b656753864fb03a985d44c648992e2cb2/infra/tools/dockerbuild/wheels.md
,
Mar 31 2018
Issue 703275 has been merged into this issue.
,
Mar 31 2018
,
Mar 31 2018
,
Mar 31 2018
,
Mar 31 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/c7cb8ed2a1ec6a4610f894df18fd3d520aa3a8c7 commit c7cb8ed2a1ec6a4610f894df18fd3d520aa3a8c7 Author: Allen Li <ayatane@chromium.org> Date: Sat Mar 31 02:46:59 2018 Rename venv_python script python_venv is the name we have standardized on (although this is using vpython instead of cros_venv, this should be transparent). BUG=chromium:810460 TEST=None Change-Id: I0d22994d053e8c0e757d45409bab6e99b1ee07f7 Reviewed-on: https://chromium-review.googlesource.com/985120 Commit-Ready: Allen Li <ayatane@chromium.org> Tested-by: Allen Li <ayatane@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [rename] https://crrev.com/c7cb8ed2a1ec6a4610f894df18fd3d520aa3a8c7/bin/python_venv [modify] https://crrev.com/c7cb8ed2a1ec6a4610f894df18fd3d520aa3a8c7/bin/sysmon
,
Mar 31 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e11a8910b46299ad2bd6d3860016a3dd4b5043f9 commit e11a8910b46299ad2bd6d3860016a3dd4b5043f9 Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Sat Mar 31 04:01:58 2018 Roll src/third_party/chromite/ 907239157..c7cb8ed2a (4 commits) https://chromium.googlesource.com/chromiumos/chromite.git/+log/907239157e6a..c7cb8ed2a1ec $ git log 907239157..c7cb8ed2a --date=short --no-merges --format='%ad %ae %s' 2018-03-28 ayatane Rename venv_python script 2018-03-30 yunlian chromeos_config: Remove USE=thinlto from release builders. 2018-03-30 shapiroc CBuildBot support for YAML file 2018-03-25 jclinton cbuildbot/stages/build_stages: Make model probe at end silent Created with: roll-dep src/third_party/chromite BUG=chromium:810460, chromium:827217 ,chromium:none The AutoRoll server is located here: https://chromite-chromium-roll.skia.org Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. TBR=chrome-os-gardeners@chromium.org Change-Id: Ifa38cba737199e75e2a1319150248a9bc1b31166 Reviewed-on: https://chromium-review.googlesource.com/989304 Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#547372} [modify] https://crrev.com/e11a8910b46299ad2bd6d3860016a3dd4b5043f9/DEPS
,
Apr 12 2018
,
Apr 13 2018
,
Apr 24 2018
,
Apr 25 2018
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/1ae0366ad505e3d833a7d58fe31172f78fc50f6a commit 1ae0366ad505e3d833a7d58fe31172f78fc50f6a Author: Allen Li <ayatane@chromium.org> Date: Wed Apr 25 00:25:05 2018 Add dependencies for astroid I missed these dependencies that are added dynamically for Python 2. Bug: 810460 Change-Id: I6ade26cace376b332c98f563b88ad00483c70771 Reviewed-on: https://chromium-review.googlesource.com/1018010 Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Reviewed-by: Daniel Jacques <dnj@chromium.org> [modify] https://crrev.com/1ae0366ad505e3d833a7d58fe31172f78fc50f6a/infra/tools/dockerbuild/wheel.py
,
Apr 25 2018
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/01702be37a1daa95c32005c8fe306f428812a7c4 commit 01702be37a1daa95c32005c8fe306f428812a7c4 Author: Allen Li <ayatane@chromium.org> Date: Wed Apr 25 00:41:45 2018 Add Python wheels for Chrome OS Autotest Bug: 810460 Change-Id: I5de84c45216e67f2d85e9f0998bb4c60a495a48d Reviewed-on: https://chromium-review.googlesource.com/1020230 Commit-Queue: Allen Li <ayatane@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> [modify] https://crrev.com/01702be37a1daa95c32005c8fe306f428812a7c4/infra/tools/dockerbuild/wheel.py [modify] https://crrev.com/01702be37a1daa95c32005c8fe306f428812a7c4/infra/tools/dockerbuild/wheels.md
,
Apr 26 2018
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/bee3ecbf0ceec14fb4b8674291019affaec25ac3 commit bee3ecbf0ceec14fb4b8674291019affaec25ac3 Author: Allen Li <ayatane@chromium.org> Date: Thu Apr 26 21:17:48 2018 [dockerbuild] Add Jinja2 wheel Bug: 810460 Change-Id: I89dc3c321c308dd18beb1364e088af523653f270 Reviewed-on: https://chromium-review.googlesource.com/1031071 Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Commit-Queue: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/bee3ecbf0ceec14fb4b8674291019affaec25ac3/infra/tools/dockerbuild/wheel.py
,
Apr 27 2018
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/7b0f9183765aeac481a8ef98e4426aa26b797286 commit 7b0f9183765aeac481a8ef98e4426aa26b797286 Author: Allen Li <ayatane@chromium.org> Date: Fri Apr 27 19:43:16 2018 [dockerbuild] Add configparser backport wheel Bug: 810460 Change-Id: Ie41c31340e9587f4a464e8c22466bd696ccfe02b Reviewed-on: https://chromium-review.googlesource.com/1031775 Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Commit-Queue: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/7b0f9183765aeac481a8ef98e4426aa26b797286/infra/tools/dockerbuild/wheel.py
,
May 15 2018
,
Oct 18
,
Oct 25
The easiest way to fix Autotest is to wait for it to die at this point.
,
Dec 17
However, Chromite will live on for awhile longer. iannucci@, way up you offered to help a little. We're very close and just need someone to drive this across the finish line. There's a CL in flight <https://chromium-review.googlesource.com/c/chromiumos/chromite/+/985121/7> that's almost there but had some trouble. Can you take a look? |
||||||||||||||
►
Sign in to add a comment |
||||||||||||||
Comment 1 by ayatane@chromium.org
, Feb 8 2018