Update "depot_tools" Python and Git bundling. |
|||||
Issue description"depot_tools" Python and Git bundles are old, unmanaged, and have been supplanted by "third_party_packages"-packaged CIPD bundles. Unfortunately, the installer hasn't been updated to accommodate this. This task involves: 1) Replace Git CIPD package/installer invocation with CIPD bundle/deployment. 2) Replace Python ZIP package and unzip-based deployment with CIPD bundle/deployment. Along the way, let's remove legacy code to simplify the scripts in general.
,
Jul 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/c66dcc25362463d342aade0c51b7bdf10f6b7d99 commit c66dcc25362463d342aade0c51b7bdf10f6b7d99 Author: Dan Jacques <dnj@chromium.org> Date: Sat Jul 08 00:42:52 2017 [depot_tools] Remove Git installer bundling. Remove Git installer bundling from the recipe. As of: https://chromium-review.googlesource.com/c/562531/ The bleeding-edge Git bundle uses CIPD directly and does not need an installer package. BUG= chromium:740171 TEST=expectations R=iannucci@chromium.org Change-Id: I9ea63690f604c1badddf982764899cd0b88ccd35 Reviewed-on: https://chromium-review.googlesource.com/564339 Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Commit-Queue: Robbie Iannucci <iannucci@chromium.org> [modify] https://crrev.com/c66dcc25362463d342aade0c51b7bdf10f6b7d99/recipes/recipes/depot_tools_builder.py [modify] https://crrev.com/c66dcc25362463d342aade0c51b7bdf10f6b7d99/recipes/recipes/depot_tools_builder.expected/basic.json [delete] https://crrev.com/29b7f60fe76f344eb134c2fb92efce3e067c9aaa/recipes/recipes/depot_tools_builder.expected/identical_normal_bleeding_edge_git_versions.json
,
Jul 10 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infra/puppet/+/ce4e16e6ffae71846e4a30c39f1494f3e97935be commit ce4e16e6ffae71846e4a30c39f1494f3e97935be Author: Dan Jacques <dnj@chromium.org> Date: Mon Jul 10 20:20:09 2017
,
Jul 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools/+/13e88538fcb3c6d84610345874d4b85bb5a97106 commit 13e88538fcb3c6d84610345874d4b85bb5a97106 Author: Dan Jacques <dnj@chromium.org> Date: Mon Jul 10 21:25:08 2017 Ignore the ".bleeding_edge" file. This file is introduced via Puppet and will be migrated to from ".git_bleeding_edge". In the meantime, ignore it so Git doesn't think it's an untracked modication. TBR=iannucci@chromium.org BUG= chromium:740171 TEST=None Change-Id: Ia878996c3c32c49e1f7d80293bef55ed0a09dbab Reviewed-on: https://chromium-review.googlesource.com/565888 Reviewed-by: Daniel Jacques <dnj@chromium.org> Commit-Queue: Daniel Jacques <dnj@chromium.org> [modify] https://crrev.com/13e88538fcb3c6d84610345874d4b85bb5a97106/.gitignore
,
Jul 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools/+/4cb9d7422d545a9a08d544e07a9565b8c3fb9e60 commit 4cb9d7422d545a9a08d544e07a9565b8c3fb9e60 Author: Dan Jacques <dnj@chromium.org> Date: Tue Jul 11 00:32:07 2017 [bootstrap/win] Enable revert from upcoming patch. The upcoming CL, https://chromium-review.googlesource.com/c/563036, updates "bootstrap/win"'s mechanism. However, if that patch needs to be reverted, its "python.bat" stub may still exist in the reverted repository, and will continue to point to the reverted toolchain. We don't have a good way to detect whether we should revert "python.bat". Instead, we will look a file generated by the CL, "//python_bin_reldir.txt". If we find it, we will assume that the updated "python.bat" is in use, and reinstall. Note that if that CL lands and does not require a revert, this logic will be deleted. It's just there as a safety mechanism to allow a safe revert. BUG= chromium:740171 TEST=local Change-Id: Ifc638cf0512d2a889c37fbf6b8e3f7a3269331b1 Reviewed-on: https://chromium-review.googlesource.com/566073 Commit-Queue: Daniel Jacques <dnj@chromium.org> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> [modify] https://crrev.com/4cb9d7422d545a9a08d544e07a9565b8c3fb9e60/bootstrap/win/win_tools.bat
,
Jul 11 2017
,
Jul 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/f906cbf330eeff5faa999a487009914772344784 commit f906cbf330eeff5faa999a487009914772344784 Author: Dan Jacques <dnj@google.com> Date: Tue Jul 11 17:11:05 2017 [bot_setup] Use python.bat instead of exe path. The Python exe path is not a convention, and should not be hard-coded into scripts. Prefer the entry point, "python.bat", instead of the actual executable. BUG= chromium:740171 TEST=None Change-Id: Iac6e86987ba6839c923fd198c65325771968eb06 Reviewed-on: https://chromium-review.googlesource.com/567135 Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Commit-Queue: Daniel Jacques <dnj@chromium.org> [modify] https://crrev.com/f906cbf330eeff5faa999a487009914772344784/infra/tools/bot_setup/start/chrome.py
,
Jul 11 2017
,
Jul 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools/+/265255f501b5954d24c851e360b5350322a6f6b3 commit 265255f501b5954d24c851e360b5350322a6f6b3 Author: Dan Jacques <dnj@chromium.org> Date: Tue Jul 11 17:54:13 2017 [bootstrap/win] Replace "python.bat" only on downgrade. We're implementing a bootstrap downgrade for potential revert of https://chromium-review.googlesource.com/c/563036 Currently, the downgrade sees the presence of the landed patch as a sign to completely reinstall Python. However, this causes the "python276_bin" directory to be deleted, which both ruins any running process (notably "service manager" and BuildBot) and fails due to those running processes, leaving the system in a broken state. Instead, we'll view the revert path as a signal to swap in the old "python.bat" specifically, then fall through to standard installation detection. Since the upgrade will not actually delete "python276_bin", the expected revert case will leave it alone, preserving existing execution environments. BUG=chromium:740966, chromium:740171 TEST=local - Ran simulation of revert path with this change, observed better outcome. Change-Id: I0dfa5924a27bcaba49134272a344f7b9f1d475c5 Reviewed-on: https://chromium-review.googlesource.com/567167 Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Commit-Queue: Daniel Jacques <dnj@chromium.org> [modify] https://crrev.com/265255f501b5954d24c851e360b5350322a6f6b3/bootstrap/win/win_tools.bat
,
Jul 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools/+/bf1446791e4b1c5b102559de9fb090ed5ba50cf5 commit bf1446791e4b1c5b102559de9fb090ed5ba50cf5 Author: Dan Jacques <dnj@google.com> Date: Tue Jul 11 21:00:16 2017 [win_tools] Use bundled Python CIPD packages. Enable bundled Python CIPD packages in bleeding-edge mode. This replaces the ZIP unpacking approach used before, and introduces validation and management through the CIPD tool. The bleeding edge version will only install if a sentinel file is present in the "depot_tools" root; otherwise, default behavior will continue. This method adds a upgrade and downgrade path to/from ZIP and CIPD installations. This is done by rewriting the "win_tools.bat" process: 1) Ensure that a bootstrap Python is present. 2) Use it to run "win_tools.py", which has the functionality of "git_bootstrap.py" plus Python installation. 3) Run "win_tools.py" with appropriate flags. Some tricks were employed to handle cases where there is an already-running Python instance that uses the current Python installation and executable. This happens on bots because the system uses the same "depot_tools" checkout at multiple launch layers. To this end, we use the "python.bat" as the "current Python" authority and refrain from cleaning up old Python directories if their "python.exe" binaries are currently in use. We change the Git bleeding edge file to share the same sentinel file as Python, ".bleeding_edge". The new Python should have the same facilities as the original Python bundle. BUG= chromium:740171 TEST=local Change-Id: I1b3b7d31d47d1a37a9dba9114d31681bec558736 Reviewed-on: https://chromium-review.googlesource.com/563036 Commit-Queue: Daniel Jacques <dnj@chromium.org> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> [modify] https://crrev.com/bf1446791e4b1c5b102559de9fb090ed5ba50cf5/bootstrap/win/win_tools.bat [delete] https://crrev.com/b80fac66ada1f27e61d7425d71fa8b631c477230/bootstrap/win/git_bootstrap.py [modify] https://crrev.com/bf1446791e4b1c5b102559de9fb090ed5ba50cf5/bootstrap/win/README.md [modify] https://crrev.com/bf1446791e4b1c5b102559de9fb090ed5ba50cf5/.gitignore [add] https://crrev.com/bf1446791e4b1c5b102559de9fb090ed5ba50cf5/bootstrap/win/manifest_bleeding_edge.txt [modify] https://crrev.com/bf1446791e4b1c5b102559de9fb090ed5ba50cf5/bootstrap/win/git.template.bat [modify] https://crrev.com/bf1446791e4b1c5b102559de9fb090ed5ba50cf5/bootstrap/win/git-bash.template.sh [add] https://crrev.com/bf1446791e4b1c5b102559de9fb090ed5ba50cf5/bootstrap/win/win_tools.py [add] https://crrev.com/bf1446791e4b1c5b102559de9fb090ed5ba50cf5/bootstrap/win/python27.new.bat
,
Jul 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools/+/af5c20f5a4e9a894b17d4a148773b77de91d277a commit af5c20f5a4e9a894b17d4a148773b77de91d277a Author: Andrii Shyshkalov <tandrii@chromium.org> Date: Wed Jul 12 13:52:54 2017 Revert "[win_tools] Use bundled Python CIPD packages." This reverts commit bf1446791e4b1c5b102559de9fb090ed5ba50cf5. Reason for revert: broke win bots http://o/410053 Original change's description: > [win_tools] Use bundled Python CIPD packages. > > Enable bundled Python CIPD packages in bleeding-edge mode. This > replaces the ZIP unpacking approach used before, and introduces > validation and management through the CIPD tool. The bleeding edge > version will only install if a sentinel file is present in the > "depot_tools" root; otherwise, default behavior will continue. > > This method adds a upgrade and downgrade path to/from ZIP and > CIPD installations. This is done by rewriting the "win_tools.bat" > process: > > 1) Ensure that a bootstrap Python is present. > 2) Use it to run "win_tools.py", which has the functionality of > "git_bootstrap.py" plus Python installation. > 3) Run "win_tools.py" with appropriate flags. > > Some tricks were employed to handle cases where there is an > already-running Python instance that uses the current Python > installation and executable. This happens on bots because the > system uses the same "depot_tools" checkout at multiple launch > layers. To this end, we use the "python.bat" as the "current Python" > authority and refrain from cleaning up old Python directories if their > "python.exe" binaries are currently in use. > > We change the Git bleeding edge file to share the same > sentinel file as Python, ".bleeding_edge". > > The new Python should have the same facilities as the original Python > bundle. > > BUG= chromium:740171 > TEST=local > > Change-Id: I1b3b7d31d47d1a37a9dba9114d31681bec558736 > Reviewed-on: https://chromium-review.googlesource.com/563036 > Commit-Queue: Daniel Jacques <dnj@chromium.org> > Reviewed-by: Robbie Iannucci <iannucci@chromium.org> TBR=iannucci@chromium.org,dnj@chromium.org Change-Id: I84574a01bbad6596912e4aaa34f019d24720b638 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: chromium:740171 Reviewed-on: https://chromium-review.googlesource.com/567009 Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org> [modify] https://crrev.com/af5c20f5a4e9a894b17d4a148773b77de91d277a/bootstrap/win/win_tools.bat [add] https://crrev.com/af5c20f5a4e9a894b17d4a148773b77de91d277a/bootstrap/win/git_bootstrap.py [modify] https://crrev.com/af5c20f5a4e9a894b17d4a148773b77de91d277a/bootstrap/win/README.md [modify] https://crrev.com/af5c20f5a4e9a894b17d4a148773b77de91d277a/.gitignore [delete] https://crrev.com/7dadf05dad8e882f81dfc1eb032ab99bba76358f/bootstrap/win/manifest_bleeding_edge.txt [modify] https://crrev.com/af5c20f5a4e9a894b17d4a148773b77de91d277a/bootstrap/win/git.template.bat [modify] https://crrev.com/af5c20f5a4e9a894b17d4a148773b77de91d277a/bootstrap/win/git-bash.template.sh [delete] https://crrev.com/7dadf05dad8e882f81dfc1eb032ab99bba76358f/bootstrap/win/win_tools.py [delete] https://crrev.com/7dadf05dad8e882f81dfc1eb032ab99bba76358f/bootstrap/win/python27.new.bat
,
Jul 12 2017
,
Jul 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools/+/3d9b58819b77e1dc1451b6874d1f8e7b666d8f40 commit 3d9b58819b77e1dc1451b6874d1f8e7b666d8f40 Author: Dan Jacques <dnj@chromium.org> Date: Wed Jul 12 22:31:55 2017 [win_tools] Use bundled Python CIPD packages (#2). Re-land of bf1446791e4b1c5b102559de9fb090ed5ba50cf5. Enable bundled Python CIPD packages in bleeding-edge mode. This replaces the ZIP unpacking approach used before, and introduces validation and management through the CIPD tool. The bleeding edge version will only install if a sentinel file is present in the "depot_tools" root; otherwise, default behavior will continue. This method adds a upgrade and downgrade path to/from ZIP and CIPD installations. This is done by rewriting the "win_tools.bat" process: 1) Ensure that a bootstrap Python is present. 2) Use it to run "win_tools.py", which has the functionality of "git_bootstrap.py" plus Python installation. 3) Run "win_tools.py" with appropriate flags. Some tricks were employed to handle cases where there is an already-running Python instance that uses the current Python installation and executable. This happens on bots because the system uses the same "depot_tools" checkout at multiple launch layers. To this end, we use the "python.bat" as the "current Python" authority and refrain from cleaning up old Python directories if their "python.exe" binaries are currently in use. We change the Git bleeding edge file to share the same sentinel file as Python, ".bleeding_edge". The new Python should have the same facilities as the original Python bundle. BUG= chromium:740171 TEST=local Change-Id: I51ba6415c60b95c2aaba94b6e21bd9b3fc82f35d Reviewed-on: https://chromium-review.googlesource.com/568524 Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Commit-Queue: Robbie Iannucci <iannucci@chromium.org> [modify] https://crrev.com/3d9b58819b77e1dc1451b6874d1f8e7b666d8f40/bootstrap/win/win_tools.bat [delete] https://crrev.com/209a681f9cb0e955d5659216f262ef5b00861e8a/bootstrap/win/git_bootstrap.py [modify] https://crrev.com/3d9b58819b77e1dc1451b6874d1f8e7b666d8f40/bootstrap/win/README.md [modify] https://crrev.com/3d9b58819b77e1dc1451b6874d1f8e7b666d8f40/.gitignore [add] https://crrev.com/3d9b58819b77e1dc1451b6874d1f8e7b666d8f40/bootstrap/win/manifest_bleeding_edge.txt [modify] https://crrev.com/3d9b58819b77e1dc1451b6874d1f8e7b666d8f40/bootstrap/win/git.template.bat [modify] https://crrev.com/3d9b58819b77e1dc1451b6874d1f8e7b666d8f40/bootstrap/win/git-bash.template.sh [add] https://crrev.com/3d9b58819b77e1dc1451b6874d1f8e7b666d8f40/bootstrap/win/win_tools.py [add] https://crrev.com/3d9b58819b77e1dc1451b6874d1f8e7b666d8f40/bootstrap/win/python27.new.bat
,
Jul 13 2017
So far landing, Windows builders on bleeding edge and normal living through 2+ syncs after recipe rolled.
,
Jul 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools/+/3a8717e17fbf9904cd6671a3c477a45d5be31a5a commit 3a8717e17fbf9904cd6671a3c477a45d5be31a5a Author: Dan Jacques <dnj@google.com> Date: Thu Jul 13 17:45:21 2017 [bootstrap/win] Remove legacy Git support. Remove non-CIPD Git installation code path. This changes the stable Git version to use the Git package and removes legacy Git support. Note that this is different from legacy (non-CIPD-managed) bootstrap. This will be removed in a future patch. This is effectively a roll of bleeding-edge into stable. BUG= chromium:740171 TEST=local Change-Id: Ia320efd7866f6c11e8aa18d6ce6157614b44fcab Reviewed-on: https://chromium-review.googlesource.com/567764 Commit-Queue: Daniel Jacques <dnj@chromium.org> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> [modify] https://crrev.com/3a8717e17fbf9904cd6671a3c477a45d5be31a5a/bootstrap/win/git_version.txt [modify] https://crrev.com/3a8717e17fbf9904cd6671a3c477a45d5be31a5a/bootstrap/win/win_tools.py
,
Jul 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/depot_tools/+/c4dd3e8264dbeda10f5923bceff984ff66756902 commit c4dd3e8264dbeda10f5923bceff984ff66756902 Author: Dan Jacques <dnj@chromium.org> Date: Thu Jul 13 23:52:14 2017 [bootstrap/win] Remove support for legacy installs Migrate CIPD bundle installations to stable and remove support in Windows bootstrap code for legacy (non-Windows) bootstrap installations. This includes removal of support for - ZIP installation paths - everything through CIPD now. - Python legacy code (though cleanup logic still exists). - JavaScript downloads. - Separate Git package versioning. This is a push from "bleeding edge" to production. BUG= chromium:740171 TEST=local, prod-exemplar - Tested cold and warm upgrade and downgrade. - Tested upgrade and downgrade step via "update_scripts". Change-Id: I3dc4392ef2eaa2d7cf829d560128b74e4dbd2cba Reviewed-on: https://chromium-review.googlesource.com/567767 Commit-Queue: Daniel Jacques <dnj@chromium.org> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> [modify] https://crrev.com/c4dd3e8264dbeda10f5923bceff984ff66756902/bootstrap/win/win_tools.bat [delete] https://crrev.com/cc27ecb005d4c62be5580d6d22c9b246a84d81c9/bootstrap/win/python276.new.bat [delete] https://crrev.com/cc27ecb005d4c62be5580d6d22c9b246a84d81c9/bootstrap/win/git_version_bleeding_edge.txt [add] https://crrev.com/c4dd3e8264dbeda10f5923bceff984ff66756902/bootstrap/win/manifest.txt [delete] https://crrev.com/cc27ecb005d4c62be5580d6d22c9b246a84d81c9/bootstrap/win/git_version.txt [delete] https://crrev.com/cc27ecb005d4c62be5580d6d22c9b246a84d81c9/bootstrap/win/unzip.js [modify] https://crrev.com/c4dd3e8264dbeda10f5923bceff984ff66756902/bootstrap/win/win_tools.py [delete] https://crrev.com/cc27ecb005d4c62be5580d6d22c9b246a84d81c9/bootstrap/win/get_file.js
,
Jul 14 2017
This seems to be complete. I'm optimistically marking this Fixed!
,
Sep 5 2017
Issue 612757 has been merged into this issue. |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by bugdroid1@chromium.org
, Jul 7 2017