New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 714852 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner: ----
Closed: Oct 1
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug

Blocked on:
issue 714297
issue 718868

Blocking:
issue 419453
issue 629311
issue 714293
issue 717736



Sign in to add a comment

deploy git and python to all bots via cipd as a part of build bootstrap

Project Member Reported by no...@chromium.org, Apr 24 2017

Issue description

Problem. Today some build scripts in depot tools (e.g. apply_issue.bat [1]) expect its depot_tools  dir to provision python/git. Other build scripts may be expecting the depot_tools in $PATH to provision git or python. In both cases, they do it because historically git/python in $PATH may not exist or may be old because we don't update git/python on machines.

Soltuion. Build CIPD packages for python and git, install (ensure) them during build bootstrap, e.g. annotated_run.py, remote_run.py, kitchen-level and make them available in $PATH, so that git/python in $PATH are always safe to use.

The motivation is make win_chromium_rel_ng work on LUCI. Today it does not work because unlike buildbot where depot_tools is updated as as part of update_scripts step, on LUCI the depot_tools in $PATH is old; and we cannot insert recipe-pinned depot_tools into the beginning of $PATH because it is expected to have git/python, which it does not. Fixing this bug will remove the expectation. Once done, we will insert the recipe-pinned depot_tools to the beginning of $PATH.

[1]: https://cs.chromium.org/chromium/tools/depot_tools/apply_issue.bat?dr=C&q=apply_issu&l=11

 

Comment 1 by no...@chromium.org, Apr 24 2017

Blockedon: 714297

Comment 2 by no...@chromium.org, Apr 27 2017

this does not include 64-bit python on windows
CIPD packages for python on windows are based on existing gs://chrome-infra/python276_bin.zip which is 32bit
Project Member

Comment 3 by bugdroid1@chromium.org, May 1 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/f00c4aba35dc7b72edd8568c8e73de2c4136c265

commit f00c4aba35dc7b72edd8568c8e73de2c4136c265
Author: Nodir Turakulov <nodir@google.com>
Date: Mon May 01 17:37:29 2017

third_party_packages recipe: package Git and Python

We use Git and Python through the fleet, but their deployment to build machines
is not controlled. Different machines have different versions. Some are old
which is a security risk. We must update entire fleet when a security patch is
released. On Windows we deploy Python via depot_tools.

The purpose of this new recipe is for each third party software that we care
about, check the latest version, check if we have a CIPD package for it, and if
not, build it and upload the package. The initial version covers only
- Git for Linux, Mac and Windows (32bit)
  CIPD prefix is "infra/git/"
- Python for Linux and Mac
  CIPD prefix is "infra/python/cpython/"

This recipe is meant to be run periodically, for example once a day.

R=iannucci@chromium.org, vadimsh@chromium.org
BUG= 714852 
Change-Id: I58b6be3b283ad7cd8b40bb3f300a83d540da8631

Change-Id: I58b6be3b283ad7cd8b40bb3f300a83d540da8631
Reviewed-on: https://chromium-review.googlesource.com/483944
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_python_on_linux-amd64.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_python_on_linux-386.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.py
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_git_on_windows-386.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_git_on_mac-amd64.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_python_on_windows-amd64.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_git_on_linux-386.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_git_on_linux-amd64.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_python_on_windows-386.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_git_on_windows-amd64.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_python_on_mac-amd64.json

Project Member

Comment 4 by bugdroid1@chromium.org, May 1 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/f00c4aba35dc7b72edd8568c8e73de2c4136c265

commit f00c4aba35dc7b72edd8568c8e73de2c4136c265
Author: Nodir Turakulov <nodir@google.com>
Date: Mon May 01 17:37:29 2017

third_party_packages recipe: package Git and Python

We use Git and Python through the fleet, but their deployment to build machines
is not controlled. Different machines have different versions. Some are old
which is a security risk. We must update entire fleet when a security patch is
released. On Windows we deploy Python via depot_tools.

The purpose of this new recipe is for each third party software that we care
about, check the latest version, check if we have a CIPD package for it, and if
not, build it and upload the package. The initial version covers only
- Git for Linux, Mac and Windows (32bit)
  CIPD prefix is "infra/git/"
- Python for Linux and Mac
  CIPD prefix is "infra/python/cpython/"

This recipe is meant to be run periodically, for example once a day.

R=iannucci@chromium.org, vadimsh@chromium.org
BUG= 714852 
Change-Id: I58b6be3b283ad7cd8b40bb3f300a83d540da8631

Change-Id: I58b6be3b283ad7cd8b40bb3f300a83d540da8631
Reviewed-on: https://chromium-review.googlesource.com/483944
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_python_on_linux-amd64.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_python_on_linux-386.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.py
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_git_on_windows-386.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_git_on_mac-amd64.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_python_on_windows-amd64.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_git_on_linux-386.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_git_on_linux-amd64.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_python_on_windows-386.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_git_on_windows-amd64.json
[add] https://crrev.com/f00c4aba35dc7b72edd8568c8e73de2c4136c265/recipes/recipes/third_party_packages.expected/new_python_on_mac-amd64.json

Project Member

Comment 5 by bugdroid1@chromium.org, May 2 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/1afc3517609286df3f4b3fce730ddad3e1eefedc

commit 1afc3517609286df3f4b3fce730ddad3e1eefedc
Author: Nodir Turakulov <nodir@google.com>
Date: Tue May 02 17:42:08 2017

master.chromium.infra.cron: add 3p package builders

Add builders that create CIPD packages of 3p software on linux-amd64, linux-386,
mac-amd64, windows-amd64, windows-386. The bots were provided in crbug.com/714297

R=iannucci@chromium.org
Bug= 714852 

Change-Id: Icd3dcff0ab1edddda0173c06e2d979b3fdea4c70
Reviewed-on: https://chromium-review.googlesource.com/491566
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/1afc3517609286df3f4b3fce730ddad3e1eefedc/masters/master.chromium.infra.cron/slaves.cfg
[modify] https://crrev.com/1afc3517609286df3f4b3fce730ddad3e1eefedc/masters/master.chromium.infra.cron/master.cfg

Project Member

Comment 6 by bugdroid1@chromium.org, May 2 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager/+/468df51926e984d8bba477d528278101e695a1f2

commit 468df51926e984d8bba477d528278101e695a1f2
Author: Nodir Turakulov <nodir@google.com>
Date: Tue May 02 18:12:27 2017

Project Member

Comment 7 by bugdroid1@chromium.org, May 2 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/e25477477759e86493e45c1431d4258db24f33ae

commit e25477477759e86493e45c1431d4258db24f33ae
Author: Nodir Turakulov <nodir@google.com>
Date: Tue May 02 19:24:48 2017

Blocking: 717736
Blocking: 629311
Project Member

Comment 10 by bugdroid1@chromium.org, May 5 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager/+/bc5f9f03844b12a4ece2e1476e0c43b939388a64

commit bc5f9f03844b12a4ece2e1476e0c43b939388a64
Author: Nodir Turakulov <nodir@google.com>
Date: Fri May 05 05:07:29 2017

Blockedon: 718868

Comment 12 by no...@chromium.org, May 17 2017

Blocking: 419453
Project Member

Comment 13 by bugdroid1@chromium.org, May 23 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c515f554a989bf87765c5adde855afcdaa80f87e

commit c515f554a989bf87765c5adde855afcdaa80f87e
Author: Dan Jacques <dnj@chromium.org>
Date: Tue May 23 15:33:34 2017

[third_party_packages] Build full Git.

Current Git builds on Linux don't include "libcurl", which means that
they can't fetch from HTTPS URLs. Update the Git build process to
include static versions of "libcurl", "libssl", and "libz".

In order to avoid double-building LibSSL, which can take a *while*, we
consolidate our support packages into a support class instance which we
share amongst Python and Git builds. Packages are built on-demand, but
if two different build processes demand the same package, it is now only
built once. The two also share the same prefix space and source
downloads.

BUG= chromium:714852 
TEST=local,target
  - Ran locally.
  - Ran on 64-bit Mac, 64-bit Linux, 32-bit Linux systems.

Change-Id: I9f4ae8a18c2a6a4e265c5d0e2b9a64bdb88b4bae
Reviewed-on: https://chromium-review.googlesource.com/511723
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/c515f554a989bf87765c5adde855afcdaa80f87e/recipes/recipes/third_party_packages.expected/new_python_on_linux-amd64.json
[modify] https://crrev.com/c515f554a989bf87765c5adde855afcdaa80f87e/recipes/recipes/third_party_packages.expected/new_python_on_linux-386.json
[modify] https://crrev.com/c515f554a989bf87765c5adde855afcdaa80f87e/recipes/recipes/third_party_packages.py
[modify] https://crrev.com/c515f554a989bf87765c5adde855afcdaa80f87e/recipes/recipes/third_party_packages.expected/mac_failure.json
[modify] https://crrev.com/c515f554a989bf87765c5adde855afcdaa80f87e/recipes/recipes/third_party_packages.expected/mac_specific_git_tag.json
[modify] https://crrev.com/c515f554a989bf87765c5adde855afcdaa80f87e/recipes/recipes/third_party_packages.expected/new_git_on_windows-386.json
[modify] https://crrev.com/c515f554a989bf87765c5adde855afcdaa80f87e/recipes/recipes/third_party_packages.expected/new_git_on_mac-amd64.json
[modify] https://crrev.com/c515f554a989bf87765c5adde855afcdaa80f87e/recipes/recipes/third_party_packages.expected/new_python_on_windows-amd64.json
[modify] https://crrev.com/c515f554a989bf87765c5adde855afcdaa80f87e/recipes/recipes/third_party_packages.expected/new_git_on_linux-386.json
[modify] https://crrev.com/c515f554a989bf87765c5adde855afcdaa80f87e/recipes/recipes/third_party_packages.expected/new_git_on_linux-amd64.json
[modify] https://crrev.com/c515f554a989bf87765c5adde855afcdaa80f87e/recipes/recipes/third_party_packages.expected/new_python_on_windows-386.json
[modify] https://crrev.com/c515f554a989bf87765c5adde855afcdaa80f87e/recipes/recipes/third_party_packages.expected/new_git_on_windows-amd64.json
[modify] https://crrev.com/c515f554a989bf87765c5adde855afcdaa80f87e/recipes/recipes/third_party_packages.expected/new_python_on_mac-amd64.json

Project Member

Comment 14 by bugdroid1@chromium.org, May 23 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/34399778c588f197bd7c7164ad4e90c3e8bda17d

commit 34399778c588f197bd7c7164ad4e90c3e8bda17d
Author: Dan Jacques <dnj@chromium.org>
Date: Tue May 23 22:52:00 2017

[third_party_packages] Relocatable Git!

Update POSIX Git building to allow it to be installed to arbitrary
directories.

Prior to this patch, POSIX Git uses the hard-coded compile prefix to
locate its support utilities and other files. Git's relative-path
addressing is only supported for Windows. Update our build to:
- Apply a patch to Git to enable relative pathing for non-Windows OSes.
- Configure (via Makefile) Git for relative path addressing.
- Use a better injection vector for EXTLIBS.

Additionally, pass "-flto" to build and link to encourage link-time
analysis and omission of unused code. This dramatically reduces the
binary sizes now that they're statically linked.

BUG= chromium:714852 
TEST=local,target
  - Built on local 64-bit Linux and OSX.
  - Built on target 32- and 64-bit Linux, and 64-bit OSX.

Change-Id: I3b79f272ab3ea70219591a176121112e3103c854
Reviewed-on: https://chromium-review.googlesource.com/513362
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/34399778c588f197bd7c7164ad4e90c3e8bda17d/recipes/recipes/third_party_packages.expected/new_python_on_linux-amd64.json
[modify] https://crrev.com/34399778c588f197bd7c7164ad4e90c3e8bda17d/recipes/recipes/third_party_packages.expected/new_python_on_linux-386.json
[modify] https://crrev.com/34399778c588f197bd7c7164ad4e90c3e8bda17d/recipes/recipes/third_party_packages.py
[modify] https://crrev.com/34399778c588f197bd7c7164ad4e90c3e8bda17d/recipes/recipes/third_party_packages.expected/mac_failure.json
[modify] https://crrev.com/34399778c588f197bd7c7164ad4e90c3e8bda17d/recipes/recipes/third_party_packages.expected/mac_specific_git_tag.json
[modify] https://crrev.com/34399778c588f197bd7c7164ad4e90c3e8bda17d/recipes/recipes/third_party_packages.expected/new_git_on_windows-386.json
[modify] https://crrev.com/34399778c588f197bd7c7164ad4e90c3e8bda17d/recipes/recipes/third_party_packages.expected/new_git_on_mac-amd64.json
[modify] https://crrev.com/34399778c588f197bd7c7164ad4e90c3e8bda17d/recipes/recipes/third_party_packages.expected/new_python_on_windows-amd64.json
[modify] https://crrev.com/34399778c588f197bd7c7164ad4e90c3e8bda17d/recipes/recipes/third_party_packages.expected/new_git_on_linux-386.json
[modify] https://crrev.com/34399778c588f197bd7c7164ad4e90c3e8bda17d/recipes/recipes/third_party_packages.expected/new_git_on_linux-amd64.json
[modify] https://crrev.com/34399778c588f197bd7c7164ad4e90c3e8bda17d/recipes/recipes/third_party_packages.expected/new_python_on_windows-386.json
[modify] https://crrev.com/34399778c588f197bd7c7164ad4e90c3e8bda17d/recipes/recipes/third_party_packages.expected/new_git_on_windows-amd64.json
[modify] https://crrev.com/34399778c588f197bd7c7164ad4e90c3e8bda17d/recipes/recipes/third_party_packages.expected/new_python_on_mac-amd64.json

Project Member

Comment 15 by bugdroid1@chromium.org, May 23 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/41c2a64552ed1fbdbb6fb4ac695e34749224eddd

commit 41c2a64552ed1fbdbb6fb4ac695e34749224eddd
Author: Dan Jacques <dnj@chromium.org>
Date: Tue May 23 23:51:53 2017

[third_party_packages] Actually include the patch.

Include the patch referenced in:
34399778c588f197bd7c7164ad4e90c3e8bda17d :/

TBR=iannucci@chromium.org
BUG= chromium:714852 
TEST=None

Change-Id: Ibca9d06a4a09ced0947253d593690b3e48e624c3
Reviewed-on: https://chromium-review.googlesource.com/513365
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[add] https://crrev.com/41c2a64552ed1fbdbb6fb4ac695e34749224eddd/recipes/recipes/third_party_packages.resources/git_2_13_0.posix.patch

Project Member

Comment 16 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/576e1416ff26ab7e9f40b5284ad953f5ee1f3441

commit 576e1416ff26ab7e9f40b5284ad953f5ee1f3441
Author: Dan Jacques <dnj@chromium.org>
Date: Wed May 24 00:52:44 2017

Project Member

Comment 17 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/7a3da2193b02a35c088c23713feb7d8c9236cdf6

commit 7a3da2193b02a35c088c23713feb7d8c9236cdf6
Author: Dan Jacques <dnj@chromium.org>
Date: Wed May 24 01:42:35 2017

[third_party_packages] Specify unicode size.

Use "ucs4" on Linux Python, and "ucs2" on Mac Python. This parallels
typical (and production) system configurations.

TBR=iannucci@chromium.org
BUG= chromium:714852 
TEST=expectations

[third_party_packages] Actually include the patch.

Include the patch referenced in:
34399778c588f197bd7c7164ad4e90c3e8bda17d :/

TBR=iannucci@chromium.org
BUG= chromium:714852 
TEST=None

Change-Id: Ic4acc8639efb8c46e74025373a047582ae31af7c
Reviewed-on: https://chromium-review.googlesource.com/513507
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/7a3da2193b02a35c088c23713feb7d8c9236cdf6/recipes/recipes/third_party_packages.expected/new_python_on_linux-amd64.json
[modify] https://crrev.com/7a3da2193b02a35c088c23713feb7d8c9236cdf6/recipes/recipes/third_party_packages.expected/new_python_on_linux-386.json
[modify] https://crrev.com/7a3da2193b02a35c088c23713feb7d8c9236cdf6/recipes/recipes/third_party_packages.py
[modify] https://crrev.com/7a3da2193b02a35c088c23713feb7d8c9236cdf6/recipes/recipes/third_party_packages.expected/mac_failure.json
[modify] https://crrev.com/7a3da2193b02a35c088c23713feb7d8c9236cdf6/recipes/recipes/third_party_packages.expected/mac_specific_git_tag.json
[modify] https://crrev.com/7a3da2193b02a35c088c23713feb7d8c9236cdf6/recipes/recipes/third_party_packages.expected/new_git_on_mac-amd64.json
[modify] https://crrev.com/7a3da2193b02a35c088c23713feb7d8c9236cdf6/recipes/recipes/third_party_packages.expected/new_git_on_linux-386.json
[modify] https://crrev.com/7a3da2193b02a35c088c23713feb7d8c9236cdf6/recipes/recipes/third_party_packages.expected/new_git_on_linux-amd64.json
[modify] https://crrev.com/7a3da2193b02a35c088c23713feb7d8c9236cdf6/recipes/recipes/third_party_packages.expected/new_python_on_mac-amd64.json

Project Member

Comment 18 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/39ba9c2d634102bc73ada387819fa10f0c0df72c

commit 39ba9c2d634102bc73ada387819fa10f0c0df72c
Author: Dan Jacques <dnj@chromium.org>
Date: Wed May 24 02:18:25 2017

[third_party_packages] Use symlinks.

Use symlinks instead of hardlinks when packaging Git. This will reduce
the CIPD package size considerably.

TBR=iannucci@chromium.org
BUG= chromium:714852 
TEST=local
  - Ran build locally (64-bit Linux), seems to work fine.

Change-Id: I02ed4c41335cf61732505cdf46de1f48ffba8483
Reviewed-on: https://chromium-review.googlesource.com/513392
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/39ba9c2d634102bc73ada387819fa10f0c0df72c/recipes/recipes/third_party_packages.expected/new_python_on_linux-amd64.json
[modify] https://crrev.com/39ba9c2d634102bc73ada387819fa10f0c0df72c/recipes/recipes/third_party_packages.expected/new_python_on_linux-386.json
[modify] https://crrev.com/39ba9c2d634102bc73ada387819fa10f0c0df72c/recipes/recipes/third_party_packages.py
[modify] https://crrev.com/39ba9c2d634102bc73ada387819fa10f0c0df72c/recipes/recipes/third_party_packages.expected/mac_failure.json
[modify] https://crrev.com/39ba9c2d634102bc73ada387819fa10f0c0df72c/recipes/recipes/third_party_packages.expected/mac_specific_git_tag.json
[modify] https://crrev.com/39ba9c2d634102bc73ada387819fa10f0c0df72c/recipes/recipes/third_party_packages.expected/new_git_on_windows-386.json
[modify] https://crrev.com/39ba9c2d634102bc73ada387819fa10f0c0df72c/recipes/recipes/third_party_packages.expected/new_git_on_mac-amd64.json
[modify] https://crrev.com/39ba9c2d634102bc73ada387819fa10f0c0df72c/recipes/recipes/third_party_packages.expected/new_python_on_windows-amd64.json
[modify] https://crrev.com/39ba9c2d634102bc73ada387819fa10f0c0df72c/recipes/recipes/third_party_packages.expected/new_git_on_linux-386.json
[modify] https://crrev.com/39ba9c2d634102bc73ada387819fa10f0c0df72c/recipes/recipes/third_party_packages.expected/new_git_on_linux-amd64.json
[modify] https://crrev.com/39ba9c2d634102bc73ada387819fa10f0c0df72c/recipes/recipes/third_party_packages.expected/new_python_on_windows-386.json
[modify] https://crrev.com/39ba9c2d634102bc73ada387819fa10f0c0df72c/recipes/recipes/third_party_packages.expected/new_git_on_windows-amd64.json
[modify] https://crrev.com/39ba9c2d634102bc73ada387819fa10f0c0df72c/recipes/recipes/third_party_packages.expected/new_python_on_mac-amd64.json

Project Member

Comment 20 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/19e26e5360a19c02c23c54d5dae045158405dba7

commit 19e26e5360a19c02c23c54d5dae045158405dba7
Author: Dan Jacques <dnj@chromium.org>
Date: Wed May 24 03:09:32 2017

Project Member

Comment 21 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/bbf0e6547274e31f90df400c898a510b30768353

commit bbf0e6547274e31f90df400c898a510b30768353
Author: Dan Jacques <dnj@chromium.org>
Date: Wed May 24 04:50:55 2017

[third_party_packages] Bundle SSL certs.

Now that Python verifies SSL by default, we need to bundle SSL certs
with it and install them on startup.

Use "usercustomize.py" to do this (as per iannucci@'s previous
prototype).

Also support a "dry_run" property b/c I'm sick of commenting out major
parts to test!

TBR=iannucci@chromium.org
BUG= chromium:714852 
TEST=expectations

Change-Id: I6608a9d97c5fa978c94d3e20cfe0011bde61075d
Reviewed-on: https://chromium-review.googlesource.com/513413
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Daniel Jacques <dnj@chromium.org>

[add] https://crrev.com/bbf0e6547274e31f90df400c898a510b30768353/recipes/recipes/third_party_packages.resources/python_usercustomize.py
[add] https://crrev.com/bbf0e6547274e31f90df400c898a510b30768353/recipes/recipes/third_party_packages.expected/dry_run.json
[modify] https://crrev.com/bbf0e6547274e31f90df400c898a510b30768353/recipes/recipes/third_party_packages.expected/new_python_on_linux-amd64.json
[modify] https://crrev.com/bbf0e6547274e31f90df400c898a510b30768353/recipes/recipes/third_party_packages.expected/new_python_on_linux-386.json
[modify] https://crrev.com/bbf0e6547274e31f90df400c898a510b30768353/recipes/recipes/third_party_packages.py
[modify] https://crrev.com/bbf0e6547274e31f90df400c898a510b30768353/recipes/recipes/third_party_packages.expected/mac_failure.json
[modify] https://crrev.com/bbf0e6547274e31f90df400c898a510b30768353/recipes/recipes/third_party_packages.expected/mac_specific_git_tag.json
[modify] https://crrev.com/bbf0e6547274e31f90df400c898a510b30768353/recipes/recipes/third_party_packages.expected/new_git_on_mac-amd64.json
[modify] https://crrev.com/bbf0e6547274e31f90df400c898a510b30768353/recipes/recipes/third_party_packages.expected/new_git_on_linux-386.json
[modify] https://crrev.com/bbf0e6547274e31f90df400c898a510b30768353/recipes/recipes/third_party_packages.expected/new_git_on_linux-amd64.json
[modify] https://crrev.com/bbf0e6547274e31f90df400c898a510b30768353/recipes/recipes/third_party_packages.expected/new_python_on_mac-amd64.json

Project Member

Comment 22 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/0c91d7dcdbd166bc5dad1cd3f48e1cd8a204bced

commit 0c91d7dcdbd166bc5dad1cd3f48e1cd8a204bced
Author: Dan Jacques <dnj@chromium.org>
Date: Wed May 24 05:14:38 2017

Project Member

Comment 23 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/6b3e1444637eff67f010bd7409e03424f0232395

commit 6b3e1444637eff67f010bd7409e03424f0232395
Author: Dan Jacques <dnj@chromium.org>
Date: Wed May 24 18:09:23 2017

[third_party_packages] Disable GECOS in Git.

GECOS is a field in "/etc/passwd" that allows a verbose user name.
Unfortunately, this is (mis-)configured on our bots to include a quote,
and Git identifies that quote as the user name, which promptly fails
a commit author name validity check.

There is little advantage to allowing this obscure mechanism to remain,
both because it is obviously problematic on our fleet and also because
any serious Git committer should have set their own name independently
via "git config".

TBR=iannucci@chromium.org
BUG=chromium:725724,  chromium:714852 
TEST=prod
  - Ran this in a prod environment that was encountering this issue.
    Issue was no longer present in new package.

Change-Id: I75c76e0bcd66b7960149ae3d9fa0cf54d434d8c7
Reviewed-on: https://chromium-review.googlesource.com/513500
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/6b3e1444637eff67f010bd7409e03424f0232395/recipes/recipes/third_party_packages.expected/dry_run.json
[modify] https://crrev.com/6b3e1444637eff67f010bd7409e03424f0232395/recipes/recipes/third_party_packages.expected/new_python_on_linux-amd64.json
[modify] https://crrev.com/6b3e1444637eff67f010bd7409e03424f0232395/recipes/recipes/third_party_packages.expected/new_python_on_linux-386.json
[modify] https://crrev.com/6b3e1444637eff67f010bd7409e03424f0232395/recipes/recipes/third_party_packages.py
[modify] https://crrev.com/6b3e1444637eff67f010bd7409e03424f0232395/recipes/recipes/third_party_packages.expected/mac_failure.json
[modify] https://crrev.com/6b3e1444637eff67f010bd7409e03424f0232395/recipes/recipes/third_party_packages.expected/mac_specific_git_tag.json
[modify] https://crrev.com/6b3e1444637eff67f010bd7409e03424f0232395/recipes/recipes/third_party_packages.expected/new_git_on_windows-386.json
[modify] https://crrev.com/6b3e1444637eff67f010bd7409e03424f0232395/recipes/recipes/third_party_packages.expected/new_git_on_mac-amd64.json
[modify] https://crrev.com/6b3e1444637eff67f010bd7409e03424f0232395/recipes/recipes/third_party_packages.expected/new_python_on_windows-amd64.json
[modify] https://crrev.com/6b3e1444637eff67f010bd7409e03424f0232395/recipes/recipes/third_party_packages.expected/new_git_on_linux-386.json
[modify] https://crrev.com/6b3e1444637eff67f010bd7409e03424f0232395/recipes/recipes/third_party_packages.expected/new_git_on_linux-amd64.json
[modify] https://crrev.com/6b3e1444637eff67f010bd7409e03424f0232395/recipes/recipes/third_party_packages.expected/new_python_on_windows-386.json
[modify] https://crrev.com/6b3e1444637eff67f010bd7409e03424f0232395/recipes/recipes/third_party_packages.expected/new_git_on_windows-amd64.json
[modify] https://crrev.com/6b3e1444637eff67f010bd7409e03424f0232395/recipes/recipes/third_party_packages.expected/new_python_on_mac-amd64.json

Project Member

Comment 24 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/1a36316e97480c4a0b05d6a4fc43a397cd34f05d

commit 1a36316e97480c4a0b05d6a4fc43a397cd34f05d
Author: Dan Jacques <dnj@chromium.org>
Date: Wed May 24 18:36:46 2017

Project Member

Comment 25 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/84dc45cd7d746b77b3c7a6623849780077180c33

commit 84dc45cd7d746b77b3c7a6623849780077180c33
Author: Dan Jacques <dnj@chromium.org>
Date: Wed May 24 19:39:52 2017

Project Member

Comment 26 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/5a6a0ade28b87223188b31253848c27a79dd2f54

commit 5a6a0ade28b87223188b31253848c27a79dd2f54
Author: Dan Jacques <dnj@chromium.org>
Date: Wed May 24 22:38:33 2017

[git] Fix Git wrapper resoltion from symlink.

The Git wrapper will try and resolve CIPD-packaged system Git by using a
directory relative to its executable. However, the current method used
to identify the executable path will resolve symlinks, which will, in
turn, affect the relative directory location.

Change the Git wrapper to safely prefer argv[0] as the path to "self"
instead of the resolved executable path. This will point to the
file/directory that the symlink is in, rather than the executable that
the symlink resolves to, regardless of what "os.Executable" says.

The argv[0] resolution is still checked against the "os.Executable"
resolution via SameFile; argv[0] is only preferred if it is actually the
same file as the resolved executable.

BUG= chromium:714852 
TEST=local
R=iannucci@chromium.org

Change-Id: Id53311661bbe6b53385fe50fbee93850d8de6524
Reviewed-on: https://chromium-review.googlesource.com/513526
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/5a6a0ade28b87223188b31253848c27a79dd2f54/go/src/infra/tools/git/probe_test.go
[modify] https://crrev.com/5a6a0ade28b87223188b31253848c27a79dd2f54/go/src/infra/tools/git/probe.go
[modify] https://crrev.com/5a6a0ade28b87223188b31253848c27a79dd2f54/go/src/infra/tools/git/main.go
[modify] https://crrev.com/5a6a0ade28b87223188b31253848c27a79dd2f54/go/src/infra/tools/git/main_test.go

Project Member

Comment 27 by bugdroid1@chromium.org, May 25 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/72babc157f856a3808e638099226d97a87dee711

commit 72babc157f856a3808e638099226d97a87dee711
Author: Dan Jacques <dnj@google.com>
Date: Thu May 25 02:35:43 2017

[third_party_packages] Match Python parameters, fix Git.

Compile Python with the same flags that are used on Ubuntu Linux and Mac
OSX.

Update Git binary-relative path with a new patch that uses system
facilities to resolve the binary against either:
- /proc/self/exe on Linux
- _NSGetExecutablePath on Mac

This will enable Git to self-resolve even if argv[0] is not a path
to it (e.g., PATH=... git).

TBR=nodir@chromium.org,iannucci@chromium.org
BUG= chromium:714852 
TEST=local

Change-Id: I5dbaf8eb7a2c6ae806922a36adb75bd0f7cea5a0
Reviewed-on: https://chromium-review.googlesource.com/514405
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/72babc157f856a3808e638099226d97a87dee711/recipes/recipes/third_party_packages.expected/dry_run.json
[modify] https://crrev.com/72babc157f856a3808e638099226d97a87dee711/recipes/recipes/third_party_packages.expected/new_python_on_linux-amd64.json
[modify] https://crrev.com/72babc157f856a3808e638099226d97a87dee711/recipes/recipes/third_party_packages.expected/new_python_on_linux-386.json
[modify] https://crrev.com/72babc157f856a3808e638099226d97a87dee711/recipes/recipes/third_party_packages.py
[modify] https://crrev.com/72babc157f856a3808e638099226d97a87dee711/recipes/recipes/third_party_packages.expected/mac_failure.json
[modify] https://crrev.com/72babc157f856a3808e638099226d97a87dee711/recipes/recipes/third_party_packages.expected/mac_specific_git_tag.json
[modify] https://crrev.com/72babc157f856a3808e638099226d97a87dee711/recipes/recipes/third_party_packages.resources/git_2_13_0.posix.patch
[modify] https://crrev.com/72babc157f856a3808e638099226d97a87dee711/recipes/recipes/third_party_packages.expected/new_git_on_mac-amd64.json
[modify] https://crrev.com/72babc157f856a3808e638099226d97a87dee711/recipes/recipes/third_party_packages.expected/new_git_on_linux-386.json
[modify] https://crrev.com/72babc157f856a3808e638099226d97a87dee711/recipes/recipes/third_party_packages.expected/new_git_on_linux-amd64.json
[modify] https://crrev.com/72babc157f856a3808e638099226d97a87dee711/recipes/recipes/third_party_packages.expected/new_python_on_mac-amd64.json

Project Member

Comment 28 by bugdroid1@chromium.org, May 25 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/658e904ffb62c2534100e92b5b5bc94b73be4de0

commit 658e904ffb62c2534100e92b5b5bc94b73be4de0
Author: Dan Jacques <dnj@chromium.org>
Date: Thu May 25 02:41:35 2017

[third_party_packages] Increase Git version.

(Forgot to do this in preceding patch).

NOTRY=true
NOPRESUBMIT=true
TBR=nodir@chromium.org
BUG= chromium:714852 
TEST=expectations

Change-Id: I1e33b6ab429e2ca3c9fa94b1ec809c9d8b0a76cb
Reviewed-on: https://chromium-review.googlesource.com/514529
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/658e904ffb62c2534100e92b5b5bc94b73be4de0/recipes/recipes/third_party_packages.expected/dry_run.json
[modify] https://crrev.com/658e904ffb62c2534100e92b5b5bc94b73be4de0/recipes/recipes/third_party_packages.expected/new_python_on_linux-amd64.json
[modify] https://crrev.com/658e904ffb62c2534100e92b5b5bc94b73be4de0/recipes/recipes/third_party_packages.expected/new_python_on_linux-386.json
[modify] https://crrev.com/658e904ffb62c2534100e92b5b5bc94b73be4de0/recipes/recipes/third_party_packages.py
[modify] https://crrev.com/658e904ffb62c2534100e92b5b5bc94b73be4de0/recipes/recipes/third_party_packages.expected/mac_failure.json
[modify] https://crrev.com/658e904ffb62c2534100e92b5b5bc94b73be4de0/recipes/recipes/third_party_packages.expected/mac_specific_git_tag.json
[modify] https://crrev.com/658e904ffb62c2534100e92b5b5bc94b73be4de0/recipes/recipes/third_party_packages.expected/new_git_on_windows-386.json
[modify] https://crrev.com/658e904ffb62c2534100e92b5b5bc94b73be4de0/recipes/recipes/third_party_packages.expected/new_git_on_mac-amd64.json
[modify] https://crrev.com/658e904ffb62c2534100e92b5b5bc94b73be4de0/recipes/recipes/third_party_packages.expected/new_python_on_windows-amd64.json
[modify] https://crrev.com/658e904ffb62c2534100e92b5b5bc94b73be4de0/recipes/recipes/third_party_packages.expected/new_git_on_linux-386.json
[modify] https://crrev.com/658e904ffb62c2534100e92b5b5bc94b73be4de0/recipes/recipes/third_party_packages.expected/new_git_on_linux-amd64.json
[modify] https://crrev.com/658e904ffb62c2534100e92b5b5bc94b73be4de0/recipes/recipes/third_party_packages.expected/new_python_on_windows-386.json
[modify] https://crrev.com/658e904ffb62c2534100e92b5b5bc94b73be4de0/recipes/recipes/third_party_packages.expected/new_git_on_windows-amd64.json
[modify] https://crrev.com/658e904ffb62c2534100e92b5b5bc94b73be4de0/recipes/recipes/third_party_packages.expected/new_python_on_mac-amd64.json

Project Member

Comment 29 by bugdroid1@chromium.org, May 25 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/57ade1dfa66d1480248906ed160f132722d3940a

commit 57ade1dfa66d1480248906ed160f132722d3940a
Author: Dan Jacques <dnj@chromium.org>
Date: Thu May 25 04:00:17 2017

Project Member

Comment 30 by bugdroid1@chromium.org, May 25 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/f63f10898e6e2726dc9504acc95137dbac19342a

commit f63f10898e6e2726dc9504acc95137dbac19342a
Author: Dan Jacques <dnj@chromium.org>
Date: Thu May 25 04:25:10 2017

Project Member

Comment 31 by bugdroid1@chromium.org, May 27 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/b9a08eec4868d1f256be926a8f1d9a3486e3090f

commit b9a08eec4868d1f256be926a8f1d9a3486e3090f
Author: Dan Jacques <dnj@chromium.org>
Date: Sat May 27 02:22:52 2017

[third_party_packages] Update Python/Git bundles.

Several updates to Python/Git bundling.

Python
======

- By default, fold in system installed package paths.
  On Linux systems, the bundled Python will ignore installed
  "dist-packages" (Ubuntu / Debian) in favor of its local bundled
  packages. While this is good for hermetic boundaries, it violates user
  expectations that installed packages will be available to the local
  Python interpreter. This, in turn, breaks things such as "gsutil" on
  GCE, which expects the local system installed "google-cloud-platform"
  package to be available.

- Use system SSL certificates instead of bundling.
  On Mac, this means that we dynamically link against the system's
  OpenSSL. On Linux, this means that we append logic to the end of
  "ssl.py" to locate and use the system's certificate authority bundle
  from a known path.

- Fix SSL integration from VirtualEnv by augmenting "ssl.py" directly in
  Linux builds instead of "usercustomize.py", which does not get
  propagated into a VirtualEnv.

- Add a "python_test.py" to exercise a Python package against known
  flaws.

Git
===

- Updated the Git patch to fix a bug where the environment variable export
  was having an extra path element truncated and, consqeuently, falling
  back to static prefix resolution.
- Add a "git_test.py" to exercise a Git package against known flaws.

Recipe
======

To support these changes, the internal package construction logic has
been updated to more formally describe the packages and their
dependencies. Each package is also deployed into its own PREFIX so
includes and exports from one package don't encourage the build to pull
in other packages unintentionally. This was necessary to link against
Mac's OpenSSL.

TODO: Integrate the tests into the actual CI in a later CL.

TBR=iannucci@chromium.org
BUG= chromium:714852 
TEST=expectations,local
  - Tested on local Linux and Mac, working as expected.
  - Tested in 64-bit Mac and 32-bit and 64-bit Linux builders.

Change-Id: I75512f8b36c8b68ad8a87cc9d238fbc55363ec94
Reviewed-on: https://chromium-review.googlesource.com/514983
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.resources/python_usercustomize.py
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.resources/git_2_13_0.posix.patch
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.expected/dry_run.json
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.expected/new_python_on_linux-amd64.json
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.expected/new_python_on_linux-386.json
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.py
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.expected/mac_failure.json
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.expected/mac_specific_git_tag.json
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.expected/new_git_on_windows-386.json
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.expected/new_git_on_mac-amd64.json
[add] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.resources/python_ssl_suffix.py
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.expected/new_python_on_windows-amd64.json
[add] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.resources/git_test.py
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.expected/new_git_on_linux-386.json
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.expected/new_git_on_linux-amd64.json
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.expected/new_python_on_windows-386.json
[add] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.resources/python_test.py
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.expected/new_git_on_windows-amd64.json
[modify] https://crrev.com/b9a08eec4868d1f256be926a8f1d9a3486e3090f/recipes/recipes/third_party_packages.expected/new_python_on_mac-amd64.json

Project Member

Comment 32 by bugdroid1@chromium.org, May 27 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/53b2e72131a462de99966e0bb8ce9131892dc9b9

commit 53b2e72131a462de99966e0bb8ce9131892dc9b9
Author: Dan Jacques <dnj@chromium.org>
Date: Sat May 27 03:20:33 2017

Project Member

Comment 33 by bugdroid1@chromium.org, May 27 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/31c8716f2a3af7b443b0ded7be3869b39c94fdc8

commit 31c8716f2a3af7b443b0ded7be3869b39c94fdc8
Author: Dan Jacques <dnj@chromium.org>
Date: Sat May 27 04:20:04 2017

Project Member

Comment 34 by bugdroid1@chromium.org, May 27 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/fbc8f23d0987a0cd61b89408892a1f31a04d3789

commit fbc8f23d0987a0cd61b89408892a1f31a04d3789
Author: Dan Jacques <dnj@chromium.org>
Date: Sat May 27 04:54:13 2017

Project Member

Comment 35 by bugdroid1@chromium.org, May 27 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/343bdd4c8b51fc87ddd2b6094713ef6a0219f004

commit 343bdd4c8b51fc87ddd2b6094713ef6a0219f004
Author: Dan Jacques <dnj@google.com>
Date: Sat May 27 16:08:09 2017

[third_party_packages] Install Git via "copy".

Install the "git" package via CIPD "copy" mode (instead of default,
which is symlink on UNIX platforms).

When Git initializes a new repository, it uses files from its "templates"
directory to seed it by copying those files verbatim. If those files
happen to be symlinks, it will just copy that symlink, at worst dropping it
into a location where it points to junk and at best pointing back to the
ioriginal template, both of which are invalid since they are supposed to
be user-editable. Other uses of templates are similarly broken.

TBR=nodir@chromium.org
BUG= chromium:714852 
TEST=expectations,local
  - Reproduced problem locally, confirmed that copy-mode does not
    encounter this problem.

Change-Id: Iebf0f5247bb1b0a3414a59fadf7efdb4902cfca8
Reviewed-on: https://chromium-review.googlesource.com/517354
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/343bdd4c8b51fc87ddd2b6094713ef6a0219f004/recipes/recipes/third_party_packages.expected/dry_run.json
[modify] https://crrev.com/343bdd4c8b51fc87ddd2b6094713ef6a0219f004/recipes/recipes/third_party_packages.expected/new_python_on_linux-amd64.json
[modify] https://crrev.com/343bdd4c8b51fc87ddd2b6094713ef6a0219f004/recipes/recipes/third_party_packages.expected/new_python_on_linux-386.json
[modify] https://crrev.com/343bdd4c8b51fc87ddd2b6094713ef6a0219f004/recipes/recipes/third_party_packages.py
[modify] https://crrev.com/343bdd4c8b51fc87ddd2b6094713ef6a0219f004/recipes/recipes/third_party_packages.expected/mac_failure.json
[modify] https://crrev.com/343bdd4c8b51fc87ddd2b6094713ef6a0219f004/recipes/recipes/third_party_packages.expected/mac_specific_git_tag.json
[modify] https://crrev.com/343bdd4c8b51fc87ddd2b6094713ef6a0219f004/recipes/recipes/third_party_packages.expected/new_git_on_windows-386.json
[modify] https://crrev.com/343bdd4c8b51fc87ddd2b6094713ef6a0219f004/recipes/recipes/third_party_packages.expected/new_git_on_mac-amd64.json
[modify] https://crrev.com/343bdd4c8b51fc87ddd2b6094713ef6a0219f004/recipes/recipes/third_party_packages.expected/new_python_on_windows-amd64.json
[modify] https://crrev.com/343bdd4c8b51fc87ddd2b6094713ef6a0219f004/recipes/recipes/third_party_packages.expected/new_git_on_linux-386.json
[modify] https://crrev.com/343bdd4c8b51fc87ddd2b6094713ef6a0219f004/recipes/recipes/third_party_packages.expected/new_git_on_linux-amd64.json
[modify] https://crrev.com/343bdd4c8b51fc87ddd2b6094713ef6a0219f004/recipes/recipes/third_party_packages.expected/new_python_on_windows-386.json
[modify] https://crrev.com/343bdd4c8b51fc87ddd2b6094713ef6a0219f004/recipes/recipes/third_party_packages.expected/new_git_on_windows-amd64.json
[modify] https://crrev.com/343bdd4c8b51fc87ddd2b6094713ef6a0219f004/recipes/recipes/third_party_packages.expected/new_python_on_mac-amd64.json

Project Member

Comment 36 by bugdroid1@chromium.org, May 27 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/d6a90978d009a197acb012874228a9960d431d71

commit d6a90978d009a197acb012874228a9960d431d71
Author: Dan Jacques <dnj@chromium.org>
Date: Sat May 27 16:37:41 2017

Project Member

Comment 37 by bugdroid1@chromium.org, May 27 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/2f91f4f7a57d72d8e9a2c416194d62d86a8a9ca6

commit 2f91f4f7a57d72d8e9a2c416194d62d86a8a9ca6
Author: Dan Jacques <dnj@chromium.org>
Date: Sat May 27 17:06:48 2017

Project Member

Comment 38 by bugdroid1@chromium.org, May 28 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/537aec6b9029efebf8fa37b55c5bd90f16507a6a

commit 537aec6b9029efebf8fa37b55c5bd90f16507a6a
Author: Dan Jacques <dnj@chromium.org>
Date: Sun May 28 15:09:52 2017

Project Member

Comment 39 by bugdroid1@chromium.org, May 28 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/cfa26968c49a2b22729999e039cde4ecfe8dd258

commit cfa26968c49a2b22729999e039cde4ecfe8dd258
Author: Dan Jacques <dnj@chromium.org>
Date: Sun May 28 18:23:19 2017

[third_party_packages] Python copy install mode.

No specific motivation for this, but for something systemic like Python,
it is probably more conformant to copy install instead of symlink.

TBR=nodir@chromium.org
BUG= chromium:714852 
TEST=expectations

Change-Id: I2d2275f3fc7e981f9d1e18b0330eafb2669c2f5a
Reviewed-on: https://chromium-review.googlesource.com/517723
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/cfa26968c49a2b22729999e039cde4ecfe8dd258/recipes/recipes/third_party_packages.expected/dry_run.json
[modify] https://crrev.com/cfa26968c49a2b22729999e039cde4ecfe8dd258/recipes/recipes/third_party_packages.expected/new_python_on_linux-amd64.json
[modify] https://crrev.com/cfa26968c49a2b22729999e039cde4ecfe8dd258/recipes/recipes/third_party_packages.expected/new_python_on_linux-386.json
[modify] https://crrev.com/cfa26968c49a2b22729999e039cde4ecfe8dd258/recipes/recipes/third_party_packages.py
[modify] https://crrev.com/cfa26968c49a2b22729999e039cde4ecfe8dd258/recipes/recipes/third_party_packages.expected/mac_failure.json
[modify] https://crrev.com/cfa26968c49a2b22729999e039cde4ecfe8dd258/recipes/recipes/third_party_packages.expected/mac_specific_git_tag.json
[modify] https://crrev.com/cfa26968c49a2b22729999e039cde4ecfe8dd258/recipes/recipes/third_party_packages.expected/new_git_on_mac-amd64.json
[modify] https://crrev.com/cfa26968c49a2b22729999e039cde4ecfe8dd258/recipes/recipes/third_party_packages.expected/new_git_on_linux-386.json
[modify] https://crrev.com/cfa26968c49a2b22729999e039cde4ecfe8dd258/recipes/recipes/third_party_packages.expected/new_git_on_linux-amd64.json
[modify] https://crrev.com/cfa26968c49a2b22729999e039cde4ecfe8dd258/recipes/recipes/third_party_packages.expected/new_python_on_mac-amd64.json

Project Member

Comment 40 by bugdroid1@chromium.org, May 28 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/ca5e648193719ab5768ec98883624f69043886d1

commit ca5e648193719ab5768ec98883624f69043886d1
Author: Dan Jacques <dnj@chromium.org>
Date: Sun May 28 20:35:41 2017

Project Member

Comment 41 by bugdroid1@chromium.org, Jun 1 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/19c1c1166bd8a26a067f6a32ab6d29bfa585c330

commit 19c1c1166bd8a26a067f6a32ab6d29bfa585c330
Author: Dan Jacques <dnj@chromium.org>
Date: Thu Jun 01 21:48:29 2017

Blocking: 731573

Comment 43 by no...@chromium.org, Jun 12 2017

Blocking: -731573
Labels: -Pri-1 Pri-2
Owner: ----
Status: Available (was: Started)
dnj@ has made a git package at infra/git/* [1] and is deployed to all LUCI builds [2]
Packaging Python turned out to be difficult and we've solved the underlying problem ( bug 731573 ) in a different way, so this bug is not a priority for us anymore.

[1]: $ cipd ls infra/git/
infra/git/linux-386
infra/git/linux-amd64
infra/git/mac-amd64
infra/git/windows-386
infra/git/windows-amd64
[2]: https://chrome-internal.googlesource.com/infradata/config/+/96dc147/configs/cr-buildbucket/swarming_task_template.json#40
Project Member

Comment 44 by bugdroid1@chromium.org, Nov 10 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/f6afc375f6b449c68a25bffb9c56afa77ca1b705

commit f6afc375f6b449c68a25bffb9c56afa77ca1b705
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Nov 10 22:09:16 2017

3p package builder: send email to troopers

On failures, send emails to troopers instead of luci-eng. Motivation is that
luci-eng currently does not allow buildbot to post messages, but also troopers
is a better recepient.

R=iannucci@chromium.org
BUG= 714852 

Change-Id: Id1bca2490dd5c988d2f2e7f6eaea1967e08f2421
Reviewed-on: https://chromium-review.googlesource.com/493875
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/f6afc375f6b449c68a25bffb9c56afa77ca1b705/masters/master.chromium.infra.cron/master.cfg

Labels: -Pri-2 cit-pm-47 Pri-3
Status: Fixed (was: Available)
git and python are deployed via CIPD on LUCI.

Sign in to add a comment