New issue
Advanced search Search tips

Issue 714293 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 731573
Owner:
Closed: Jun 2017
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocked on:
issue 714852
issue 722476
issue 723763
issue 724902
issue 727917



Sign in to add a comment

make win_chromium_rel_ng WAI on LUCI

Project Member Reported by no...@chromium.org, Apr 21 2017

Issue description

Make win_chromium_rel_ng on LUCI correct and fast.


 

Comment 1 by no...@chromium.org, Apr 21 2017

Components: Infra>Platform
Labels: -Pri-3 luci Pri-1
Owner: no...@chromium.org
Status: Started (was: Untriaged)

Comment 2 by no...@chromium.org, Apr 21 2017

So far it is incorrect because depot_tools in $PATH is old. We insert therecipe-pinned depot_tools into the beginning of $PATH because there is code that expects depot_tools in $PATH to contain python.exe.

To mitigate that, I've started working on a builder that prepares CIPD packages for Git and Python, so we can deploy them separately, not through depot_tools. Once done, I will try to insert reicpe-pinned depot_tools into the beginning of $PATH again.

[1]: https://luci-logdog.appspot.com/v/?s=chromium%2Fswarm%2Fchromium-swarm.appspot.com%2F35ab4896f2dcf011%2F%2B%2Fsteps%2Fgclient_runhooks__with_patch_%2F0%2Fstdout

Comment 3 by no...@chromium.org, Apr 21 2017

Blockedon: 714297

Comment 4 by no...@chromium.org, Apr 24 2017

Blockedon: 714852

Comment 5 by no...@chromium.org, Apr 24 2017

Blockedon: -714297

Comment 6 by no...@chromium.org, Apr 24 2017

I've moved the python/git bootstrapping work into  bug 714852 

Comment 7 by efoo@chromium.org, May 6 2017

Labels: LUCI-M2-Dev2 REQBY-LUCI-M3-ClosedBeta
Blockedon: 722476

Comment 9 by d...@chromium.org, May 17 2017

Blockedon: 723763
Project Member

Comment 10 by bugdroid1@chromium.org, May 17 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/5e7db1588c5a375b1e77c074896ec1d28e5c2717

commit 5e7db1588c5a375b1e77c074896ec1d28e5c2717
Author: Dan Jacques <dnj@chromium.org>
Date: Wed May 17 21:35:39 2017

Bootstrap 'gsutil_cp_dir.py'.

Bootstrap 'gsutil_cp_dir_py' through "runit.py" so it picks up
PYTHONPATHs.

BUG= chromium:714293 
TEST=expectations
R=nodir@chromium.org

Change-Id: Ibe511d977b302d7802b8f9bd172a6947912ad68e
Reviewed-on: https://chromium-review.googlesource.com/508108
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/5e7db1588c5a375b1e77c074896ec1d28e5c2717/scripts/slave/recipes/syzygy/continuous.expected/full_master_client_syzygy_Syzygy_Official.json
[modify] https://crrev.com/5e7db1588c5a375b1e77c074896ec1d28e5c2717/scripts/slave/recipe_modules/syzygy/__init__.py
[modify] https://crrev.com/5e7db1588c5a375b1e77c074896ec1d28e5c2717/scripts/slave/recipe_modules/syzygy/example.expected/full_master_client_syzygy_Syzygy_Official.json
[modify] https://crrev.com/5e7db1588c5a375b1e77c074896ec1d28e5c2717/scripts/slave/recipe_modules/syzygy/api.py
[modify] https://crrev.com/5e7db1588c5a375b1e77c074896ec1d28e5c2717/scripts/slave/recipes/syzygy/coverage.expected/full_master_client_syzygy_Syzygy_Coverage.json
[modify] https://crrev.com/5e7db1588c5a375b1e77c074896ec1d28e5c2717/scripts/slave/recipe_modules/syzygy/example.expected/full_master_client_syzygy_Syzygy_Coverage.json
[modify] https://crrev.com/5e7db1588c5a375b1e77c074896ec1d28e5c2717/scripts/slave/recipes/syzygy/kasko_official.expected/full_master_client_syzygy_Kasko_Official.json

Comment 11 by d...@chromium.org, May 22 2017

Owner: d...@chromium.org
The current failure is: 

  File "c:\b\swarm_slave\w\ir\kitchen-checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_engine\fetch.py", line 245, in _git
    raise GitError('Git "%s" failed: %s' % (subcommand, e.message,))
GitError: Git "diff-tree" failed: 
Uncaught exception (GitError): Git "diff-tree" failed: 

This is also believed to be due to a too-old system Git.

PATH=c:\b\swarm_slave\w\ir\cipd_bin_packages;c:\b\swarm_slave\cipd_cache\bin;c:\b\depot_tools\python276_bin;c:\b\depot_tools\python276_bin\Scripts;C:\ProgramData\chocolatey\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;c:\tools;c:\b\depot_tools;C:\Program Files\Dell\SysMgt\oma\bin;C:\Program Files\Dell\SysMgt\shared\bin;C:\Program Files\Dell\SysMgt\idrac;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files (x86)\Microsoft SDKs\TypeScript\1.0\;C:\Program Files (x86)\AMD\ATI.ACE\Core-Static;C:\Program Files (x86)\QuickTime\QTSystem\;C:\Program Files\Puppet Labs\Puppet\bin

Plan:
1) Use "try-recipe" to demonstrate successful CIPD Git/Python injection.
2) Implement this on dev -> canary -> full.

Comment 12 by d...@chromium.org, May 22 2017

Blockedon: 724902
Project Member

Comment 13 by bugdroid1@chromium.org, May 22 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/560feed612418784554f1c7d825add2aeb336874

commit 560feed612418784554f1c7d825add2aeb336874
Author: Dan Jacques <dnj@chromium.org>
Date: Mon May 22 15:59:42 2017

[git] Add self-relative PATH overrides.

Currently, the Git wrapper searches for Git by probing the system PATH.
When the Git wrapper is bundled with Git, we want it to always use that
Git regardless of PATH. We accomplish this by configuring the wrapper to
prefer Git in directories relative to its executable.

The current Git deployment places Git in the "bin/" subdirectory
relative to the CIPD root, so we will hard-code that into the wrapper.
On existing systems, no Git will exist at this path, so this will have
no effect.

BUG= chromium:714293 
TEST=unit
R=iannucci@chromium.org, nodir@chromium.org

Change-Id: Ifdabe84349fd89d44b981d39aa22de648710bced
Reviewed-on: https://chromium-review.googlesource.com/510702
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/560feed612418784554f1c7d825add2aeb336874/go/src/infra/tools/git/probe_test.go
[modify] https://crrev.com/560feed612418784554f1c7d825add2aeb336874/go/src/infra/tools/git/main.go
[modify] https://crrev.com/560feed612418784554f1c7d825add2aeb336874/go/src/infra/tools/git/probe.go

Comment 14 by efoo@chromium.org, May 23 2017

Labels: LUCI-M1-S2
Project Member

Comment 15 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/1b9a43aab385070fd75263a9e0f4b4312b5f510e

commit 1b9a43aab385070fd75263a9e0f4b4312b5f510e
Author: Dan Jacques <dnj@chromium.org>
Date: Wed May 24 22:22:52 2017

Remove special Cygwin PATH manipulation from .bat.

Several boilerplate batch files include a provision to prepend
"depot_tools" to PATH prior to running those tools. This undermines
the utility of PATH overrides, since these tools specifically force their
"depot_tools" sub-paths to be used regardless of environment.

The origin of this behavior is likely limited to a specific fix for a
specific problem, but was then perpetuated by the copy/paste of
boilerplate bootstrap code as more bootstraps were added.

This is important in upcoming configurations, where core tools such as
Python and Git will be overridden via PATH on the bots.

Cygwin users who depended on this behavior should just add "depot_tools"
to their PATH in the appropriate location (i.e. in their .bashrc).

BUG= chromium:714293 ,  chromium:724902 
TEST=None

Change-Id: Ie948a430847d20326d2411e9296cacd02f83a537
Reviewed-on: https://chromium-review.googlesource.com/510290
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/1b9a43aab385070fd75263a9e0f4b4312b5f510e/depot-tools-auth.bat
[modify] https://crrev.com/1b9a43aab385070fd75263a9e0f4b4312b5f510e/cit.bat
[modify] https://crrev.com/1b9a43aab385070fd75263a9e0f4b4312b5f510e/gn.bat
[modify] https://crrev.com/1b9a43aab385070fd75263a9e0f4b4312b5f510e/clang-format.bat
[modify] https://crrev.com/1b9a43aab385070fd75263a9e0f4b4312b5f510e/fetch.bat
[modify] https://crrev.com/1b9a43aab385070fd75263a9e0f4b4312b5f510e/commit_queue.bat
[modify] https://crrev.com/1b9a43aab385070fd75263a9e0f4b4312b5f510e/clang_format_merge_driver.bat
[modify] https://crrev.com/1b9a43aab385070fd75263a9e0f4b4312b5f510e/roll-dep-svn.bat
[modify] https://crrev.com/1b9a43aab385070fd75263a9e0f4b4312b5f510e/roll-dep.bat
[modify] https://crrev.com/1b9a43aab385070fd75263a9e0f4b4312b5f510e/apply_issue.bat
[modify] https://crrev.com/1b9a43aab385070fd75263a9e0f4b4312b5f510e/gclient.bat
[modify] https://crrev.com/1b9a43aab385070fd75263a9e0f4b4312b5f510e/cpplint.bat
[modify] https://crrev.com/1b9a43aab385070fd75263a9e0f4b4312b5f510e/download_from_google_storage.bat

Comment 16 by d...@chromium.org, May 30 2017

Blockedon: 727917
Project Member

Comment 17 by bugdroid1@chromium.org, Jun 1 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d

commit 2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d
Author: Dan Jacques <dnj@chromium.org>
Date: Thu Jun 01 19:48:49 2017

[.bat] Remove "depot_tools" override.

Currently, all ".bat" entry points use "~dp0python" to ensure that the
Python that is used to execute the tool is the one in depot_tools. This
prevents any sort of system override.

Remove this override so that PATH solely determines which Python is
used. To accommodate users who invoked these tools without Python on the
PATH, we still still add "depot_tools" as a catch-all PATH suffix.

Some tools were also not using DOS-style line endings. This CL fixes
this.

BUG= chromium:714293 ,  chromium:724902 
TEST=None

Change-Id: I06e9583a668c767196a2a335547aded868f2a2b5
Reviewed-on: https://chromium-review.googlesource.com/517236
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d/depot-tools-auth.bat
[modify] https://crrev.com/2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d/cit.bat
[modify] https://crrev.com/2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d/gn.bat
[modify] https://crrev.com/2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d/clang-format.bat
[modify] https://crrev.com/2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d/fetch.bat
[modify] https://crrev.com/2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d/commit_queue.bat
[modify] https://crrev.com/2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d/clang_format_merge_driver.bat
[modify] https://crrev.com/2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d/roll-dep-svn.bat
[modify] https://crrev.com/2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d/roll-dep.bat
[modify] https://crrev.com/2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d/apply_issue.bat
[modify] https://crrev.com/2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d/gclient.bat
[modify] https://crrev.com/2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d/cpplint.bat
[modify] https://crrev.com/2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d/download_from_google_storage.bat

Project Member

Comment 18 by bugdroid1@chromium.org, Jun 1 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/96fa295ab453feee177e3b6cf0992d576cf87af5

commit 96fa295ab453feee177e3b6cf0992d576cf87af5
Author: Daniel Jacques <dnj@chromium.org>
Date: Thu Jun 01 20:44:29 2017

Revert "[.bat] Remove "depot_tools" override."

This reverts commit 2f5f0b7a99c9df653b090e3ef00d13a2cab00a8d.

Reason for revert: <INSERT REASONING HERE>

Original change's description:
> [.bat] Remove "depot_tools" override.
> 
> Currently, all ".bat" entry points use "~dp0python" to ensure that the
> Python that is used to execute the tool is the one in depot_tools. This
> prevents any sort of system override.
> 
> Remove this override so that PATH solely determines which Python is
> used. To accommodate users who invoked these tools without Python on the
> PATH, we still still add "depot_tools" as a catch-all PATH suffix.
> 
> Some tools were also not using DOS-style line endings. This CL fixes
> this.
> 
> BUG= chromium:714293 ,  chromium:724902 
> TEST=None
> 
> Change-Id: I06e9583a668c767196a2a335547aded868f2a2b5
> Reviewed-on: https://chromium-review.googlesource.com/517236
> Commit-Queue: Daniel Jacques <dnj@chromium.org>
> Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
> 

TBR=maruel@chromium.org,iannucci@chromium.org,vadimsh@chromium.org,dnj@chromium.org
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
BUG= chromium:714293 ,  chromium:724902 

Change-Id: I822abdd4e02abd32d2f4789fb16d5a7f78fdd02d
Reviewed-on: https://chromium-review.googlesource.com/521867
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/96fa295ab453feee177e3b6cf0992d576cf87af5/depot-tools-auth.bat
[modify] https://crrev.com/96fa295ab453feee177e3b6cf0992d576cf87af5/cit.bat
[modify] https://crrev.com/96fa295ab453feee177e3b6cf0992d576cf87af5/gn.bat
[modify] https://crrev.com/96fa295ab453feee177e3b6cf0992d576cf87af5/clang-format.bat
[modify] https://crrev.com/96fa295ab453feee177e3b6cf0992d576cf87af5/fetch.bat
[modify] https://crrev.com/96fa295ab453feee177e3b6cf0992d576cf87af5/commit_queue.bat
[modify] https://crrev.com/96fa295ab453feee177e3b6cf0992d576cf87af5/clang_format_merge_driver.bat
[modify] https://crrev.com/96fa295ab453feee177e3b6cf0992d576cf87af5/roll-dep-svn.bat
[modify] https://crrev.com/96fa295ab453feee177e3b6cf0992d576cf87af5/roll-dep.bat
[modify] https://crrev.com/96fa295ab453feee177e3b6cf0992d576cf87af5/apply_issue.bat
[modify] https://crrev.com/96fa295ab453feee177e3b6cf0992d576cf87af5/gclient.bat
[modify] https://crrev.com/96fa295ab453feee177e3b6cf0992d576cf87af5/cpplint.bat
[modify] https://crrev.com/96fa295ab453feee177e3b6cf0992d576cf87af5/download_from_google_storage.bat

Project Member

Comment 19 by bugdroid1@chromium.org, Jun 1 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/74809c1b318e7ae7cd472990728e76c8b5816f4c

commit 74809c1b318e7ae7cd472990728e76c8b5816f4c
Author: Dan Jacques <dnj@chromium.org>
Date: Thu Jun 01 21:05:21 2017

[.bat] Remove "depot_tools" override. (#2)

Second attempt at landing. Fix quotes around python in "gclient.bat".

Currently, all ".bat" entry points use "~dp0python" to ensure that the
Python that is used to execute the tool is the one in depot_tools. This
prevents any sort of system override.

Remove this override so that PATH solely determines which Python is
used. To accommodate users who invoked these tools without Python on the
PATH, we still still add "depot_tools" as a catch-all PATH suffix.

Some tools were also not using DOS-style line endings. This CL fixes
this.

BUG= chromium:714293 ,  chromium:724902 
TEST=None

Change-Id: I0fceb99c8adb96e72dac706819be032d400aad37
Reviewed-on: https://chromium-review.googlesource.com/521704
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/depot-tools-auth.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/cit.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/gn.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/clang-format.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/fetch.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/commit_queue.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/clang_format_merge_driver.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/roll-dep-svn.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/roll-dep.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/apply_issue.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/gclient.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/cpplint.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/download_from_google_storage.bat

Project Member

Comment 20 by bugdroid1@chromium.org, Jun 1 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/74809c1b318e7ae7cd472990728e76c8b5816f4c

commit 74809c1b318e7ae7cd472990728e76c8b5816f4c
Author: Dan Jacques <dnj@chromium.org>
Date: Thu Jun 01 21:05:21 2017

[.bat] Remove "depot_tools" override. (#2)

Second attempt at landing. Fix quotes around python in "gclient.bat".

Currently, all ".bat" entry points use "~dp0python" to ensure that the
Python that is used to execute the tool is the one in depot_tools. This
prevents any sort of system override.

Remove this override so that PATH solely determines which Python is
used. To accommodate users who invoked these tools without Python on the
PATH, we still still add "depot_tools" as a catch-all PATH suffix.

Some tools were also not using DOS-style line endings. This CL fixes
this.

BUG= chromium:714293 ,  chromium:724902 
TEST=None

Change-Id: I0fceb99c8adb96e72dac706819be032d400aad37
Reviewed-on: https://chromium-review.googlesource.com/521704
Commit-Queue: Daniel Jacques <dnj@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Scott Graham <scottmg@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/depot-tools-auth.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/cit.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/gn.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/clang-format.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/fetch.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/commit_queue.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/clang_format_merge_driver.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/roll-dep-svn.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/roll-dep.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/apply_issue.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/gclient.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/cpplint.bat
[modify] https://crrev.com/74809c1b318e7ae7cd472990728e76c8b5816f4c/download_from_google_storage.bat

Comment 21 by efoo@chromium.org, Jun 9 2017

Labels: LUCI-M1-S3
Mergedinto: 731573
Status: Duplicate (was: Started)
Let's track the progress there now

Comment 23 by efoo@chromium.org, Nov 8 2017

Labels: -REQBY-LUCI-M3-ClosedBeta REQBY-LUCI-M3-Beta

Comment 24 by efoo@chromium.org, Jan 31 2018

Labels: LUCI-Dev2

Comment 25 by efoo@chromium.org, Jan 31 2018

Labels: -luci -LUCI-M2-Dev2

Comment 26 by efoo@chromium.org, Feb 13 2018

Labels: -REQBY-LUCI-M3-Beta REQBY-LUCI-Beta

Sign in to add a comment