New issue
Advanced search Search tips

Issue 855137 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2018
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

stop doing shallow checkouts in bot_update

Project Member Reported by tandrii@chromium.org, Jun 21 2018

Issue description

aka bigger harddrives are cheaper than wasted human time, network bandwidth and cpu of Git servers.
 
Project Member

Comment 1 by bugdroid1@chromium.org, Jun 21 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/124365b8ea553111c78af62c0d7455423a2238e5

commit 124365b8ea553111c78af62c0d7455423a2238e5
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Jun 21 20:01:20 2018

bot_update: default to non-shallow checkouts.

shallow mode is a special git mode that only checks out a shallow copy
of git so the full history is not downloaded. It is not well supported
by googlesource.com servers because git has to do a giant negotiation
with the server and transmit every single commit that the local checkout
has or doesn't have (instead of just the latest commit of each ref),
which ends up taking more than 10x longer than a non-shallow checkout.

R=ehmaldonado, hinoka, iannucci

Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: build
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Bug:  855137 
Change-Id: I5f9e31f8b2730cf80b79bfd0e08201c33eb942f5
Reviewed-on: https://chromium-review.googlesource.com/1110450
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[rename] https://crrev.com/124365b8ea553111c78af62c0d7455423a2238e5/recipes/recipe_modules/bot_update/examples/full.expected/shallow.json
[modify] https://crrev.com/124365b8ea553111c78af62c0d7455423a2238e5/recipes/recipe_modules/bot_update/examples/full.py
[modify] https://crrev.com/124365b8ea553111c78af62c0d7455423a2238e5/recipes/recipe_modules/bot_update/resources/bot_update.py
[modify] https://crrev.com/124365b8ea553111c78af62c0d7455423a2238e5/recipes/README.recipes.md
[modify] https://crrev.com/124365b8ea553111c78af62c0d7455423a2238e5/recipes/recipe_modules/bot_update/api.py

Project Member

Comment 2 by bugdroid1@chromium.org, Jun 22 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/82b417fbe73f20b2fce41ef81a04c1954646dcc6

commit 82b417fbe73f20b2fce41ef81a04c1954646dcc6
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Jun 22 00:10:41 2018

infra: no_shallow is now default for bot_update.

R=jchinlee

Bug:  855137 
Change-Id: I20b208c42fc82151318b02088f74c29ac18bb1e6
Reviewed-on: https://chromium-review.googlesource.com/1111206
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/82b417fbe73f20b2fce41ef81a04c1954646dcc6/recipes/recipe_modules/infra_checkout/api.py

Project Member

Comment 3 by bugdroid1@chromium.org, Jun 22 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1321b38de5a9837f5260ec9a76ef519d2385032f

commit 1321b38de5a9837f5260ec9a76ef519d2385032f
Author: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri Jun 22 01:56:35 2018

Roll src/third_party/depot_tools 4099daa97b38..09098853e107 (8 commits)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/4099daa97b38..09098853e107


git log 4099daa97b38..09098853e107 --date=short --no-merges --format='%ad %ae %s'
2018-06-21 vadimsh@chromium.org Demote linux-386 to "best effort support", just like e.g. linux-ppc64.
2018-06-21 recipe-roller@chromium.org Roll recipe dependencies (trivial).
2018-06-21 ehmaldonado@chromium.org gclient: Make gclient respect unmanaged dependencies when syncing.
2018-06-21 recipe-roller@chromium.org Roll recipe dependencies (trivial).
2018-06-21 vadimsh@chromium.org Stop checking CIPD packages exist on linux-386.
2018-06-21 ahassani@google.com cpplint: Pull in upstream changes
2018-06-21 tandrii@chromium.org bot_update: default to non-shallow checkouts.
2018-06-21 iannucci@chromium.org Fix minor regression in git_upstream_diff.


Created with:
  gclient setdep -r src/third_party/depot_tools@09098853e107

The AutoRoll server is located here: https://depot-tools-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG=chromium:854300,chromium:853032,chromium:854300,chromium:852898, chromium:855137 
TBR=agable@chromium.org

Change-Id: Id8a265a396d0d7fdbdad58688673e2acb30a3446
Reviewed-on: https://chromium-review.googlesource.com/1111137
Reviewed-by: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#569505}
[modify] https://crrev.com/1321b38de5a9837f5260ec9a76ef519d2385032f/DEPS

Project Member

Comment 4 by bugdroid1@chromium.org, Jun 22 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/01e8f2db34b1ce17a2df22e3895ce25d164e65b3

commit 01e8f2db34b1ce17a2df22e3895ce25d164e65b3
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Jun 22 18:05:19 2018

Project Member

Comment 5 by bugdroid1@chromium.org, Jun 22 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/3cb25b1ad3a15a76f8a5f8dcc1d0e23b65705ac3

commit 3cb25b1ad3a15a76f8a5f8dcc1d0e23b65705ac3
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Jun 22 18:07:09 2018

v8: no_shallow is now default for bot_update.

TBR=machenbach, sergiyb

Bug:  855137 
Change-Id: I8f7cdfa532850130fcb450de03000e60da5829a1
Reviewed-on: https://chromium-review.googlesource.com/1111116
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/3cb25b1ad3a15a76f8a5f8dcc1d0e23b65705ac3/scripts/slave/recipe_modules/v8/api.py

Project Member

Comment 7 by bugdroid1@chromium.org, Jun 22 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/565fa6a4078091814163e8449609619953545947

commit 565fa6a4078091814163e8449609619953545947
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Jun 22 19:50:46 2018

Revert "Wipe out shallow checkout support from bot_update."

This reverts commit 568b7a8998b127cb5e34149cd63e9b16fe823a40.

Reason for revert: temporary revert to revert previous CL.

Original change's description:
> Wipe out shallow checkout support from bot_update.
> 
> Bug:  855137 
> Change-Id: I60b8d69f9ab353c12120d4216e3a12518f99732c
> Reviewed-on: https://chromium-review.googlesource.com/1111107
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Ryan Tseng <hinoka@chromium.org>

TBR=iannucci@chromium.org,hinoka@chromium.org,tandrii@chromium.org

Change-Id: Iede67717be448be9bdd7c5dd4895fbab9fd65c66
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  855137 ,  855700 
Reviewed-on: https://chromium-review.googlesource.com/1111924
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[add] https://crrev.com/565fa6a4078091814163e8449609619953545947/recipes/recipe_modules/bot_update/examples/full.expected/shallow.json
[modify] https://crrev.com/565fa6a4078091814163e8449609619953545947/tests/bot_update_coverage_test.py
[modify] https://crrev.com/565fa6a4078091814163e8449609619953545947/recipes/recipe_modules/bot_update/examples/full.py
[modify] https://crrev.com/565fa6a4078091814163e8449609619953545947/recipes/README.recipes.md
[modify] https://crrev.com/565fa6a4078091814163e8449609619953545947/recipes/recipe_modules/bot_update/api.py
[modify] https://crrev.com/565fa6a4078091814163e8449609619953545947/recipes/recipe_modules/bot_update/resources/bot_update.py

Project Member

Comment 8 by bugdroid1@chromium.org, Jun 22 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5bee76eb1685337262ac8f43ad401dc7b72d6200

commit 5bee76eb1685337262ac8f43ad401dc7b72d6200
Author: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri Jun 22 22:59:09 2018

Roll src/third_party/depot_tools c48318a35b17..08d753e88484 (4 commits)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/c48318a35b17..08d753e88484


git log c48318a35b17..08d753e88484 --date=short --no-merges --format='%ad %ae %s'
2018-06-22 recipe-roller@chromium.org Roll recipe dependencies (trivial).
2018-06-22 tandrii@chromium.org Revert "bot_update: Don't force solutions to be unmanaged."
2018-06-22 tandrii@chromium.org Revert "Wipe out shallow checkout support from bot_update."
2018-06-22 tandrii@chromium.org Wipe out shallow checkout support from bot_update.


Created with:
  gclient setdep -r src/third_party/depot_tools@08d753e88484

The AutoRoll server is located here: https://depot-tools-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG=chromium:850812,chromium:853032, chromium:855700 , chromium:855137 , chromium:855700 , chromium:855137 
TBR=agable@chromium.org

Change-Id: I9f3c95031934c4be4cbdb53960a09bd707f61bc6
Reviewed-on: https://chromium-review.googlesource.com/1112227
Reviewed-by: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#569820}
[modify] https://crrev.com/5bee76eb1685337262ac8f43ad401dc7b72d6200/DEPS

Project Member

Comment 9 by bugdroid1@chromium.org, Jun 25 2018

Status: Fixed (was: Assigned)
Project Member

Comment 11 by bugdroid1@chromium.org, Jun 26 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/802c0858f8b701ba5e67e4a8188de346ff9ca711

commit 802c0858f8b701ba5e67e4a8188de346ff9ca711
Author: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Tue Jun 26 02:37:05 2018

Roll src/third_party/depot_tools d91a468aa50d..4c50badb2927 (3 commits)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/d91a468aa50d..4c50badb2927


git log d91a468aa50d..4c50badb2927 --date=short --no-merges --format='%ad %ae %s'
2018-06-26 recipe-roller@chromium.org Roll recipe dependencies (trivial).
2018-06-25 vadimsh@chromium.org [cipd] Update cipd client and vpython 4d19637ec -> 550cea2b9.
2018-06-25 tandrii@chromium.org Reland of "Wipe out shallow checkout support from bot_update."


Created with:
  gclient setdep -r src/third_party/depot_tools@4c50badb2927

The AutoRoll server is located here: https://depot-tools-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG=chromium:801423, chromium:855137 
TBR=agable@chromium.org

Change-Id: I7aed0e8eeb410cdc42f0468278104037dd9edbf3
Reviewed-on: https://chromium-review.googlesource.com/1114049
Reviewed-by: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#570305}
[modify] https://crrev.com/802c0858f8b701ba5e67e4a8188de346ff9ca711/DEPS

Sign in to add a comment