New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 825370 link

Starred by 4 users

Issue metadata

Status: Fixed
Owner:
Closed: Apr 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Remove ChromeOS git retry logic.

Project Member Reported by dgarr...@chromium.org, Mar 23 2018

Issue description

ChromeOS has a lot of embedded error retry logic around git operations.

Our builders are now using a git wrapper which implements that invisibly for us, which are at least partially derived from our logic.

To avoid inner and outer retry loops for the same things (which can sometimes be valid errors), we should remove the appropriate retry logic in cbuildbot.

However, since the git wrapper is not used on developer workstations, this means they will lose the relevant retry logic.
 
Cc: akes...@chromium.org
We have been seeing a lot of git-ls quota usage, which is conceivably related to excessive retries.

Comment 2 by vapier@chromium.org, Mar 23 2018

Labels: OS-Chrome
where does the git wrapper live ?

i'd be for dropping retries even for devs and wait for reports.  sometimes it's good to re-evaluate the state of the world.
chromite/lib/git.py. 

Look for GIT_TRANSIENT_ERRORS and related logic.

Comment 4 by vapier@chromium.org, Mar 23 2018

i mean where's the wrapper the builders are providing, not the wrappers in chromite that we're using :)
iannucci@ can answer in more detail, but it's a binary named "git" in our path when the recipe is invoked.
Source code: https://chromium.googlesource.com/infra/infra/+/master/go/src/infra/tools/git/
It's deployed as CIPD package 'infra/tools/git/${platform}': https://chromium.googlesource.com/infra/infra/+/master/build/packages/git.yaml

You can poke it locally:

mkdir git_wrapper
echo 'infra/tools/git/${platform} latest' | cipd ensure -ensure-file - -root git_wrapper
export PATH=`pwd`/git_wrapper:$PATH

which git
git version

(Should tell something like: "git version 2.13.1.chromium10 / Infra wrapper (infra/tools/git/mac-amd64 @ 64ccae1cbeb88f728bd6f50d65eb7b19f5fcd7a3)").

Comment 7 by vapier@chromium.org, Mar 23 2018

i'm wondering if it's something we can get in depot_tools if needed.  i guess on the Chromium side, there isn't a strong need/desire for retries for devs ?  they just live with/manually retry when doing things like `git cl` ?
I think GoB is mostly robust enough to not be a problem if you aren't doing 1000's of builds a day.

Comment 9 by vapier@chromium.org, Mar 23 2018

i'm cool with dropping the transparent retries for devs and wait for data to show up to see what we want/need
Labels: -Pri-3 Pri-2
Owner: jclinton@chromium.org
Status: Assigned (was: Untriaged)
One thought that occurred to me.... do we have the git wrapper inside the chroot? I have no idea.

We don't perform the major sync operations inside the chroot, but I'm not sure what other cases might exist, such as ebuilds that fetch code directly.

Cc: ehmaldonado@chromium.org aga...@chromium.org
We do actually want to put the git wrapper (and probably git of some known up-to-date version too) into depot_tools eventually, but it's relatively low priority at the moment. +agable/+ehmaldonado who are currently the depot_tools wranglers.
Components: -Infra>Client>ChromeOS Infra>Client>ChromeOS>CI
Owner: mikenichols@chromium.org
Project Member

Comment 16 by bugdroid1@chromium.org, Apr 11 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/a6818c5b704dc10718f963778cf4b85fa017908c

commit a6818c5b704dc10718f963778cf4b85fa017908c
Author: Mike Nichols <mikenichols@chromium.org>
Date: Wed Apr 11 02:12:59 2018

Removing git retry logic

Removing all git retry logic from git module.  Retry logic will
now be handled by the git wrapper within the builders.

BUG= chromium:825370 
TEST=`./cbuildbot/run_tests` PASS

Change-Id: I592267bf6013acf56ca87d2719bd73a189496153
Reviewed-on: https://chromium-review.googlesource.com/1003273
Commit-Ready: Mike Nichols <mikenichols@chromium.org>
Tested-by: Mike Nichols <mikenichols@chromium.org>
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/a6818c5b704dc10718f963778cf4b85fa017908c/scripts/cros_mark_as_stable_unittest.py
[modify] https://crrev.com/a6818c5b704dc10718f963778cf4b85fa017908c/lib/git_unittest.py
[modify] https://crrev.com/a6818c5b704dc10718f963778cf4b85fa017908c/cbuildbot/stages/config_stages_unittest.py
[modify] https://crrev.com/a6818c5b704dc10718f963778cf4b85fa017908c/scripts/cros_mark_as_stable.py
[modify] https://crrev.com/a6818c5b704dc10718f963778cf4b85fa017908c/scripts/upload_prebuilts.py
[modify] https://crrev.com/a6818c5b704dc10718f963778cf4b85fa017908c/cbuildbot/stages/config_stages.py
[modify] https://crrev.com/a6818c5b704dc10718f963778cf4b85fa017908c/lib/patch.py
[modify] https://crrev.com/a6818c5b704dc10718f963778cf4b85fa017908c/lib/constants.py
[modify] https://crrev.com/a6818c5b704dc10718f963778cf4b85fa017908c/lib/git.py

Project Member

Comment 17 by bugdroid1@chromium.org, Apr 11 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1b2b5547d17eeb4f9013b00a1ea611854223dc1e

commit 1b2b5547d17eeb4f9013b00a1ea611854223dc1e
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Wed Apr 11 08:10:58 2018

Roll src/third_party/chromite/ 871e7778e..a6818c5b7 (2 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/871e7778ef8b..a6818c5b704d

$ git log 871e7778e..a6818c5b7 --date=short --no-merges --format='%ad %ae %s'
2018-04-09 mikenichols Removing git retry logic
2018-04-09 mortonm Adjust tester for debugfs-access group.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:825370 ,chromium:649417


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I29f963f82c2e6af780e4eab40e528b3f960c75b1
Reviewed-on: https://chromium-review.googlesource.com/1006477
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#549820}
[modify] https://crrev.com/1b2b5547d17eeb4f9013b00a1ea611854223dc1e/DEPS

Status: Fixed (was: Assigned)

Sign in to add a comment