New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 663990 link

Starred by 5 users
Status: Fixed
Owner:
no...@chromium.org
Closed: Nov 2016
Cc:
█ phajdan.jr@chromium.org
d...@chromium.org
no...@chromium.org
estaab@chromium.org
chrome-trooper-bugs@chromium.org
Components:
EstimatedDays: ----
NextAction: 2016-11-10
OS: Linux
Pri: 0
Type: Bug



Sign in to add a comment

git cache checkout access denied on Windows

Project Member Reported by dsansome@chromium.org, Nov 10 2016 
Seen here https://luci-milo.appspot.com/swarming/task/3264eac98df7dd10/steps/bot_update/0/stdout:

infra/appengine/third_party/google-api-python-client
----------------------------------------
[0:00:08] Started.
_____ infra\appengine\third_party\google-api-python-client at 49d45a6c3318b75e551c3022020f46c78655f365
[0:00:11] Downloading gs://chromium-git-cache/chromium.googlesource.com-external-github.com-google-google--api--python--client/823.zip
[0:00:15] running "git config gc.autodetach 0" in "e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\_cache_tmp2kzssmchromium.googlesource.com-external-github.com-google-google--api--python--client"
[0:00:15] running "git config gc.autopacklimit 0" in "e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\_cache_tmp2kzssmchromium.googlesource.com-external-github.com-google-google--api--python--client"
[0:00:15] running "git config core.deltaBaseCacheLimit 512m" in "e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\_cache_tmp2kzssmchromium.googlesource.com-external-github.com-google-google--api--python--client"
[0:00:15] running "git config remote.origin.url https://chromium.googlesource.com/external/github.com/google/google-api-python-client.git" in "e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\_cache_tmp2kzssmchromium.googlesource.com-external-github.com-google-google--api--python--client"
[0:00:15] running "git config --replace-all remote.origin.fetch +refs/heads/*:refs/heads/* \+refs/heads/\*:.*" in "e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\_cache_tmp2kzssmchromium.googlesource.com-external-github.com-google-google--api--python--client"
[0:00:15] Fetching +refs/heads/*:refs/heads/*
[0:00:15] running "git fetch -v --progress origin +refs/heads/*:refs/heads/*" in "e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\_cache_tmp2kzssmchromium.googlesource.com-external-github.com-google-google--api--python--client"
[0:00:16] From https://chromium.googlesource.com/a/external/github.com/google/google-api-python-client
[0:00:16]  = [up to date]      gh-pages   -> gh-pages
[0:00:16]  = [up to date]      httplib2-092 -> httplib2-092
[0:00:16]  = [up to date]      jonparrott-patch-1 -> jonparrott-patch-1
[0:00:16]  = [up to date]      master     -> master
[0:00:16]  = [up to date]      release-v1.5.4 -> release-v1.5.4
[0:00:16] Error moving e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\_cache_tmp2kzssmchromium.googlesource.com-external-github.com-google-google--api--python--client to e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\chromium.googlesource.com-external-github.com-google-google--api--python--client: [Error 5] Access is denied

________ running 'git -c core.deltaBaseCacheLimit=512m clone --no-checkout --progress --shared --verbose e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\chromium.googlesource.com-external-github.com-google-google--api--python--client e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\infra\appengine\third_party\_gclient_google-api-python-client_dsc8dk' in 'e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir'
[0:00:16] Cloning into 'e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\infra\appengine\third_party\_gclient_google-api-python-client_dsc8dk'...
[0:00:16] fatal: 'e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\chromium.googlesource.com-external-github.com-google-google--api--python--client' does not appear to be a git repository
[0:00:16] fatal: Could not read from remote repository.
[0:00:16] 
[0:00:16] Please make sure you have the correct access rights
[0:00:16] and the repository exists.

________ running 'git -c core.deltaBaseCacheLimit=512m clone --no-checkout --progress --shared --verbose e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\chromium.googlesource.com-external-github.com-google-google--api--python--client e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\infra\appengine\third_party\_gclient_google-api-python-client_dsc8dk' in 'e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir'
[0:00:17] Cloning into 'e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\infra\appengine\third_party\_gclient_google-api-python-client_dsc8dk'...
[0:00:17] fatal: 'e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\chromium.googlesource.com-external-github.com-google-google--api--python--client' does not appear to be a git repository
[0:00:17] fatal: Could not read from remote repository.
[0:00:17] 
[0:00:17] Please make sure you have the correct access rights
[0:00:17] and the repository exists.

________ running 'git -c core.deltaBaseCacheLimit=512m clone --no-checkout --progress --shared --verbose e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\chromium.googlesource.com-external-github.com-google-google--api--python--client e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\infra\appengine\third_party\_gclient_google-api-python-client_dsc8dk' in 'e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir'
[0:00:18] Cloning into 'e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\infra\appengine\third_party\_gclient_google-api-python-client_dsc8dk'...
[0:00:18] fatal: 'e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\chromium.googlesource.com-external-github.com-google-google--api--python--client' does not appear to be a git repository
[0:00:18] fatal: Could not read from remote repository.
[0:00:18] 
[0:00:18] Please make sure you have the correct access rights
[0:00:18] and the repository exists.

________ running 'git -c core.deltaBaseCacheLimit=512m clone --no-checkout --progress --shared --verbose e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\chromium.googlesource.com-external-github.com-google-google--api--python--client e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\infra\appengine\third_party\_gclient_google-api-python-client_dsc8dk' in 'e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir'
[0:00:20] Cloning into 'e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\infra\appengine\third_party\_gclient_google-api-python-client_dsc8dk'...
[0:00:20] fatal: 'e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir\git_cache\chromium.googlesource.com-external-github.com-google-google--api--python--client' does not appear to be a git repository
[0:00:20] fatal: Could not read from remote repository.
[0:00:20] 
[0:00:20] Please make sure you have the correct access rights
[0:00:20] and the repository exists.
Traceback (most recent call last):
  File "e:\b\depot_tools\gclient_scm.py", line 953, in _Clone
    self._Run(clone_cmd, options, cwd=self._root_dir, retry=True)
  File "e:\b\depot_tools\gclient_scm.py", line 1238, in _Run
    gclient_utils.CheckCallAndFilterAndHeader(cmd, env=env, **kwargs)
  File "e:\b\depot_tools\gclient_utils.py", line 293, in CheckCallAndFilterAndHeader
    return CheckCallAndFilter(args, **kwargs)
  File "e:\b\depot_tools\gclient_utils.py", line 538, in CheckCallAndFilter
    rv, args, kwargs.get('cwd', None), None, None)
CalledProcessError: Command 'git -c core.deltaBaseCacheLimit=512m clone --no-checkout --progress --shared --verbose e:\\b\\swarm_slave\\w\\irh6hnvj\\kitchen-workdir\\git_cache\\chromium.googlesource.com-external-github.com-google-google--api--python--client e:\\b\\swarm_slave\\w\\irh6hnvj\\kitchen-workdir\\infra\\appengine\\third_party\\_gclient_google-api-python-client_dsc8dk' returned non-zero exit status 128 in e:\b\swarm_slave\w\irh6hnvj\kitchen-workdir
----------------------------------------

I logged into that bot (vm304-m4) and the temporary directory is gone, so I can't diagnose.
This is failing on most runs of Infra Win Tester as far as I can tell: https://luci-milo.appspot.com/buildbucket/luci.infra.try/Infra%20Win%20Tester?server=cr-buildbucket.appspot.com, so it's probably preventing lots of CLs from landing.

Comment 1 by estaab@chromium.org, Nov 10 2016

Cc: no...@chromium.org

Comment 2 by no...@chromium.org, Nov 10 2016

NextAction: 2016-11-10
Owner: no...@chromium.org
Status: Assigned (was: Unconfirmed)

Comment 3 by no...@chromium.org, Nov 10 2016 

Issue 663869 has been merged into this issue.

Comment 4 by no...@chromium.org, Nov 10 2016

Status: Started (was: Assigned)

Comment 5 by dsansome@chromium.org, Nov 10 2016

Labels: -Pri-1 Pri-0
Any updates?  This is stopping me from landing any CLs in the infra repo.

Comment 6 by seanmccullough@chromium.org, Nov 10 2016 

*Any* cls? I find if I just keep retrying CQ until Infra Win Tester finally decides to drop its obstructionist agenda for a sec and actually run my change, sometimes it goes through.

Comment 7 by no...@chromium.org, Nov 11 2016 

Issue 663440 has been merged into this issue.

Comment 8 by no...@chromium.org, Nov 11 2016 

Issue 664254 has been merged into this issue.

Comment 9 by no...@chromium.org, Nov 11 2016 

https://codereview.chromium.org/2492963002/ should fix the problem. We are waiting for a recipe roll

Comment 10 by no...@chromium.org, Nov 11 2016 

it took me and dnj@ whole day to find the root cause

Comment 11 by no...@chromium.org, Nov 11 2016 

FTR this is what have been happening:

Our Windows machines on Swarming need python installed. The way we install Python on Windows is through depot_tools, so we had e:/b/depot_tools. Since we don't update python, we never updated that depot tools, so it was very old (SVN). Since we need python in $PATH, e:/b/depot_tools was in $PATH.

The bug was that bot_update.py used gclient.py in $PATH instead of gclient.py in the same repo as the bot_update.py itself. So it implicitly used e:/b/depot_tools/gclient.py which was very old instead of the one we fetch. Also it called old git_cache.py.

What we saw on the surface is bugs in gclient.py and git_cache.py that were apparently fixed a while ago.

We think that after https://codereview.chromium.org/2492963002/ is rolled, we will stop getting Access Denied and "you are using SVN" errors.

Comment 12 by no...@chromium.org, Nov 11 2016 

Recipe Roller finally filed a CL https://codereview.chromium.org/2494913002/

Comment 13 by no...@chromium.org, Nov 11 2016

Cc: d...@chromium.org
Still having access denied errors https://luci-milo.appspot.com/swarming/task/3269b926c286b910/steps/bot_update/0/stdout

Comment 14 by no...@chromium.org, Nov 11 2016 

Line "Copying gs://chromium-git-cache/chromium.googlesource.com-infra-testing-expect_tests/9.zip" appears twice. bot_update.py or gclient.py is still trying to fetch the same thing twice concurrently

Comment 15 by hinoka@chromium.org, Nov 11 2016 

That might be try #1 and try #2

Comment 16 by d...@chromium.org, Nov 11 2016 

  File "e:\b\swarm_slave\w\irxleijh\infra\checkout\recipes\.recipe_deps\depot_tools\git_cache.py", line 444, in populate
    os.rename(tempdir, self.mirror_path)
WindowsError: [Error 5] Access is denied

Looks like that "git_cache" error ignore thing.

Maybe this should be in a retry/sleep loop before it errors.

Comment 17 by no...@chromium.org, Nov 11 2016 

note that there is also 

e:\b\swarm_slave\w\irxleijh\kitchen-workdir\git_cache\_cache_tmpsecpew\863.zip - The process cannot access the file because it is being used by another process.

Comment 18 by d...@chromium.org, Nov 11 2016 

That could be antivirus or scanning that I talked about earlier.

Comment 19 by no...@chromium.org, Nov 11 2016 

https://codereview.chromium.org/2497503002/ adds retries. These bugs was always there, but we didn't see them because git caches on buildbot machines were initialized once long time ago. Currently we are deleting caches each time. Reusing git caches is  issue 661390

Comment 21 by no...@chromium.org, Nov 11 2016

Status: Fixed (was: Started)
looks like the problem is solved https://luci-milo.appspot.com/buildbucket/luci.infra.try/Infra%20Win%20Tester?numbuilds=200
►

Sign in to add a comment