So, git fetch fails due to timeout. I think that's because chromium repo is HUGE and hence it's just very expensive to do fetch. Either way, I'm nuking checkout and let's see what happens.

URL to log: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.infra.cron%2Fgsubtreed%2F95293%2F%2B%2Frecipes%2Fsteps%2Fgsubtreed%2F0%2Fstdout

Dir nuked. New build is running: 
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.infra.cron%2Fgsubtreed%2F95295%2F%2B%2Frecipes%2Fsteps%2Fgsubtreed%2F0%2Fstdout

It's been 25 minutes and bot is still checking out chromium/src. Currently, git process is runnign at 99% CPU and 22% mmeory, strongly suggesting that some kind of gc is happening.

I looked at git2 source -> no gc is called there. And luckily fetch just finished.

Hm, it appears gsubtreed instructed git to fetch all refs, and as a result all Gerrit CL and each patchset's ref was fetched too. I think this is what took Git process so long.

https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.infra.cron%2Fgsubtreed%2F95295%2F%2B%2Frecipes%2Fsteps%2Fgsubtreed%2F0%2Fstdout

My Chrome instance took already 1 minute to count occurrences of refs/changes/ resulting in 100k and still counting.

So, i stopped the build to see if this problem is resolved. Turns out not really. git fetch call simply spends too much time because of the number of refs. As we upload more and more patches to chromium/src, this will only get worse.

If you look at what gsubtreed has in its workdir in chromium/src checkout, you can see git config file:

[remote "origin"]
        url = https://chromium.googlesource.com/chromium/src
        fetch = +refs/*:refs/*

which is what's apparently configured in git2 library repo.reify()

https://cs.chromium.org/chromium/infra/infra/libs/git2/repo.py?l=124


So, the way to resolve this is to somehow exclude refs/changes from the list here.

So, I modified the config above like this:

[remote "origin"]
        url = https://chromium.googlesource.com/chromium/src
        # fetch = +refs/*:refs/*
        fetch = +refs/heads/*:refs/heads/*
        fetch = +refs/branch-heads/*:refs/branch-heads/*
        fetch = +refs/tags/*:refs/tags/*


and restarted the bot. Let's see: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.infra.cron%2Fgsubtreed%2F95297%2F%2B%2Frecipes%2Fsteps%2Fgsubtreed%2F0%2Fstdout

Yay, it helped:

[I2017-07-12T13:22:12.844651+00:00 15459 140435104905024 infra.libs.git2.repo:231] Running ('git', 'fetch')
[I2017-07-12T13:22:18.908702+00:00 15459 140435104905024 infra.libs.git2.repo:261] Finished in 6.1 sec

However, as soon as somebody wipes out the builder dir, we'll get into trouble again.

I think the above hack should be a good enough temporary workaround, and after a few hours gsubtreed should update the outdated mirrors.

However, a permanent fix is necessary. I've sent email to gerrit team asking if they is a way to blacklist fetch refspec, instead of specifying elaborate whitelist.

If not, gsubtreed should probably initialize local clone with the same refspec as given in gsubtreed config:
https://chromium.googlesource.com/chromium/src/+/refs/gsubtreed-config/main/config.json#78

Assigning to agable@ to decide what to do and find an owner.

gsubtreed uses infra.libs.git2.repo.reify() to instantiate the underlying repo. That method runs "git remote add --mirror=fetch origin <url>". It's the "--mirror=fetch" that causes the remote.origin.fetch to be configured as "+refs/*:+refs/*". It then also immediately fetches the repo, without an opportunity to change the fetch spec first.

The only solution I see is to plumb the desired fetch spec all the way from gsubtreed through to git2.repo.reify().

Do we have any updates for this bug? Is it really a Pri-1?

This must have been fixed back in July.