New issue
Advanced search Search tips
Starred by 60 users

Issue metadata

Status: Accepted
Owner: ----



Sign in to add a comment

"missing tree" when pushing from some git versions

Reported by yuriypad...@gmail.com, Sep 26 2012

Issue description

************************************************************
***** NOTE: THIS BUG TRACKER IS FOR GERRIT CODE REVIEW *****
***** DO NOT SUBMIT BUGS FOR CHROME, ANDROID, INTERNAL *****
***** ISSUES WITH YOUR COMPANY'S GERRIT SETUP, ETC.    *****
***** THOSE ISSUE BELONG IN DIFFERENT ISSUE TRACKERS!  *****
************************************************************

Affected Version: 2.4.2

What steps will reproduce the problem?
1. git push
2.
3.

What is the expected output? What do you see instead?

We expect successfull push, instead we get:
o$ git push
Counting objects: 5389, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (1801/1801), done.
Writing objects: 100% (5190/5190), 2.27 MiB | 15 KiB/s, done.
Total 5190 (delta 3401), reused 5006 (delta 3275)
remote: Resolving deltas: 100% (3401/3401)
error: unpack failed: error Missing tree bfe8d24e0232d8e8969fdadddea7b81aca863664
fatal: Unpack error, check server log
To ssh://**********:29418/********.git
 ! [remote rejected] HEAD -> refs/for/cmedia_3.0.9 (n/a (unpacker error))
error: failed to push some refs to 'ssh://************8:29418/ticketco.git'

Please provide any additional information below.

 

Comment 1 Deleted

Hi guys, we have the same issue with 2.5.1.

Direct push or push to review or draft fails sometimes for some developers with error:

$ git review master
Counting objects: 152, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (66/66), done.
Writing objects: 100% (90/90), 15.94 KiB, done.
Total 90 (delta 33), reused 30 (delta 13)
remote: Resolving deltas: 100% (33/33)
error: unpack failed: error Missing tree 42d769150decc8ee96bd9eb7afacc01fe14ffc55
fatal: Unpack error, check server log
To ssh://....
 ! [remote rejected] HEAD -> refs/for/master (n/a (unpacker error))
error: failed to push some refs to 'ssh://....'

In logs/error_log

Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing tree 42d769150decc8ee96bd9eb7afacc01fe14ffc55
        at org.eclipse.jgit.transport.BaseReceivePack.checkConnectivity(BaseReceivePack.java:996)
        at org.eclipse.jgit.transport.BaseReceivePack.receivePackAndCheckConnectivity(BaseReceivePack.java:756)
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:167)
        ... 15 more

Missing tree sha was different, but every time git ls-tree showed not-corrupted existed tree.
git fsck showed danglings only.

After investigation I found if I set Read right on /ref/* for Registered Users, issue would disappear.

What is the link between Read right on /ref/* for Registered Users on time-to-time missing tree exception? 




Hello guys,
we are exiriencing the same issues with 2.5.1 and 2.5.2.

Aleksey's workaround hasn't helped.

Any suggestions on cause of this issue and possible workarounds?
Hello again,
we started to receive such errors very often.

Push fails with such errors (even push force!! through the Gerrit):
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing tree
or Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob

Any thoughts what could we check?

Those missing trees and blobs are really exist in both local and gerrit repository and i can look in them by: git show

I mentioned that one of missing trees was in Packed objects, but can't say for all of them.

Also I can suggest that such errors come up with commits, that was not uploaded to the Gerrit review previously.

The only way we were able to push changes, is to push them through the pure Git on port 22.




Comment 5 by knibb...@gmail.com, Nov 20 2013

Good day. I've tried to configure local Gerrit for evaluation purposes and got the same problem. Seems that there is some problem in git protocol implementation.
I've checked on git 1.8.3.2 and everything worked fine. When I've updated to 1.8.4.3 problems on commit changes were similar to the one in this topic:

k.stolyarov@DGIS-184 ~/test $ git clone ssh://a.baskanov@dgis-184.dvlp.2gis.local:29418/v2
Cloning into 'v2'...
The authenticity of host '[dgis-184.dvlp.2gis.local]:29418 ([10.54.200.106]:29418)' can't be established.
RSA key fingerprint is c7:4d:d5:23:d1:b3:02:24:c6:21:2c:1d:3e:4e:de:8b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[dgis-184.dvlp.2gis.local]:29418,[10.54.200.106]:29418' (RSA) to the list of known hosts.
Enter passphrase for key '/home/k.stolyarov/.ssh/id_rsa': 
remote: Counting objects: 2, done
remote: Finding sources: 100% (2/2)
remote: Total 2 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (2/2), done.
skjdhjksdf
Checking connectivity... done
k.stolyarov@DGIS-184 ~/test $ cd v2
k.stolyarov@DGIS-184 ~/test/v2 $ vim 1 
k.stolyarov@DGIS-184 ~/test/v2 $ scp -p -P 29418 a.baskanov@dgis-184.dvlp.2gis.local:hooks/commit-msg .git/hooks/
Enter passphrase for key '/home/k.stolyarov/.ssh/id_rsa': 
commit-msg                                                                                                                                                 100% 4367     4.3KB/s   4.3KB/s   00:00    
k.stolyarov@DGIS-184 ~/test/v2 $ git add 1
k.stolyarov@DGIS-184 ~/test/v2 $ git commit
[master dd26813] sdfhsdf
 1 file changed, 1 insertion(+)
 create mode 100644 1
k.stolyarov@DGIS-184 ~/test/v2 $ git push origin HEAD:refs/for/master
Enter passphrase for key '/home/k.stolyarov/.ssh/id_rsa': 
Counting objects: 4, done.
Writing objects: 100% (3/3), 269 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Processing changes: new: 1, refs: 1, done    
remote: 
remote: New Changes:
remote:   http://dgis-184.dvlp.2gis.local:8080/1
remote: 
To ssh://a.baskanov@dgis-184.dvlp.2gis.local:29418/v2
 * [new branch]      HEAD -> refs/for/master
k.stolyarov@DGIS-184 ~/test/v2 $ vim  2
k.stolyarov@DGIS-184 ~/test/v2 $ git add 2
k.stolyarov@DGIS-184 ~/test/v2 $ git commit -a  --amend
[master 6026dbb] sdfhsdf
 2 files changed, 2 insertions(+)
 create mode 100644 1
 create mode 100644 2
k.stolyarov@DGIS-184 ~/test/v2 $ git push origin HEAD:refs/for/master
Enter passphrase for key '/home/k.stolyarov/.ssh/id_rsa': 
Counting objects: 5, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 302 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
error: unpack failed: error Missing blob e89ef0eb5feb4db1e4c757374b83ed1d8e6081ca
fatal: Unpack error, check server log
To ssh://a.baskanov@dgis-184.dvlp.2gis.local:29418/v2
 ! [remote rejected] HEAD -> refs/for/master (n/a (unpacker error))
error: failed to push some refs to 'ssh://a.baskanov@dgis-184.dvlp.2gis.local:29418/v2'

I've checked this on both Gerrit 2.7 and 2.8-rc2 versions from two different linux machines (Gentoo and Ubunto builds). Latest git version bakes review updates.

Hope that this information will help you to fix the issue.
This is much more than a minor issue as it is leading us to have corrupted repos at this time. Can someone please take a deeper look into why this might be happening?
Project Member

Comment 7 by sschube...@gmail.com, Jan 29 2014

 Issue 2296 , which seems to duplicate this issue, suggests to set receive.checkReferencedObjectsAreReachable = false (available since Gerrit 2.6) to work around this.
Why in the world would one turn that off - it sounds like the suggestion is to allow what looks like corruption in the reference tree...?
Project Member

Comment 9 by sschube...@gmail.com, Jan 29 2014

Just consult the Gerrit 2.6 release notes [1] for why it would make sense to turn off that option:

"New config option receive.checkReferencedObjectsAreReachable

If set to true, Gerrit will validate that all referenced objects that are not included in the received pack are reachable by the user.

Carrying out this check on Git repositories with many refs and commits can be a very CPU-heavy operation. For non public Gerrit servers it may make sense to disable this check, which is now possible."

[1] http://gerrit-documentation.googlecode.com/svn-history/r63/ReleaseNotes/ReleaseNotes-2.6.html
Project Member

Comment 10 by sschube...@gmail.com, Jan 29 2014

Also, I don't believe turning off that option allows for corruption in the reference tree. It's just that Gerrit's JGit implementation cannot handle [1] yet (see  issue 2296 ), which is why you get this bogus error with recent Git clients.

[1] https://github.com/git/git/commit/fbd4a7036dfa71ec89e7c441cef1ac9aaa59a315

Comment 11 by onlyn...@gmail.com, Apr 23 2014

Is there any work around for this from the client side? If say I can't upgrade the version of gerrit being used, or change server settings? Maybe there's an option to git I could set to not do the optimization that sends fewer trees?

Comment 12 by mpe...@gmail.com, Jun 10 2014

Per a coworker, sounds like specifying --no-thin on the git client command line avoids this optimization without having to set receive.checkReferencedObjectsAreReachable = false on Gerrit.
Project Member

Comment 14 by jrn@google.com, Sep 5 2014

 Issue 2296  has been merged into this issue.
I'm using gerrit 2.9.1, which already has jgit update (change 59935), but I still can reproduce the missing tree error. It seems to happen only for pushes where *only commit message is changed* without any other commit modifications o.O

$ echo 'a' > a
$ git commit -a -m 'msg'
$ git push origin HEAD:refs/for/master
remote: Resolving deltas: 100% (1/1)
remote: Processing changes: new: 1, refs: 1, done    
remote: New Changes:
remote:   https://gerrit/2202

$ echo 'b' > a
$ git commit -a --amend
/ possibly some changes in msg in editor/
$ git push origin HEAD:refs/for/master
remote: Resolving deltas: 100% (1/1)
remote: Processing changes: updated: 1, refs: 1, done    
remote: Updated Changes:
remote:   https://gerrit.sio2project.mimuw.edu.pl/2202

$ git commit -a --amend
/ changes in msg/
$ git push origin HEAD:refs/for/master 
error: unpack failed: error Missing tree a9596a3d9e2f4fa81912124d3dacd58113f11e85
fatal: Unpack error, check server log
To ssh://gerrit/project
 ! [remote rejected] HEAD -> refs/for/master (n/a (unpacker error))
error: failed to push some refs to 'ssh://gerrit/project'

This is not too common to change only commit msg, also it can be done from gerrit web ui, but still is annoying.

Can someone try to reproduce and confirm it?
Thanks!
Project Member

Comment 17 by nas...@codeaurora.org, Oct 8 2014

How did you determine that 2.9.1 has 59935? 59935 was uploaded and merged to master. The only JGit update I see on stable-2.9 is from May: https://gerrit-review.googlesource.com/#/q/status:merged+project:gerrit+branch:stable-2.9+message:%22JGit%22
Oops, right, I thought I've seen it in some changelog, but apparently it's not there.
Sorry!
Is there gonna be a new release soon? We're plagued by this bug in various teams.
Or is installing 2.10.rc0 a good alternative to waiting for the release?

Comment 20 by jlst...@gmail.com, Nov 14 2014

I've been hitting this issue for quite a while and it still exists in 2.9.1. A simple case to duplicate is to push a change, merge, and try to push the git reverted change. I've been able to reproduce this in more complicated cases where the commit tree ends up being the same as one that was pushed prior.
Project Member

Comment 21 by bassem.r...@gmail.com, Dec 2 2014

If you are using git 1.8.4.2 or later, there is some incompatibility between jgit 2.3 shipped with Gerrit 2.7 and jgit 3.1 shipped with Gerrit 2.8 and this version of git due to this enhancement, i.e. https://github.com/git/git/commit/fbd4a7036dfa71ec89e7c441cef1ac9aaa59a315

With this enhancement, if git finds out that the tree sha1 already exists in the server, it wont send it again, but gerrit wants to search the tree sha1 associated with the commit sha1 in the uploaded pack. There is a bug report on jgit library for this issue at: https://bugs.eclipse.org/bugs/show_bug.cgi?id=426044. This was reported to be fixed for jgit 3.4 which is shipped with Gerrit 2.9, but there is still the same behavior reported with Gerrit 2.9.1.

A proposed workaround for this issue is to go to the local branch you are trying to push, and pull from origin and then retry pushing, or use older version of git to push such branches. 
Project Member

Comment 22 by jrn@google.com, Dec 3 2014

Status: Submitted
Fix submitted and should be part of 2.10-rc1.
Project Member

Comment 23 by jrn@google.com, Dec 3 2014

(The fix I'm referring to is https://gerrit-review.googlesource.com/59935.)

Comment 24 by jlst...@gmail.com, Feb 12 2015

Can anyone verify this is fixed in 2.10?
Got this error in 2.9.3. After update to 2.10 works.
I have this problem in Gerrit 2.11.3. git push --no-thin solves the problem.
@johannes: Wich version of the git client are you using?

On Gerrit 2.10.4 i solved this issue by using Git for Windows 2.x instead of version 1.9.5
@fabio: Sorry for late reply!
I just saw this with gerrit 2.12 from a windows client running git version 1.9.4.msysgit.2
Using --no-thin solved the problem, next time I will try to upgrade git (I am using the version supplied by git extensions which is getting old).
A colleague just got this with git version 2.1.4 from a host running Ubuntu 15.04. --no-thin took care of the problem.
I have the same problem on 2.12.2 gerrit too.

MyDesk> git push origin HEAD:refs/for/Migration
Counting objects: 14, done.
Delta compression using up to 24 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 2.48 KiB | 0 bytes/s, done.
Total 4 (delta 0), reused 0 (delta 0)
error: unpack failed: error Missing tree 2a9930e342ee60aa6fd039dddd6fdae527cc2be4
fatal: Unpack error, check server log
To ssh://$id@gerrit.com:29418/My_Repo
! [remote rejected] HEAD -> refs/for/Migration (n/a (unpacker error))
error: failed to push some refs to 'ssh://$id@gerrit.com:29418/My_Repo'

Using --no-thin also didn't worked as expected.

MyDesk> git push --no-thin origin HEAD:refs/for/Migration
Counting objects: 4, done.
Delta compression using up to 24 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 2.48 KiB | 0 bytes/s, done.
Total 4 (delta 0), reused 0 (delta 0)
error: unpack failed: error Missing tree 2a9930e342ee60aa6fd039dddd6fdae527cc2be4
fatal: Unpack error, check server log
To ssh://$id@gerrit.com:29418/My_Repo
! [remote rejected] HEAD -> refs/for/Migration (n/a (unpacker error))
error: failed to push some refs to 'ssh://$id@gerrit.com:29418/My_Repo'

To reproduce,
Error scenario.( https://gerrit.com/#/c/11349/)
1.	Publish first patch set  - compile error
2.	Fix compile error issue and publish second patch set - “Missing tree” issue

Success scenario.( https://gerrit.com/#/c/11489/)
1.	Publish patch set(in this patch set, compile issue has been fixed.)  - successful

-Thanks

I would like to confirm the bug.
Gerrit version: 2.12
Git version: 1.9.1

Minimal reproduction steps:
01. Create new Gerrit repo
02. Clone Gerrit repo locally
03. cd to local repo
04. Create text file containing two lines
05. Commit file
06. Push
07. Edit file and remove one line
08. Commit file
09. Push
10. Edit file and restore the deleted line (so that file is exactly the same as after step 04)
11. Commit file
12. Push --> Error occurs!

Patches were not yet applied and tested.
Further I would like to suggest to raise the bug priority to major since this bug does prevent to revert changes in git!
Project Member

Comment 33 by bassem.r...@gmail.com, Sep 27 2016

Did you try applying this parameter to false on your server side gerrit.config ?

receive.checkReferencedObjectsAreReachable
If set to true, Gerrit will validate that all referenced objects that are not included in the received pack are reachable by the user.

Carrying out this check on gits with many refs and commits can be a very CPU-heavy operation. For non public Gerrit-servers this check may be overkill.

Only disable this check if you trust the clients not to forge SHA1 references to access commits intended to be hidden from the user.

Default is true.


Project Member

Comment 34 by osw...@gmail.com, Oct 18 2016

@rajesh: Could you please post the git version you were using to push?

@choose: Did you test passing the --no-thin option to git push? Any option of trying with a more up to date git version? (2.x.y, the latest the better)

Similar bug [1] is workaround it by using a newer git version or --no-thin when pushing or receive.checkReferencedObjectsAreReachable = false in gerrit.config

[1] https://bugs.chromium.org/p/gerrit/issues/detail?id=4622

Project Member

Comment 35 by jrn@google.com, Nov 16 2016

Labels: -Priority-Minor Priority-3
Status: Accepted (was: Submitted)
Reopening since the fix to https://bugs.eclipse.org/bugs/show_bug.cgi?id=422988 didn't land.
Project Member

Comment 36 by jrn@google.com, Nov 16 2016

Summary: "missing tree" when pushing from some git versions (was: Issue with git push-ing to Gerrit)
Project Member

Comment 37 by jrn@google.com, Nov 16 2016

 Issue 4622  has been merged into this issue.
Reply to Comment 34:
- Using git push --no-thin does NOT FIX the issue.
- setting receive.checkReferencedObjectsAreReachable to false FIXES the issue.

However it does not seem to be a proper bugfix. So I would like to ask for suggestions solving this issue.

Sign in to add a comment