New issue
Advanced search Search tips
Starred by 19 users

Issue metadata

Status: Released
Owner: ----
Closed: Jun 11
Cc:
Components:


Show other hotlists

Hotlists containing this issue:
Hotlist-1


Sign in to add a comment

File handle leak when GC schedule is enabled

Reported by a...@tech-advantage.com, May 4

Issue description

Affected Version: 2.15+

What steps will reproduce the problem?
1. Run 2.15+ in NoteDB and GC schedule enable.

What is the expected output?
Everything should work fine.

What do you see instead?
File handle leak.
Handles are kept open to objects/pack directories of all garbage-collected projects, at every GC run.


Please provide any additional information below.
See https://groups.google.com/d/msg/repo-discuss/y2zp8bgLV-0/Nxl1NEFWBQAJ for more informations.
 
For is, with an uptime of 9 days we have this:

[root@git site]# lsof | grep objects/pack | grep DIR | sort -u | wc -l
41800

It seems JGIT GC keeps handles open for some reason.

It also seems to grow unproportionally quick. On a server with only 12 repositories, doing 'gerrit gc --aggressive All-Users' does this (compared to above):

[root@git site]# lsof | grep objects/pack | grep DIR | sort -u | wc -l
42218

running it once more:

[root@git site]# lsof | grep objects/pack | grep DIR | sort -u | wc -l
42636

Even GC'ing only this single repository will open handles to ALL repositories on Gerrit it seems...
Project Member

Comment 2 by thomasmu...@yahoo.com, May 4

Cc: dborowitz@google.com
Components: NoteDb
Project Member

Comment 3 by thomasmu...@yahoo.com, May 4

Cc: luca.mil...@gmail.com
Labels: -Priority-3 Priority-0
And once more after the weekend (2 full GC cycles on all repositories, GC is configured to run every day):

[root@git site]# lsof | grep objects/pack | grep DIR | sort -u | wc -l
57684

Components: -NoteDb Backend
A small site of mine (~30 users, ~400 projects) is also affected by this issue. Each gc --all adds about 800 file handles (so 2 per repositoriy). Both of these file handles seem to be the pack files.

Since in the mailing list there were also sites not affected by the issue, I wanted to offer info about my setup should you desire it.
Unaffected sites have GC disabled.
Disabling GC just keep an almost flat open-file graph.

The 2-handles-per-repo is what I noticed too (i.e. not related to pushed changes or active users)
we have same issue after upgraded from 2.14.4 to 2.15.2, in our environment, each gerrit gc run consumes about 4000 file descriptors.
In a quick test, it seems that 2.16 (master) is not affected by this issue. Not entirely sure though whether I did the test correctly :D
Status: ChangeUnderReview (was: New)
https://gerrit-review.googlesource.com/#/c/gerrit/+/183810
Labels: FixedIn-2.15.3
Status: Submitted (was: ChangeUnderReview)
Status: Released (was: Submitted)

Sign in to add a comment