New issue
Advanced search Search tips

Issue 8915 link

Starred by 19 users

Issue metadata

Status: Released
Owner: ----
Closed: Jun 11
Cc:
Components:


Show other hotlists

Hotlists containing this issue:
Hotlist-1


Sign in to add a comment

File handle leak when GC schedule is enabled

Reported by a...@tech-advantage.com, May 4 2018

Issue description

Affected Version: 2.15+

What steps will reproduce the problem?
1. Run 2.15+ in NoteDB and GC schedule enable.

What is the expected output?
Everything should work fine.

What do you see instead?
File handle leak.
Handles are kept open to objects/pack directories of all garbage-collected projects, at every GC run.


Please provide any additional information below.
See https://groups.google.com/d/msg/repo-discuss/y2zp8bgLV-0/Nxl1NEFWBQAJ for more informations.
 
For is, with an uptime of 9 days we have this:

[root@git site]# lsof | grep objects/pack | grep DIR | sort -u | wc -l
41800

It seems JGIT GC keeps handles open for some reason.

It also seems to grow unproportionally quick. On a server with only 12 repositories, doing 'gerrit gc --aggressive All-Users' does this (compared to above):

[root@git site]# lsof | grep objects/pack | grep DIR | sort -u | wc -l
42218

running it once more:

[root@git site]# lsof | grep objects/pack | grep DIR | sort -u | wc -l
42636

Even GC'ing only this single repository will open handles to ALL repositories on Gerrit it seems...
Project Member

Comment 2 by thomasmu...@yahoo.com, May 4 2018

Cc: dborowitz@google.com
Components: NoteDb
Project Member

Comment 3 by thomasmu...@yahoo.com, May 4 2018

Cc: luca.mil...@gmail.com
Labels: -Priority-3 Priority-0
And once more after the weekend (2 full GC cycles on all repositories, GC is configured to run every day):

[root@git site]# lsof | grep objects/pack | grep DIR | sort -u | wc -l
57684

Components: -NoteDb Backend

Comment 6 by schae...@gmail.com, May 10 2018

A small site of mine (~30 users, ~400 projects) is also affected by this issue. Each gc --all adds about 800 file handles (so 2 per repositoriy). Both of these file handles seem to be the pack files.

Since in the mailing list there were also sites not affected by the issue, I wanted to offer info about my setup should you desire it.
Unaffected sites have GC disabled.
Disabling GC just keep an almost flat open-file graph.

The 2-handles-per-repo is what I noticed too (i.e. not related to pushed changes or active users)

Comment 8 by cdlee...@gmail.com, May 29 2018

we have same issue after upgraded from 2.14.4 to 2.15.2, in our environment, each gerrit gc run consumes about 4000 file descriptors.
In a quick test, it seems that 2.16 (master) is not affected by this issue. Not entirely sure though whether I did the test correctly :D
Status: ChangeUnderReview (was: New)
https://gerrit-review.googlesource.com/#/c/gerrit/+/183810
Labels: FixedIn-2.15.3
Status: Submitted (was: ChangeUnderReview)
Status: Released (was: Submitted)

Sign in to add a comment