New issue
Advanced search Search tips
Starred by 19 users

Issue metadata

Status: Released
Owner: ----
Closed: Jun 2018
Cc:
Components:


Show other hotlists

Hotlists containing this issue:
Hotlist-1


Sign in to add a comment
link

Issue 8915: File handle leak when GC schedule is enabled

Reported by a...@tech-advantage.com, May 4 2018

Issue description

Affected Version: 2.15+

What steps will reproduce the problem?
1. Run 2.15+ in NoteDB and GC schedule enable.

What is the expected output?
Everything should work fine.

What do you see instead?
File handle leak.
Handles are kept open to objects/pack directories of all garbage-collected projects, at every GC run.


Please provide any additional information below.
See https://groups.google.com/d/msg/repo-discuss/y2zp8bgLV-0/Nxl1NEFWBQAJ for more informations.
 

Comment 1 by duft.mar...@gmail.com, May 4 2018

For is, with an uptime of 9 days we have this:

[root@git site]# lsof | grep objects/pack | grep DIR | sort -u | wc -l
41800

It seems JGIT GC keeps handles open for some reason.

It also seems to grow unproportionally quick. On a server with only 12 repositories, doing 'gerrit gc --aggressive All-Users' does this (compared to above):

[root@git site]# lsof | grep objects/pack | grep DIR | sort -u | wc -l
42218

running it once more:

[root@git site]# lsof | grep objects/pack | grep DIR | sort -u | wc -l
42636

Even GC'ing only this single repository will open handles to ALL repositories on Gerrit it seems...

Comment 2 by thomasmu...@yahoo.com, May 4 2018

Project Member
Cc: dborowitz@google.com
Components: NoteDb

Comment 3 by thomasmu...@yahoo.com, May 4 2018

Project Member
Cc: luca.mil...@gmail.com
Labels: -Priority-3 Priority-0

Comment 4 by duft.mar...@gmail.com, May 7 2018

And once more after the weekend (2 full GC cycles on all repositories, GC is configured to run every day):

[root@git site]# lsof | grep objects/pack | grep DIR | sort -u | wc -l
57684

Comment 5 by david.pu...@gmail.com, May 10 2018

Components: -NoteDb Backend

Comment 6 by schae...@gmail.com, May 10 2018

A small site of mine (~30 users, ~400 projects) is also affected by this issue. Each gc --all adds about 800 file handles (so 2 per repositoriy). Both of these file handles seem to be the pack files.

Since in the mailing list there were also sites not affected by the issue, I wanted to offer info about my setup should you desire it.

Comment 7 by a...@tech-advantage.com, May 10 2018

Unaffected sites have GC disabled.
Disabling GC just keep an almost flat open-file graph.

The 2-handles-per-repo is what I noticed too (i.e. not related to pushed changes or active users)

Comment 8 by cdlee...@gmail.com, May 29 2018

we have same issue after upgraded from 2.14.4 to 2.15.2, in our environment, each gerrit gc run consumes about 4000 file descriptors.

Comment 9 by duft.mar...@gmail.com, May 29 2018

In a quick test, it seems that 2.16 (master) is not affected by this issue. Not entirely sure though whether I did the test correctly :D

Comment 10 by david.pu...@gmail.com, Jun 9 2018

Status: ChangeUnderReview (was: New)
https://gerrit-review.googlesource.com/#/c/gerrit/+/183810

Comment 11 by david.pu...@gmail.com, Jun 11 2018

Labels: FixedIn-2.15.3
Status: Submitted (was: ChangeUnderReview)

Comment 12 by david.pu...@gmail.com, Jul 18 2018

Status: Released (was: Submitted)

Sign in to add a comment