Monorail Project: gerrit Issues People Development process History Sign in
New issue
Advanced search Search tips
Starred by 1 user
Status: Released
Owner: ----
Closed: Nov 2016



Sign in to add a comment
Millions of Postgresql statements occupying GBs of memory
Reported by osw...@gmail.com, Nov 1 2016 Back to list
*****************************************************************
*****                                                       *****
***** !!!! THIS BUG TRACKER IS FOR GERRIT CODE REVIEW !!!!  *****
*****                                                       *****
***** DO NOT SUBMIT BUGS FOR CHROME, ANDROID, CYANOGENMOD,  *****
***** INTERNAL ISSUES WITH YOUR COMPANY'S GERRIT SETUP, ETC.*****
*****                                                       *****
*****   THOSE ISSUES BELONG IN DIFFERENT ISSUE TRACKERS     *****
*****                                                       *****
*****************************************************************

Affected Version: 2.12+

While tracking high memory usage in our Gerrit instance, a head dump showed there were millions of Postgresql objects occupying several GB of space in memory (18 millions of objects adding to 22GB in a 67GB head dump). Investigating a bit, we found these objects were pending finalization for a long time but they weren't being collected at all.

This situation had already being reported in the past [1] and seems to be related to the use of finalize method in older versions of the JDBC driver.

Only way to get rid of these objects is restarting Gerrit instance. Subsequent head dumps showed the same pattern.


What steps will reproduce the problem?
1. Have a high traffic, very busy Gerrit server? 
2. With a big JVM heap?
3. And Gerrit using an older JDBC driver that relies in finalize methods?


Please provide any additional information below.

After migrating to 2.12, started to see increased memory usage partially explained by increased traffic. In order to cope with this, the JVM heap size was increased and the GC algorithm changed to G1. Old generation was still growing steadily which prompted further investigation. Memory dumps showed described anomaly, along with two other ones related to JGit and Lucene (not described in this bug report to avoid confusion).

[1] http://postgresql.nabble.com/Remove-usage-of-finalizers-td5775186.html
 
Project Member Comment 1 by huga...@gmail.com, Nov 1 2016
Status: ChangeUnderReview
https://gerrit-review.googlesource.com/#/c/90532/
Comment 2 Deleted
Labels: FixedIn-2.13.3
Status: Submitted
Status: Released
Sign in to add a comment