New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 801702 link

Starred by 7 users

Issue metadata

Status: Fixed
Owner:
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux , Android , Windows , iOS , Chrome , Mac
Pri: 1
Type: Bug

Blocking:
issue 802310



Sign in to add a comment

Chrome main process memory leak [2GB] in glib.

Reported by geo...@mux.ro, Jan 12 2018

Issue description

Chrome Version       : 65.0.3315.3
OS Version: XUbuntu 17.10
URLs (if applicable) :
Other browsers tested: None

What steps will reproduce the problem?
1. Open chrome
2. Use chrome

What is the expected result?
Chrome not to use a larger and larger amount of memory.

What happens instead of that?
The Chrome _main_ process starts to use more and more memory, reaching over 10GiB in quite a short time span. It also seems to evade the oomkiller, so this causes my entire system to freeze.

Please provide any additional information below. Attach a screenshot if
possible.

I've ran chrome with "--disable-gpu --memlog=minimal", however, I've attached about:gpu.
I've also attached a memory profile and screenshots of Chrome's Task Manager after use, and after having closed all other tasks.

Here's a timeline of the increased memory usage: https://gist.github.com/noonien/c14f76ce0d4a4a6fea7e62521a6831f1
In most cases when the memory usage increased dramatically, I believe the browser was sitting idle.

UserAgentString: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3315.3 Safari/537.36

This has been happening over a month, and for quite a few versions, on stable and unstable chrome versions, and on different versions of Ubuntu.

I don't know how to continue even debugging this, apart from restarting chrome at regular intervals, there doesn't seem to be any workaround.
 
about:gpu.txt
4.7 KB View Download
task_manager_before_closing_tasks.png
131 KB View Download
task_manager_after_closing_tabs_and_extensions.png
31.3 KB View Download
task_manager_after_closing_all_other_tasks.png
25.9 KB View Download
trace_with_heap_dump.json.gz
151 KB Download
Cc: erikc...@chromium.org ajwong@chromium.org
Labels: -Pri-3 Performance-Memory Pri-2
Owner: erikc...@chromium.org
Status: Assigned (was: Unconfirmed)
Thanks, investigating.
Cc: primiano@chromium.org
Attached symbolized trace. I looked up the 4 glib addresses, and symbolized using 

https://launchpad.net/ubuntu/artful/amd64/libglib2.0-0-dbgsym/2.54.1-1ubuntu1

The four symbols correspond to:erikchen@erikchen:/tmp/out3/usr/lib/debug/.build-id$ ./symbolize.sh 
./debian/build/deb/glib/../../../../glib/gslice.c:1411
./debian/build/deb/glib/../../../../glib/gmem.c:96
./debian/build/deb/glib/../../../../glib/gmem.c:161
./debian/build/deb/glib/../../../../glib/gmem.c:126

which correspond to allocator_memalign, g_malloc, g_realloc, and g_malloc0

So - something calls glib and leaks a lot. primiano guesses dbus or gconf.
trace_with_heap_dump.json.gz
175 KB Download
Cc: thomasanderson@chromium.org
Labels: -Pri-2 M-65 Pri-1
Owner: thomasanderson@chromium.org
Over to thomas anderson, our resident Linux expert.

There's not much more I can do here without a full stack trace. ajwong@ and primiano@ are currently discussing some potential changes to get there on slack.

thomasanderson: This issue seems to have really cropped up around M63. Any change in that time period that might cause this type of issue?
Cc: hashimoto@chromium.org hidehiko@chromium.org
Summary: Chrome main process memory leak [2GB] in glib. (was: Chrome main process memory leak)
Looks like hidehiko has been making non-trivial changes to dbus.

Comment 6 by geo...@mux.ro, Jan 16 2018

Is there any more data I can gather to provide you with?
One temporary workaround would be starting chrome with
ltrace -llibglib* /opt/google/chrome and see if there is something that stands out

Comment 8 by geo...@mux.ro, Jan 16 2018

When starting with ltrace, the process seems to die before any GUI shows up. I've attached the output
ltrace.log
6.0 MB View Download
Blocking: 802310
If you have time, please take the following steps. This is a bit involved, but should get us sufficient info to debug the problem.

1) Download and extract this custom build of chromium. It's a release build, with unstripped CFI, and our heap dump code has been changed to use backtrace() instead of framepointers.
https://drive.google.com/file/d/1rxfTSYYYZQy7PNtzTt1k-on0S6k7g-s_/view?usp=sharing

2) Take your existing instance of chrome and navigate to chrome://version. Record the profile path. Mine is 
"""
/usr/local/google/home/erikchen/.config/chrome-remote-desktop/chrome-profile/Default
"""

3)_Close Chrome.

4) Make a copy of the profile [path minus the Default or Profile 1 suffix] e.g.
"""
cp -r /usr/local/google/home/erikchen/.config/chrome-remote-desktop/chrome-profile /tmp/test_profile
"""
We'll work with the test profile to prevent anything weird from happening to the real profile. 

5) Launch unstripped-chrome. I extracted to the desktop, so for me it's
"""
~/Desktop/unstripped-chrome/chrome --user-data-dir=/tmp/test_profile
"""

6) Confirm that chromium has launched. The icon should be shades of blue instead of red/yellow/green.

7) Navigate to about://flags and turn on "memlog" to "profile only the browser process".

8) Do the thing that gets chrome to bloat in memory. This might be as simple as letting it sit for a few hours.

9) Navigate to chrome://memory-internals to get a heap dump. Send it to me.

Fingers crossed. :)

Comment 11 by geo...@mux.ro, Jan 17 2018

I've attached the heap dump using the binary from that link, I've also attached a log of the cpu/memory usage over time.

The main process in the binary used a LOT of CPU and performed really badly, is this normal for an unstriped binary when profiling memory?

Cheers!
trace_with_heap_dump.json.gz
176 KB Download
unstripped-chromium-usage.log
4.0 MB View Download
Owner: erikc...@chromium.org
whoo, awesome. Attaching symbolized trace.

performance - yeah, that's a combination of the change to use backtrace(), as well as the heap profiling itself.

There's ~700MB being leaked via gio on a new pthread, along with some PasswordStore code. Will dig.
trace.json.gz
188 KB Download
Screen Shot 2018-01-17 at 11.43.08 AM.png
344 KB View Download
george@ you are our hero! Thanks so much, your help here is super useful (especially when erikchen@ does all the rest of the work :P).
danakj points out a potentially related crbug: https://bugs.chromium.org/p/chromium/issues/detail?id=393395

still digging.

Comment 16 by geo...@mux.ro, Jan 17 2018

My pleasure. I've been experiencing this bug for a few months. It sometimes interferes with work. I've even switched to firefox for a few days because of it, but the experience didn't even come close.

I asked around on IRC a few times with no reply, then was discouraged by all the non-useful "my chrome is using too much memory" bugs that have no activity for months, and finally decided to do all I can to help squash this bug!
trace with glib and gio symbolized. nothing too exciting that's new.
trace4.json.gz
188 KB Download
Screenshot of the 800MB chunk of leaked memory. Just glib/gio doing some parsing on another thread. Root cause is probably still the same [guessing password still].
Screen Shot 2018-01-17 at 1.27.31 PM.png
258 KB View Download
There's probably more than 1 problem.

More details on obvious chrome leak: https://bugs.chromium.org/p/chromium/issues/detail?id=393395

There's also the question: Why is the password manager being called millions (?) of times. This is probably not a particularly cheap operation.
Cc: cfroussios@chromium.org
+cfroussios

The heap dump suggests that we're querying the password manager 100,000s or millions of times. Any idea why that might be happening?

Comment 21 by ingen...@gmail.com, Jan 17 2018

Same issue here, the main Browser process leaks 15GB per day for me. Let me know if there's anything I can do to help.
I put up a CL to fix the obvious leak. The next thing I want to figure out is *why* this is being called so often. Will post back tomorrow with some follow up steps.
Cc: vasi...@chromium.org
I am not noticing unreasonable calls to the password manager when I run locally.
George, when you used Chrome, did you visit a website and which one?

+vasilii, who understands better what fetches the password list.

Comment 24 by geo...@mux.ro, Jan 18 2018

I've logged in on several websites, I've also got LastPass running, in case that matters for anything.

I've also managed to reproduce the leak without vising any sites with any kind of visible password fields, extensions disabled and just browsing.

This happens when I'm idle as well, so I don't think user input is relevant.

I can reproduce the bug, if there's any way of me providing you with more debug info from a running process, let me know
I'm working on putting together a custom build of Chrome that will give us more debugging information - hopefully to the point where we can figure out what's causing all these password manager invocations. :)
We retrieve passwords when we see a form on a site. Thus, thousands of PasswordStore::GetLoginsImpl calls is extraordinary.

Can you open chrome://password-manager-internals/ next to your other tabs and browse normally? It will record the interesting events while the tab is open. Then you can post the log here.

Comment 27 by geo...@mux.ro, Jan 18 2018

I've attached a paste of the chrome://password-manager-internals/ page
password-manager-internals.log
115 KB View Download
Oh, awesome. Even as someone who knows nothing about what this log means, there are several things that stand out:

"Number of pending login managers" increases over time, ending at 57 at the end of the log.

Problem is probably with "https://www.irccloud.com/".

There are quite a few "Message: Generation invalid PasswordForm " and "Message: Generation: no non-blacklisted confirmation " messages.
I made an account on irccloud.com and logged in to the freenode #chromium channel. Every minute, there are 13 calls to "NativeBackendLibsecret::GetLoginsList".

At a guess, any single page site which dynamically creates forms will cause this issue.
I posted a test page. It causes an infinite stream of calls to GetLoginsList.

Notably - if I don't remove the previous password forms, every single password form is queried every time any password form is added, which causes N^2 calls to GetLoginsList. :O
infinite_forms.html
680 bytes View Download
Project Member

Comment 31 by bugdroid1@chromium.org, Jan 18 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e3e1bc737140de3f68b2a40dfa1f51ad3815460c

commit e3e1bc737140de3f68b2a40dfa1f51ad3815460c
Author: erikchen <erikchen@chromium.org>
Date: Thu Jan 18 16:31:29 2018

Fix memory leak in Linux libsecret wrapper.

secret_service_search_sync returns a glist, whose elements must also be freed.
The existing code did not do this. This CL creates a new SearchHelper to wrap
this call. The results are stored in the SearchHelper, whose destructor will
correctly free the elements.

This CL updates the key_storage_libsecret and native_backend_libsecret unittests
to return real GObjects from secret_service_search. The tear down steps now
check that the GObjects are correctly unreffed by the caller.

Bug:  801702 ,  393395 
Change-Id: I0dcac794d4d79499742aea6e7e42a99135367a51
Reviewed-on: https://chromium-review.googlesource.com/871930
Commit-Queue: Erik Chen <erikchen@chromium.org>
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Reviewed-by: Vasilii Sukhanov <vasilii@chromium.org>
Reviewed-by: Christos Froussios <cfroussios@chromium.org>
Cr-Commit-Position: refs/heads/master@{#530162}
[modify] https://crrev.com/e3e1bc737140de3f68b2a40dfa1f51ad3815460c/chrome/browser/password_manager/native_backend_libsecret.cc
[modify] https://crrev.com/e3e1bc737140de3f68b2a40dfa1f51ad3815460c/chrome/browser/password_manager/native_backend_libsecret_unittest.cc
[modify] https://crrev.com/e3e1bc737140de3f68b2a40dfa1f51ad3815460c/components/os_crypt/key_storage_libsecret.cc
[modify] https://crrev.com/e3e1bc737140de3f68b2a40dfa1f51ad3815460c/components/os_crypt/key_storage_libsecret_unittest.cc
[modify] https://crrev.com/e3e1bc737140de3f68b2a40dfa1f51ad3815460c/components/os_crypt/libsecret_util_linux.cc
[modify] https://crrev.com/e3e1bc737140de3f68b2a40dfa1f51ad3815460c/components/os_crypt/libsecret_util_linux.h

Comment 32 by geo...@mux.ro, Jan 18 2018

Can confirm that the test page causes memory usage to increase fast.
I agree that from the log www.irccloud.com generates more calls than it should. So there is a bug in the password manager here (and in the page too as it seems to generate too many forms).

george@, can you confirm that without www.irccloud.com opened your memory consumption is doing better?

Comment 34 by geo...@mux.ro, Jan 18 2018

vasilii@, the memory consumption appears to be better with www.irccloud.com closed.

The test page submitted by erikchen@ also appears to increase the memory usage at a fast pace while opene (I've modified the timer to spawn forms faster).
Owner: vasi...@chromium.org
Okay, even with the fix, there are still some slow leaks in the browser process.

repro steps: 
1) Run infinite_forms.html [change the settimeout duration to 10ms]
2) Watch browser memory climb over time.

I've attached a JSON file which contains stack traces of all leaked allocations. Load the contents in http://jsonviewer.stack.hu/ to see it easily.

Almost all leaks come from CreatePendingLoginManagers. At a guess, we never clear pending_login_managers_ unless the page goes through a navigation.

This sounds very similar to https://bugs.chromium.org/p/chromium/issues/detail?id=769617.
I'm not seeing any other leaks caused by libsecret.

So, handing bug over to vasilli. Two issues:

1) N calls to GetLoginsList every time a single password form is added.
2) Objects created in CreatePendingLoginManagers are leaked on test page.


process_100483_malloc-leaks.json
22.3 KB View Download
vasilii: Ping.
I'll look into it soon.
The regression happened in r496263. The form without name doesn't match itself. Therefore, we create enormous amount of PasswordFormManager objects and every single one creates a request to the password store.
Nice find! Thanks for following up. :)
Project Member

Comment 40 by bugdroid1@chromium.org, Mar 6 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fb3209a2a966a499243dd844490879f4fb4a6ff8

commit fb3209a2a966a499243dd844490879f4fb4a6ff8
Author: Vasilii Sukhanov <vasilii@chromium.org>
Date: Tue Mar 06 18:05:28 2018

Improve password form matching.

Current code is written in a way that a form without action or name doesn't match itself. It's a performance problem because PasswordManager::CreatePendingLoginManagers continuously creates a PasswordFormManager for the same form leading to poor performance.
The code was firstly introduced in https://chromium-review.googlesource.com/c/chromium/src/+/620768. However, motivation for it is unclear after discussing together.

Bug:  801702 , 597309 
Change-Id: Ie2458d8d5340852530675b9f5d7f627132e1f3c6
Reviewed-on: https://chromium-review.googlesource.com/946490
Commit-Queue: Vasilii Sukhanov <vasilii@chromium.org>
Reviewed-by: Vadym Doroshenko <dvadym@chromium.org>
Cr-Commit-Position: refs/heads/master@{#541145}
[modify] https://crrev.com/fb3209a2a966a499243dd844490879f4fb4a6ff8/chrome/browser/password_manager/password_manager_browsertest.cc
[modify] https://crrev.com/fb3209a2a966a499243dd844490879f4fb4a6ff8/chrome/browser/password_manager/password_manager_test_base.cc
[modify] https://crrev.com/fb3209a2a966a499243dd844490879f4fb4a6ff8/chrome/browser/password_manager/password_manager_test_base.h
[add] https://crrev.com/fb3209a2a966a499243dd844490879f4fb4a6ff8/chrome/test/data/password/recurring_dynamic_form.html
[modify] https://crrev.com/fb3209a2a966a499243dd844490879f4fb4a6ff8/components/password_manager/core/browser/password_form_manager.cc
[modify] https://crrev.com/fb3209a2a966a499243dd844490879f4fb4a6ff8/components/password_manager/core/browser/password_form_manager_unittest.cc
[modify] https://crrev.com/fb3209a2a966a499243dd844490879f4fb4a6ff8/components/password_manager/core/browser/password_manager_unittest.cc

Components: UI>Browser>Passwords
Labels: OS-Android OS-Chrome OS-iOS OS-Mac OS-Windows
Status: Fixed (was: Assigned)

Sign in to add a comment