Thanks, investigating.

Attached symbolized trace. I looked up the 4 glib addresses, and symbolized using 

https://launchpad.net/ubuntu/artful/amd64/libglib2.0-0-dbgsym/2.54.1-1ubuntu1

The four symbols correspond to:erikchen@erikchen:/tmp/out3/usr/lib/debug/.build-id$ ./symbolize.sh 
./debian/build/deb/glib/../../../../glib/gslice.c:1411
./debian/build/deb/glib/../../../../glib/gmem.c:96
./debian/build/deb/glib/../../../../glib/gmem.c:161
./debian/build/deb/glib/../../../../glib/gmem.c:126

which correspond to allocator_memalign, g_malloc, g_realloc, and g_malloc0

So - something calls glib and leaks a lot. primiano guesses dbus or gconf.

Deleted: trace_with_heap_dump.json.gz 175 KB

trace_with_heap_dump.json.gz 175 KB Download

Over to thomas anderson, our resident Linux expert.

There's not much more I can do here without a full stack trace. ajwong@ and primiano@ are currently discussing some potential changes to get there on slack.

thomasanderson: This issue seems to have really cropped up around M63. Any change in that time period that might cause this type of issue?

Looks like hidehiko has been making non-trivial changes to dbus.

Is there any more data I can gather to provide you with?

One temporary workaround would be starting chrome with
ltrace -llibglib* /opt/google/chrome and see if there is something that stands out

When starting with ltrace, the process seems to die before any GUI shows up. I've attached the output

Deleted: ltrace.log 6.0 MB

ltrace.log 6.0 MB View Download

If you have time, please take the following steps. This is a bit involved, but should get us sufficient info to debug the problem.

1) Download and extract this custom build of chromium. It's a release build, with unstripped CFI, and our heap dump code has been changed to use backtrace() instead of framepointers.
https://drive.google.com/file/d/1rxfTSYYYZQy7PNtzTt1k-on0S6k7g-s_/view?usp=sharing

2) Take your existing instance of chrome and navigate to chrome://version. Record the profile path. Mine is 
"""
/usr/local/google/home/erikchen/.config/chrome-remote-desktop/chrome-profile/Default
"""

3)_Close Chrome.

4) Make a copy of the profile [path minus the Default or Profile 1 suffix] e.g.
"""
cp -r /usr/local/google/home/erikchen/.config/chrome-remote-desktop/chrome-profile /tmp/test_profile
"""
We'll work with the test profile to prevent anything weird from happening to the real profile. 

5) Launch unstripped-chrome. I extracted to the desktop, so for me it's
"""
~/Desktop/unstripped-chrome/chrome --user-data-dir=/tmp/test_profile
"""

6) Confirm that chromium has launched. The icon should be shades of blue instead of red/yellow/green.

7) Navigate to about://flags and turn on "memlog" to "profile only the browser process".

8) Do the thing that gets chrome to bloat in memory. This might be as simple as letting it sit for a few hours.

9) Navigate to chrome://memory-internals to get a heap dump. Send it to me.

Fingers crossed. :)

I've attached the heap dump using the binary from that link, I've also attached a log of the cpu/memory usage over time.

The main process in the binary used a LOT of CPU and performed really badly, is this normal for an unstriped binary when profiling memory?

Cheers!

Deleted: trace_with_heap_dump.json.gz 176 KB

trace_with_heap_dump.json.gz 176 KB Download

Deleted: unstripped-chromium-usage.log 4.0 MB

unstripped-chromium-usage.log 4.0 MB View Download

whoo, awesome. Attaching symbolized trace.

performance - yeah, that's a combination of the change to use backtrace(), as well as the heap profiling itself.

There's ~700MB being leaked via gio on a new pthread, along with some PasswordStore code. Will dig.

Deleted: trace.json.gz 188 KB

trace.json.gz 188 KB Download

george@ you are our hero! Thanks so much, your help here is super useful (especially when erikchen@ does all the rest of the work :P).

danakj points out a potentially related crbug: https://bugs.chromium.org/p/chromium/issues/detail?id=393395

still digging.

My pleasure. I've been experiencing this bug for a few months. It sometimes interferes with work. I've even switched to firefox for a few days because of it, but the experience didn't even come close.

I asked around on IRC a few times with no reply, then was discouraged by all the non-useful "my chrome is using too much memory" bugs that have no activity for months, and finally decided to do all I can to help squash this bug!

trace with glib and gio symbolized. nothing too exciting that's new.

Deleted: trace4.json.gz 188 KB

trace4.json.gz 188 KB Download

Screenshot of the 800MB chunk of leaked memory. Just glib/gio doing some parsing on another thread. Root cause is probably still the same [guessing password still].

Deleted: Screen Shot 2018-01-17 at 1.27.31 PM.png 258 KB

	Screen Shot 2018-01-17 at 1.27.31 PM.png 258 KB View Download

There's probably more than 1 problem.

More details on obvious chrome leak: https://bugs.chromium.org/p/chromium/issues/detail?id=393395

There's also the question: Why is the password manager being called millions (?) of times. This is probably not a particularly cheap operation.

+cfroussios

The heap dump suggests that we're querying the password manager 100,000s or millions of times. Any idea why that might be happening?

Same issue here, the main Browser process leaks 15GB per day for me. Let me know if there's anything I can do to help.

I put up a CL to fix the obvious leak. The next thing I want to figure out is *why* this is being called so often. Will post back tomorrow with some follow up steps.

I am not noticing unreasonable calls to the password manager when I run locally.
George, when you used Chrome, did you visit a website and which one?

+vasilii, who understands better what fetches the password list.

I've logged in on several websites, I've also got LastPass running, in case that matters for anything.

I've also managed to reproduce the leak without vising any sites with any kind of visible password fields, extensions disabled and just browsing.

This happens when I'm idle as well, so I don't think user input is relevant.

I can reproduce the bug, if there's any way of me providing you with more debug info from a running process, let me know

I'm working on putting together a custom build of Chrome that will give us more debugging information - hopefully to the point where we can figure out what's causing all these password manager invocations. :)

We retrieve passwords when we see a form on a site. Thus, thousands of PasswordStore::GetLoginsImpl calls is extraordinary.

Can you open chrome://password-manager-internals/ next to your other tabs and browse normally? It will record the interesting events while the tab is open. Then you can post the log here.

I've attached a paste of the chrome://password-manager-internals/ page

Deleted: password-manager-internals.log 115 KB

password-manager-internals.log 115 KB View Download

Oh, awesome. Even as someone who knows nothing about what this log means, there are several things that stand out:

"Number of pending login managers" increases over time, ending at 57 at the end of the log.

Problem is probably with "https://www.irccloud.com/".

There are quite a few "Message: Generation invalid PasswordForm " and "Message: Generation: no non-blacklisted confirmation " messages.

I made an account on irccloud.com and logged in to the freenode #chromium channel. Every minute, there are 13 calls to "NativeBackendLibsecret::GetLoginsList".

At a guess, any single page site which dynamically creates forms will cause this issue.

I posted a test page. It causes an infinite stream of calls to GetLoginsList.

Notably - if I don't remove the previous password forms, every single password form is queried every time any password form is added, which causes N^2 calls to GetLoginsList. :O

Deleted: infinite_forms.html 680 bytes

infinite_forms.html 680 bytes View Download

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e3e1bc737140de3f68b2a40dfa1f51ad3815460c

commit e3e1bc737140de3f68b2a40dfa1f51ad3815460c
Author: erikchen <erikchen@chromium.org>
Date: Thu Jan 18 16:31:29 2018

Fix memory leak in Linux libsecret wrapper.

secret_service_search_sync returns a glist, whose elements must also be freed.
The existing code did not do this. This CL creates a new SearchHelper to wrap
this call. The results are stored in the SearchHelper, whose destructor will
correctly free the elements.

This CL updates the key_storage_libsecret and native_backend_libsecret unittests
to return real GObjects from secret_service_search. The tear down steps now
check that the GObjects are correctly unreffed by the caller.

Bug:  801702 ,  393395 
Change-Id: I0dcac794d4d79499742aea6e7e42a99135367a51
Reviewed-on: https://chromium-review.googlesource.com/871930
Commit-Queue: Erik Chen <erikchen@chromium.org>
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Reviewed-by: Vasilii Sukhanov <vasilii@chromium.org>
Reviewed-by: Christos Froussios <cfroussios@chromium.org>
Cr-Commit-Position: refs/heads/master@{#530162}
[modify] https://crrev.com/e3e1bc737140de3f68b2a40dfa1f51ad3815460c/chrome/browser/password_manager/native_backend_libsecret.cc
[modify] https://crrev.com/e3e1bc737140de3f68b2a40dfa1f51ad3815460c/chrome/browser/password_manager/native_backend_libsecret_unittest.cc
[modify] https://crrev.com/e3e1bc737140de3f68b2a40dfa1f51ad3815460c/components/os_crypt/key_storage_libsecret.cc
[modify] https://crrev.com/e3e1bc737140de3f68b2a40dfa1f51ad3815460c/components/os_crypt/key_storage_libsecret_unittest.cc
[modify] https://crrev.com/e3e1bc737140de3f68b2a40dfa1f51ad3815460c/components/os_crypt/libsecret_util_linux.cc
[modify] https://crrev.com/e3e1bc737140de3f68b2a40dfa1f51ad3815460c/components/os_crypt/libsecret_util_linux.h

Can confirm that the test page causes memory usage to increase fast.

I agree that from the log www.irccloud.com generates more calls than it should. So there is a bug in the password manager here (and in the page too as it seems to generate too many forms).

george@, can you confirm that without www.irccloud.com opened your memory consumption is doing better?

vasilii@, the memory consumption appears to be better with www.irccloud.com closed.

The test page submitted by erikchen@ also appears to increase the memory usage at a fast pace while opene (I've modified the timer to spawn forms faster).

Okay, even with the fix, there are still some slow leaks in the browser process.

repro steps: 
1) Run infinite_forms.html [change the settimeout duration to 10ms]
2) Watch browser memory climb over time.

I've attached a JSON file which contains stack traces of all leaked allocations. Load the contents in http://jsonviewer.stack.hu/ to see it easily.

Almost all leaks come from CreatePendingLoginManagers. At a guess, we never clear pending_login_managers_ unless the page goes through a navigation.

This sounds very similar to https://bugs.chromium.org/p/chromium/issues/detail?id=769617.
I'm not seeing any other leaks caused by libsecret.

So, handing bug over to vasilli. Two issues:

1) N calls to GetLoginsList every time a single password form is added.
2) Objects created in CreatePendingLoginManagers are leaked on test page.

Deleted: process_100483_malloc-leaks.json 22.3 KB

process_100483_malloc-leaks.json 22.3 KB View Download

vasilii: Ping.

I'll look into it soon.

The regression happened in r496263. The form without name doesn't match itself. Therefore, we create enormous amount of PasswordFormManager objects and every single one creates a request to the password store.

Nice find! Thanks for following up. :)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fb3209a2a966a499243dd844490879f4fb4a6ff8

commit fb3209a2a966a499243dd844490879f4fb4a6ff8
Author: Vasilii Sukhanov <vasilii@chromium.org>
Date: Tue Mar 06 18:05:28 2018

Improve password form matching.

Current code is written in a way that a form without action or name doesn't match itself. It's a performance problem because PasswordManager::CreatePendingLoginManagers continuously creates a PasswordFormManager for the same form leading to poor performance.
The code was firstly introduced in https://chromium-review.googlesource.com/c/chromium/src/+/620768. However, motivation for it is unclear after discussing together.

Bug:  801702 , 597309 
Change-Id: Ie2458d8d5340852530675b9f5d7f627132e1f3c6
Reviewed-on: https://chromium-review.googlesource.com/946490
Commit-Queue: Vasilii Sukhanov <vasilii@chromium.org>
Reviewed-by: Vadym Doroshenko <dvadym@chromium.org>
Cr-Commit-Position: refs/heads/master@{#541145}
[modify] https://crrev.com/fb3209a2a966a499243dd844490879f4fb4a6ff8/chrome/browser/password_manager/password_manager_browsertest.cc
[modify] https://crrev.com/fb3209a2a966a499243dd844490879f4fb4a6ff8/chrome/browser/password_manager/password_manager_test_base.cc
[modify] https://crrev.com/fb3209a2a966a499243dd844490879f4fb4a6ff8/chrome/browser/password_manager/password_manager_test_base.h
[add] https://crrev.com/fb3209a2a966a499243dd844490879f4fb4a6ff8/chrome/test/data/password/recurring_dynamic_form.html
[modify] https://crrev.com/fb3209a2a966a499243dd844490879f4fb4a6ff8/components/password_manager/core/browser/password_form_manager.cc
[modify] https://crrev.com/fb3209a2a966a499243dd844490879f4fb4a6ff8/components/password_manager/core/browser/password_form_manager_unittest.cc
[modify] https://crrev.com/fb3209a2a966a499243dd844490879f4fb4a6ff8/components/password_manager/core/browser/password_manager_unittest.cc