My colleagues' Windows canary up-time is 94 hours (so ~4 days).

We have both filed feedback reports, including this bug # in them.

The Windows system is running 54.0.2836.0

Based on the UMA data it seems that this regressed around August 20th - data for Aug 17th shows a clear spike and steep decline toward higher memory usage, while data from Aug 25th shows a much less steep decline.

Which stats did you look at?

Memory.Browser.Committed is the stat I'm looking at right now.

Effect is noticably more pronounced on x64, with a 110MB increase in mean Memory.Browser.Committed, versus a 40MB increase on x86.

+danakj/vmpstr who were discussing a potential GPU memory leak recently though IIUC it was very much an edge case whereas the repro steps here are a base case.

That was with WebGL textures, interacting with canvas.

Note Memory.Browser.Committed is a new stat added Aug 16, so the spike in #4 could just be the very first initial accrual of UMA reported?

https://chromium.googlesource.com/chromium/src/+/9114ac31b82529d145ba6fe3eda9def53df6fa06

I looked at Canary, 95th pctile, 1-day aggregation, Memory.GPU, Memory.Browser.Committed, Memory.Renderer.Committed, Memory.Browser.Large2, and leak does not look clear cut in light of above new UMAs.

wez@ unsure how to find the crashes you noted in #2. It looks like we need the crash id unless there's a field to search in crashes I don't see.

Gah, yes, it does look like UMA ramp-up, over the course of ~3-4 days - which would imply that the browser process typically creeps up over a 3-4 day timeframe. Can we break things down by browser-uptime in the UMA dashboard?

Re feedback reports: We filed feedback reports (i.e. logs etc), not crashes - you'll need to search for this bug # in the Chrome feedback tool.

Regardless of the UMA stat being misleading, things have definitely regressed recently.

I've seen several bugs relating to OOMs allocating discardable locked shared memory for Skia - could we have a GPU-memory regression, perhaps?

That's pretty handwavey. Anything causing OOM would cause skia allocation failures, since we frequently go there. Maybe ericrk has some ideas on how to track down a general memory regression.

Re #14: True, but we see an increase in GPU memory held, in UMA stats logged during ChromeOS high-memory-pressure events, hence the suggestion that GPU memory usage has gone up for some reason.

- Is this a stability issue?  Did you or Mark see an OOM crash?

- What is task manager measuring vs. Browser.Memory.Committed?  (I can research this.)

- The browser may be allocating a lot of discardable memory that hasn't been returned to the OS.  On  a 4GB or 8GB Chromebox there may not be a low memory pressure signal.

- What does chrome://discards say?

Memory.Browser.Committed has trended up slightly since 8/25 (at the 50th percentile).  This measures the total committed memory (private + shared + 'image').  It's a new metric so there isn't much data yet when viewed as a 7-day aggregate.

https://uma.googleplex.com/p/chrome/timeline_v2/?sid=f1b17707f513028639ed29deb6c9b61f

Memory.Browser.Large2 (private working set) has remained stable or even dropped slightly.  This is what is in task manager.

This is the code we use to extract memory metrics on ChromeOS.

https://cs.chromium.org/chromium/src/base/process/process_metrics_linux.cc?rcl=0&l=329

It looks like some of this can be found in ChromeOS Task Manager by right clicking on the browser process and selecting additional fields.  You might be able to read /proc/<pid>/totmaps directly if you have a developer device and can access the filesystem.

Removing S-S-D label until we gather more data.

We can probably track this down by using "memory-infra" tracing.

If you have a chrome instance that shows the problematic memory behavior locally, can you take a quick trace with only the "memory-infra" category enabled. This takes periodic dumps of memory, and we really only need 1 dump, so tracing for a few seconds is all we need.

This trace should show a much more detailed breakdown of memory in use by Chrome (over what UMA shows).

If you upload one of these traces, I'm happy to take a look.

Re #19: OK, will try to remember to that when browser process next gets bloaty (presumably in a few days' time). Unfortunately the colleague w/ the 2.5GB browser process is presently OOO.

OK, I tried getting a memory-infra trace, but it seems that the memory-dump-manager is broken right now (see issue 643438).

Please have the fix baked in canary and merged to 2840 branch before 09/05 Monday for Dev Release Scheduled on Tuesday 09/06.The same will be promoted to Beta on 09/08.

TE can try a manual bisect on the blocking issue : 643438.

As of now I am tagging with RBS,since the investigation is still in progress.   Please feel free to change if needed.

Still repros on 54.0.2840.13. Chromebox was janky after being left idle over the weekend, and Task Manager showed a ~1GB browser process. Tried to get a memory-infra trace but hit issue 643438 again.

Sorry about short comment, no time to investigate further now. However we noticed huge memory leak on 53.x when resizing window with canvas in it. It didn't matter if it was basic canvas or WebGL enabled canvas.

On latest OSX (El Capitan) and MacBook Pro with integrated GPU the total system memory usage climbed to several gigabytes in one minute. Chrome memory usage stays constant, but system memory goes up. This would indicate GPU handling memory leak. The memory is released if Chrome is completely killed (before OSX says you're short on memory and basically hang whole system).  

Not that it would be usual use case, mostly came up during canvas based page development, but alarming was that the memory was not released if the single page was closed. Only complete Chrome reboot helps.

Friendly ping, this a stable blocker for M54, please try to have a fix in by the first week of October so it can be fixed in time for the release.

> OK, I tried getting a memory-infra trace, but it seems that the memory-dump-manager is broken right now (see issue 643438).

That's fixed it looks like.

Any updates?

M54 Stable is launch is scheduled during next week.James, do you think we can punt this to M55 and continue to investigate? If so please feel free to update the labels accordingly.

Haven't repro'd this issue lately; suggest WontFix and I'll file something
new, with trace logs, if it comes back.

FWIW, I have a full memory dump of the browser process on Windows with a 2.1+GB private working set. If someone wants to look at it or has ideas on how I can poke at it to help debug this...

Leaving this bug open as per #32, but removing the blocker for M54 as per #31.

ericrk: As per #32, dcheng@ has a full memory dump from Windows, and I have a memory-infra trace of an 825MB browser process on M55 from just now. Would you like to take a look?

Based on a memory-infra trace, looks like my system has 700MB+ in "unspecified" malloc'd memory right now.

Sorry, missed this - if the memory-infra trace is showing 700MB+ in unspecified malloc memory, this is probably something that's better handled by the heap profiler portion of memory-infra. See here: https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/heap_profiler.md

dcheng@, when you say you have a full memory dump, do you mean a process memory dump? I don't know if there's a good way to extract info from that. dskiba@ may know.

wez@, happy to take a look at the trace - can you either send a link or post to the bug? Although, with 700+ in unspecified, we probably won't be able to get much out of it.

Re #35: According to Drive, two traces are already shared with you :)  Their names start with "trace_crbug-641518", so you should be able to find them in Shared With Me. Let me know if not and I'll try to re-share.

Can you also share traces with me?

Done!

Unfortunately the traces don't seem too actionable (unless dskiba@ has some ideas). I think we're likely going to have to use heap profiling (or some other technique).

I'm going to try to take some heap profiling traces on Windows to see if I can find any leaks - hopefully things are leaking consistently and we can see some small leaks in a few minutes/hours :D

In the trace wez@ shared (...longer_still) malloc is at 770 MiB. I then took a trace on my Linux machine (where Chrome was running for days) and malloc was at ~530 MiB. Less, but still roughly in the same ballpark.

So I restarted my Chrome with --enable-heap-profiling, and hopefully I'll get more data once usage climbs back to ~500 MiB. I don't know if it's possible to specify that flag on ChromeOS though.

dskiba@, were you able to catch anything with --enable-heap-profiling?

With memory-infra tracing fixed on windows, I'll try to get a repro on Windows. That said, I haven't seen browser procs on my machines > ~250mb, but I may try switching from Canary > Dev, as I currently restart ~daily which might be masking this.

Sadly no, I've hit issue 669158 twice (browser crashes while taking trace), so right now I'm on 2-day old process.

Last time Chrome was running for 6 days, and yet Browser / malloc was just 140 MiB. I guess I didn't have right tabs opened...

Is this same as  issue 678768  ? It looks like that one started at 54.0.2809.0, matches the versions here.
I guess at least one of the issue pointed out (about browser being 2GB) is same as the leak. I am not sure about the first one here.

Also wez@ have you tried a heap profiler trace?

Re #44: No; I'm not sure what a "heap profiler trace" is.  I've tried getting memory-infra traces, but they usually just crash the browser.

Heap profiler trace is a special mode of memory-infra tracing. You can grab one by running with --enable-heap-profiling - although this does have a non-zero performance impact.

This is what dskiba@ was trying to grab, but it sounds like he wasn't able to hit the leak.

Ah, in that case, no I haven't tried heap profiler, and I can't since this is a retail-mode ChromeOS device.

As per #44, it sounds like this is a duplicate of  issue 678768 .