The issue seems to be platform-specific. I was able to reproduce in Windows 10, but not on linux. So this is probably not an bug in blink since the is nothing platform-specific in the 2D canvas code, except for IOSurface code path but that only affects on Mac.

Suspecting the leak could be somewhere in windows-specific gpu-accelerated text rendering code in skia.

jvanverth@: PTAL

FYI what I am seeing on windows is that memory consumption the renderer process is fluctuating up and down, with a long-term upwards trend. Occasionaly it jumps up suddenly by a few MB.

The GPU text code is also platform-agnostic, so I'm not sure where this would be coming from. But I'll take a look.

Thank you for your quick response and your effort. Are there any news on this issue? We need this bug to be fixed, because some of our projects can't be finished otherwise. 

Sorry, I haven't had a chance to look at it yet. I'll get to it in the next couple of days.

Thank you for your feedback. No problem. I'm looking forward hearing from you.

It's almost 3 weeks, since this bug has been reported and reproduced. Could you please tell me, in which time frame bugs like these are fixed, so that we are able to plan, when we could finish our projects without workarounds? Thank you!

#7 - there is no timeline. It could also take months or more (at least while it is Priority 2).

I am struggling with this issue, too.

#2 - Additional note: Firefox works fine for me with the exact code on the same machine. So, it seems to be Chrome related only.

It would be great if this issue could get a higher priority since it results in a browser crash and I could not find a workaround for fillText() yet.

This is causing our application (CEF desktop software) to crash repeatedly. Any idea on how we can make this higher priority? Our software is a long-term data collection and viewing application. This crash is affecting all of our customers and is costing us money.

Sorry about the delay. I believe I've been able to duplicate it in the current Chrome Canary, but I can't with this Skia sample which does the same thing:

https://skia-review.googlesource.com/c/17787/

This makes me dubious that it's Skia, but I'll take a look at the drawText code. It's possible that we're internally generating text blobs and not freeing them.

I don't think this is Skia. I dug through the text code and it appears for drawText() and drawPosText() the temporary text blobs are being freed.

If I simply return at the top of SkGpuDevice::drawText(), drawPosText(), and drawTextBlob(), I get no text drawn but still one of the processes -- the one associated with the tab, it looks like -- keeps growing in memory.

Canvas2D fillText() appears to be using SkCanvas::drawTextBlob(). Are there text blobs being created somewhere above Skia and then not freed?

Re-adding junov@.

If this is in Skia, it's rather indirect. If I comment out the invocation of Draw in CanvasRenderingContext2D::DrawTextInternal() it still happens.  Commenting out parts of this method, it looks like it's this line (384 in my current tree):

   PaintCanvas* c = DrawingCanvas(); 

Maybe the AcceleratedImageBufferSurface is leaking?

I can also see this on Mac in the Activity Monitor, so it's platform-independent.

Just wanted to check in and see if there have been any new updates on this, or if there might be a timeline on a fix. We are stuck restarting the renderer process in our CEF application until this is solved. :/

any update on this?

Increasing priority given the severity of the problem.

Xida, could you take a look at this? Perhaps run it through a leak checker? On windows our options are limited. Perhaps try Dr. Memory.

martin.gawlita@: We have a windows 10 desktop and here is what I have just tried.

1. Open chrome://flags, and look for "Accelerated 2D canvas", and enable that if it is not enabled.
2. Open a new tab, and run the attached html file.

After 20 mins, the memory usage for that tab is ~15MB.

Could you please update your chrome to latest version which is 59.0.3071.15? If it still repros on your side, please log the exact steps. Thank you so much.

Thank you for your effort solving this issue. Sorry for my delayed answer. I've just followed your steps and it seems, that this issue is solved. I would run some further test to be sure and will update you tomorrow. 

Hello,

I updated to the latest Chrome Version 60.0.3112.78 (Offizieller Build) (64-Bit) on my Windows 7 machine.

See attached screenshots for result. I used Chrome's taskmanager to monitor memory usage.

Test #1 - 2D Canvas hardware acceleration enabled

fmtest-start.JPG 
shows memory right after start

fmtest-end.JPG 
shows memory after approx 45mins.


Test #2 - 2D Canvas hardware acceleration disabled

fmtest-without2dhardwareacceleration-start.JPG 
shows memory right after start 

fmtest-without2dhardwareacceleration-end.JPG 
shows memmory after approx. 60 minutes.



In both cases the memory still leaks (although it seems to leak a little bit slower now).

In addition please check this old bug:
https://bugs.chromium.org/p/chromium/issues/detail?id=674816#c_ts1487669803

Thanks for your work.

Deleted: fmtest-start.JPG 56.9 KB

	fmtest-start.JPG 56.9 KB View Download

Deleted: fmtest-end.JPG 57.0 KB

	fmtest-end.JPG 57.0 KB View Download

Deleted: fmtest-without2dhardwareacceleration-start.JPG 77.9 KB

	fmtest-without2dhardwareacceleration-start.JPG 77.9 KB View Download

Deleted: fmtest-without2dhardwareacceleration-end.JPG 77.4 KB

	fmtest-without2dhardwareacceleration-end.JPG 77.4 KB View Download

OK, here is a bit more investigation and this is the strangest thing I've ever seen.

If I open the test html without any other tab opening, no memory leak at all after 15 mins.

If I open the test html with another tab open, for example, chrome://flags, then I can see the memory usage for the fillText() tab keeps growing, roughly 1MB per second.

bennyschuetz@: could you please confirm that this is the case?

Hmm, not just windows. I can repro this on linux as well.

I was also able to reproduce the memory leak again, when the hardware acceleration is switched on. It took a bit longer than in the past but still the tab crashes after about 3-4 hours. I've had several tabs open running chrome with the following version: 60.0.3112.78 (Offizieller Build) (64-Bit) (Kohorte: 60_78_win)

@junov

Nice find. That's strange indeed.

I can confirm that with just one tab open and 2D hardware acceleration disabled - there is no memory growth.

With one tab open and 2D hardware acceleration enabled - there is still a memory leak. However, with only one tab the growth is roughly 10 x slower than with multiple tabs open.

I ran a memoryInfra profile and was able to determine that we are leaking PSS (memory shared with another process) all other chrome processes have a stable PSS, which indicates that the leak probably involves memory shared with an external process. (I think?)  Unfortunately, the current memory infra instrumentation does not provide enough insight to investigate this problem much further.

Flagging the memory instrumentation folks for further guidance.

any update on this? For what it's worth, we are using Chromium Embedded and are seeing this memory leak in our single page application (no extra tabs, just the 3 main processes)

Pinging memory instrumentation folks.
Please provide guidance for debugging this shared memory leak.  How can we track down the sources of PSS leaks? 

Are there any new findings on this issue? It's very annoying to restart our application repeatedly because of the unnessary reinit work we have to do. 

We are still looking for a update on this as well. Still having to restart our software every couple of hours.

Same here. This bug causes a lot of trouble in our software. 

Desperately looking for an update and some feedback on this, too.

ccing primiano@ to provide guidance on memory-infra stide.

Re #25, PSS is not necessarily shared. It is private+shared, so it could just be private memory.
It would help if anybody attached some traces here. 

In any case, I just tried to repro with the attachment #0 in chrome 62.0.3202.94 on Mac  and renderer's memory seems to slowly creep and increase. So there might be an actual bug here.
Instructions to get stack traces of the allocations: 
https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/heap_profiler.md

I was unable to find anything interesting in the heap dumps.  Attaching it for better trained eyes to take a look.  The attachment is a trace file with symbolized memory_infra data.

Deleted: trace_mem3.json.gz 10.1 MB

trace_mem3.json.gz 10.1 MB Download

The trace is not symbolized and is quite hard to read as such.
+erikchen can anybody in the desktop team look at this?

Please find attached my trace. I hope you find something which helps to track down the cause of the memory leak. Please let me know if you need any more information.

Thanks in advance.

Deleted: trace_issue703297trace.json.gz 14.0 MB

trace_issue703297trace.json.gz 14.0 MB Download

In the trace in #37, we see the PSS and private footprint increase over time on the renderer process that is running the experiment, but none of the allocation categories are growing.

@primiano: I don't know how this can be explained. Are there ways of allocating memory that are lacking memory-infra instrumentation?

I was able to repro this locally. It looks like checkRealloc continually allocates larger and larger segments [the total number of objects stays constant at 2].

The screenshot shows a stack trace of the offending stack trace.

We've also observed this in the wild with heap profiling. I'm not sure if this is a bug in Canvas or Skia.

Deleted: Screen Shot 2017-12-07 at 1.00.23 PM.png 212 KB

	Screen Shot 2017-12-07 at 1.00.23 PM.png 212 KB View Download

Deleted: Screen Shot 2017-12-07 at 2.59.22 PM.png 392 KB

	Screen Shot 2017-12-07 at 2.59.22 PM.png 392 KB View Download

This does look like a Skia issue after all. Adding bsalomon@ to see if he knows something about this message passing system.

My best guess is that as SkTextBlobs are being deleted they post a message that lands in SkMessageBus::Inbox::fMessages. The messages aren't being received so they just accumulate. Perhaps we should be calling GrTextBlobCache::checkPurge after each flush.

Yeah, that's what I think too. I'll see what I can add.

It'll take me a bit to get a Chromium build going. In the meantime, if someone could test out this patch that would be appreciated: https://skia-review.googlesource.com/c/skia/+/82686

Quick update: each tab has two SkMessageBox::Inbox instances -- one for Canvas and one for the compositor. Any purge messages go to both inboxes. Normally all pending messages get flushed when a new TextBlob is added, which in this case is happening for the Canvas instance, but not the for compositor one since it isn't using text. And so the compositor inbox just keeps filling up until the tab crashes.

This can't be fixed entirely on the Skia side -- we'll need some way to periodically force clean up on the compositor side. I'll add the interface and then will need some help hooking it up.

The following revision refers to this bug:
  https://skia.googlesource.com/skia/+/76d917cef19aabfdc1247336f58237800bd71875

commit 76d917cef19aabfdc1247336f58237800bd71875
Author: Jim Van Verth <jvanverth@google.com>
Date: Wed Dec 13 14:54:42 2017

Ensure we flush TextBlobCache message queue.

If we create and delete TextBlobs without actually renderering them,
their deletion messages can back up in the message queue. This adds
a routine to GrContext to ensure these messages get flushed.

Bug:  703297 
Change-Id: Icc222373ac2a954dc3b77190cad79070ea562ba2
Reviewed-on: https://skia-review.googlesource.com/82686
Commit-Queue: Jim Van Verth <jvanverth@google.com>
Reviewed-by: Brian Salomon <bsalomon@google.com>

[modify] https://crrev.com/76d917cef19aabfdc1247336f58237800bd71875/src/gpu/text/GrTextBlobCache.h
[modify] https://crrev.com/76d917cef19aabfdc1247336f58237800bd71875/src/gpu/GrContext.cpp
[modify] https://crrev.com/76d917cef19aabfdc1247336f58237800bd71875/src/gpu/text/GrTextBlobCache.cpp
[modify] https://crrev.com/76d917cef19aabfdc1247336f58237800bd71875/include/gpu/GrContext.h

The following revision refers to this bug:
  https://skia.googlesource.com/skia/+/474d68791965f20f8e0dfa2bfb4d87300f1f29e0

commit 474d68791965f20f8e0dfa2bfb4d87300f1f29e0
Author: Jim Van Verth <jvanverth@google.com>
Date: Thu Dec 14 18:52:49 2017

Send TextBlobCache purge messages only to owning cache.

Bug:  703297 
Change-Id: I95cdaa5bdebadd5ce88ae3ee468c59baa08353c6
Reviewed-on: https://skia-review.googlesource.com/85046
Reviewed-by: Brian Salomon <bsalomon@google.com>
Commit-Queue: Jim Van Verth <jvanverth@google.com>

[modify] https://crrev.com/474d68791965f20f8e0dfa2bfb4d87300f1f29e0/include/private/SkTArray.h
[modify] https://crrev.com/474d68791965f20f8e0dfa2bfb4d87300f1f29e0/include/core/SkTextBlob.h
[modify] https://crrev.com/474d68791965f20f8e0dfa2bfb4d87300f1f29e0/src/gpu/text/GrTextBlobCache.h
[modify] https://crrev.com/474d68791965f20f8e0dfa2bfb4d87300f1f29e0/src/gpu/text/GrTextBlobCache.cpp
[modify] https://crrev.com/474d68791965f20f8e0dfa2bfb4d87300f1f29e0/src/core/SkTextBlob.cpp
[modify] https://crrev.com/474d68791965f20f8e0dfa2bfb4d87300f1f29e0/src/gpu/GrContext.cpp
[modify] https://crrev.com/474d68791965f20f8e0dfa2bfb4d87300f1f29e0/include/private/SkMessageBus.h

The last patch should fix it and has been rolled into Chrome, please verify.

Awesome! Just to clarify, can you give the version of Chrome/Chromium that the fix has been rolled into? Thank you!

It looks like it's in the latest Chrome Canary. At the very least it's in version 65.0.3296.0.

Just wanted to add...we updated our software which uses CEF to a version that had this fix (CEF 3.3325.1746.ge81cdf2 / Chromium 65.0.3325.146). We've been running long term tests with many canvas elements rapidly updating text, and the memory leak, which previously would cause a quick crash, definitely seems to be fixed. Thank you!