New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 689678 link

Starred by 9 users

Issue metadata

Status: Fixed
Owner:
not on Chrome anymore
Closed: Feb 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug-Regression



Sign in to add a comment

Regression: GPU memory leak in M57

Reported by flipflop...@gmail.com, Feb 7 2017

Issue description

Using Chrome Beta 64 bit (Windows 10), I recently started having an issue where occasionally, when streaming video, Memory will continuously increase in a specific process (via Resource Monitor), causing Memory usage to reach ~99% (via Task Manager) making things go slow, auto-suspending other tabs and causing audio stutter. When watching Memory usage in the Windows Resource Monitor, I noticed that one specific chrome.exe process was continuously increasing in Working Set (KB) Memory. When I first checked and noticed it, this one process was at over 15,000,000 KB and rising (I have 20 GB of RAM). After ending the process, the problem would stop and the Memory usage would go back to normal (~45%). Image references linked at the bottom.

After doing this I checked all my chrome tabs and discovered which tab was causing the Memory build up; it was a video I was streaming on http://www.crunchyroll.com. I noticed this because after ending the process, the video stopped playing and a message popped up saying "Shockwave Flash has crashed. Reload." After clicking reload, refreshing and even restarting my computer, the issue just reappears everytime, gradually increasing Memory usage in that one process. I should also note that I was using an Incognito Window and that after ending the process, all my other tabs continued to work as normal; only the Crunchyroll tab's behaviour changed. Also something interesting to note; when an episode ended and I moved onto the next one, the Working Set Memory continued to increase with the new episode, it didn't reset even though the tab had moved to a new URL (which is what caused it to keep building so high).

Here are some image references:

http://i.imgur.com/aMG28Av.png (Streaming; Resource monitor @20:59)
http://i.imgur.com/xYvFep7.png (Streaming; Task manager @20:59)
http://i.imgur.com/UstLCC8.png (Streaming; Resource monitor @21:37)
http://i.imgur.com/ztthX1V.png (Streaming; Task manager @21:37)
http://i.imgur.com/Y2aoWo6.png (After ending process; Resource monitor @21:38)
http://i.imgur.com/NWIJtqa.png (After ending process; Task manager @21:38)
http://i.imgur.com/uzJwyDD.png (After ending process; Shockwave Flash has crashed @21:38)
 

Comment 1 by ajha@chromium.org, Feb 8 2017

Labels: Needs-Triage-M57 OS-Windows

Comment 2 by hae...@gmail.com, Feb 8 2017

I've encountered the same issue with version 57 (32 and 64 bit) when playing H.264 videos from https://vimeo.com (for example https://vimeo.com/187662096). The memory consumption of the gpu process will increase seemingly without limit. In version 56 this was not the case and the memory consumption stayed within reasonable limits. Videos with VP9 codec work fine, OGG videos have the same problem.

Comment 3 by yybt...@gmail.com, Feb 9 2017

I've encountered a similar problem with Chrome version 57.0.2987.37 beta (64-bit) with Flash-based contents (without video), so I'll try to fill in some details: 

I first noticed that my memory usage was being bloated, and managed to pinpoint a certain Flash-based browser game to be the culprit. 
(Memory increase in about-real-time (as gif): https://twitter.com/kitamiya/status/829708822033813504 )

It seems to be the case that:
 - Memory increase occurs on certain screens only; that is, those where there is some kind of motion effect; 
 - On the relevant screens, no user input is necessary for the memory increase to continue; 
 - Memory increase stops (but does not flush/compress) while viewing a separate tab and keeping the culprit tab alive; 
 - Closing the culprit tab seems to initiate a memory flush/compress to free up the bloated memory properly; 
 - Memory usage (as the Working Set) would increase to fill 95%ish of the available memory: https://pbs.twimg.com/media/C4O7ngbUcAIZLHZ.jpg .

This seems to be a problem in memory allocation/flushing/compressing/gc with Flash, since running a simple Flash-based load test will trigger the same effect, although at a slightly slower pace: 
(e.g., running the test at http://wonderfl.net/c/yIH0 worked for this purpose)
Cc: benhenry@chromium.org lafo...@chromium.org vmi...@chromium.org jsc...@chromium.org ericde@chromium.org
Components: Internals>GPU Internals>Plugins>Flash
Labels: -Pri-3 -Needs-Triage-M57 M-57 Pri-1
Summary: Regression: Flash memory leak in M57 (was: Memory leak)
Comment #2 mentions the GPU process, and indeed we seem to have a significant memory regression in the GPU process at high percentiles:

https://goto.google.com/ogsiv (link for Googlers only)

Adding a bunch of relevant folks. Can we see if we can get a repro and then a memory-infra dump?
Oh, and thanks for the report! :)

Comment 6 by wal...@gmail.com, Feb 13 2017

Just to clarify things: It's not just flash, we experienced the massive memory consumption mainly with H.264 and OGG video files (in regards to comment #2, we work in the same team). See e.g. Vimeo or any other H.264 mp4 video file for reproduction.
Cc: gov...@chromium.org abdulsyed@chromium.org
Hey Ryan,

If I take the link you provided and switch it from Beta to Dev, we can see when this hit the Dev channel.

It's a little hard to interpret if the issue started in 57.0.2970.0, or came after, but it was a big enough regression to be seen on the lower channels.

Comment 8 by gov...@chromium.org, Feb 13 2017

Cc: pbomm...@chromium.org
Labels: Performance-Regression-Manual
Cc: -ericde@chromium.org brucedaw...@chromium.org
+Bruce as FYI
https://goo.gl/VBBYqq (Googlers only) shows that it is 57.0.2970.0 that introduced the regression. Since that was the first dev-channel release for 57, not sure how to generate a diff that makes sense.
Cc: ericde@chromium.org

Comment 13 by ericde@google.com, Feb 13 2017

Cc: -abdulsyed@chromium.org dalecur...@chromium.org
Labels: ReleaseBlock-Stable Type-Bug-Regression
Owner: jsc...@chromium.org
Status: Assigned (was: Unconfirmed)
Let's get this resolved (obviously) in time for Stable.

I'm going to assign an owner to get some traction on this.
Cc: rsch...@chromium.org
Cc: jbau...@chromium.org
+jbauman given mention of Windows GPU process in c#2,#6 and assortment of DXVA changes in the change log.
Cc: hajimehoshi@chromium.org
https://uma.googleplex.com/timeline_v2?sid=e86463ba0307c447e955ea25d0ec7773

Splitting by canary makes it look like 2956..2957 is the start of the regression:
https://chromium.googlesource.com/chromium/src/+log/57.0.2956.0..57.0.2957.0?pretty=fuller&n=10000

This changes how memory is reported from the GPU shared memory buffers, +hajimehoshi in case this is just a reporting change.
https://chromium.googlesource.com/chromium/src/+/6c8897721e2df9a0ec3677d0a13d677c457e5e1e
Owner: jbau...@chromium.org
I'm pretty sure this is caused by 	7bbecdb7f45d60b81e2d2578944daaafed174d93	
Project Member

Comment 20 by bugdroid1@chromium.org, Feb 14 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ab60729c01d4e123052aa4fecdf3752dc57bf9c2

commit ab60729c01d4e123052aa4fecdf3752dc57bf9c2
Author: jbauman <jbauman@chromium.org>
Date: Tue Feb 14 03:15:00 2017

Fix IMFSample leak in DXVAVideoDecodeAccelerator

Since r439662 DXVAVideoDecodeAccelerator has been adding an extra
reference to the IMFSample, causing a leak. Switch more code to use
ScopedComPtr to fix this.

BUG= 689678 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2693923003
Cr-Commit-Position: refs/heads/master@{#450218}

[modify] https://crrev.com/ab60729c01d4e123052aa4fecdf3752dc57bf9c2/media/base/win/mf_helpers.cc
[modify] https://crrev.com/ab60729c01d4e123052aa4fecdf3752dc57bf9c2/media/base/win/mf_helpers.h
[modify] https://crrev.com/ab60729c01d4e123052aa4fecdf3752dc57bf9c2/media/gpu/media_foundation_video_encode_accelerator_win.cc

Comment 21 by laforge@google.com, Feb 15 2017

Summary: Regression: GPU memory leak in M57 (was: Regression: Flash memory leak in M57)
Updating the Summary.  Is there enough data to justify a Merge-Request?
Labels: Merge-Request-57
Yeah, the fix is in canary and seems like a good idea to merge.
Project Member

Comment 23 by sheriffbot@chromium.org, Feb 15 2017

Labels: -Merge-Request-57 Hotlist-Merge-Approved Merge-Approved-57
Your change meets the bar and is auto-approved for M57. Please go ahead and merge the CL to branch 2987 manually. Please contact milestone owner if you have questions.
Owners: amineer@(clank), cmasso@(bling), ketakid@(cros), govind@(desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 24 by bugdroid1@chromium.org, Feb 16 2017

Labels: -merge-approved-57 merge-merged-2987
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1ad1e31fe09edd48255871b00db7cd8298d9e7b6

commit 1ad1e31fe09edd48255871b00db7cd8298d9e7b6
Author: John Bauman <jbauman@chromium.org>
Date: Thu Feb 16 01:02:53 2017

Fix IMFSample leak in DXVAVideoDecodeAccelerator

Since r439662 DXVAVideoDecodeAccelerator has been adding an extra
reference to the IMFSample, causing a leak. Switch more code to use
ScopedComPtr to fix this.

BUG= 689678 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2693923003
Cr-Commit-Position: refs/heads/master@{#450218}
(cherry picked from commit ab60729c01d4e123052aa4fecdf3752dc57bf9c2)

Review-Url: https://codereview.chromium.org/2699533005 .
Cr-Commit-Position: refs/branch-heads/2987@{#535}
Cr-Branched-From: ad51088c0e8776e8dcd963dbe752c4035ba6dab6-refs/heads/master@{#444943}

[modify] https://crrev.com/1ad1e31fe09edd48255871b00db7cd8298d9e7b6/media/base/win/mf_helpers.cc
[modify] https://crrev.com/1ad1e31fe09edd48255871b00db7cd8298d9e7b6/media/base/win/mf_helpers.h
[modify] https://crrev.com/1ad1e31fe09edd48255871b00db7cd8298d9e7b6/media/gpu/media_foundation_video_encode_accelerator_win.cc

Status: Fixed (was: Assigned)

Comment 26 by yybt...@gmail.com, Feb 16 2017

Thank you for the fix. I've checked the problems mentioned in comment #3 via Canary, and most problems seem to be fixed. 

One problem still remains, however, and results in a quick memory bloat (5GB/min)
as shown here: https://pbs.twimg.com/media/C4x5HFkUkAArpdJ.jpg:orig

I'm unsure if this is a related memory issue, or separate, and also if this merits being reported as a separate issue...

The problem may be slightly difficult to reproduce as well, since the relevant page requires user signup+login, and is in Japanese as well...

I'd be happy to provide dumps and the like if necessary. 


Tested using: Chrome 58.0.3013.0 (Official Build) canary (64-bit) on Windows10
 (Not reproducible in latest stable releases of Chrome/Firefox/IE)

Steps for reproduction:

1. Open game from below URL:
 http://www.dmm.com/netgame/social/-/gadgets/=/app_id=738496/
 (requires user login)
 (initial access may provide a cushion page)
2. Click on title page to enter game;
 (first play may initiate a tutorial in the likes of https://www.youtube.com/watch?v=FXvuvOgDz5g )
3. Click on "ショップ" (shop menu) on the navigation on the left of the game screen:
 https://pbs.twimg.com/media/C4x_F2uVMAAFHUo.jpg:orig
4. Leave the resulting screen open to see memory bloat

Comment 27 by yybt...@gmail.com, Feb 16 2017

After working on comment #26, it seems this particular problem is GPU-independent
(running Chrome with the GPU disabled using the flag "--disable-gpu" does not stop the memory bloating)
so I've raised this portion as a separate issue: 

Regression: memory bloat while running Flash game (Flower Knight Girl)
https://bugs.chromium.org/p/chromium/issues/detail?id=693124
I feel like this should have been caught by https://chromeperf.appspot.com/report?sid=829d93daa5a935a7e7bae4d558adaffa97ca840abd9782e475cd5cb8e46e64d4 (media.tough_video_cases on windows 10). Maybe the test isn't running long enough?
Cc: sullivan@chromium.org
Actually the win-zenbook was affected - https://chromeperf.appspot.com/report?sid=dcc39649035ed99d76e262301b8dd3e5fd46ac3d3f22335592ed12ec1ddc0852&start_rev=433947&end_rev=452513

Do we not have alerts on this? Or was it just a small enough change to be in the noise?
I filed another bug:  crbug.com/695615 

Comment 31 by yybt...@gmail.com, Feb 26 2017

I may be a bit confused, and so I'd appreciate clarification; as I understand it, 

- the current Beta is 57.0.2987.74 based on refs/branch-heads/2987@{#632}
- the above fix has been merged into this @ refs/branch-heads/2987@{#535}

and thus, the current Beta should include the above fix. 

However, I STILL SEE the memory bloats in the current Beta, in cases that were tested as fixed in the Canary 58.0.3013.0 based on master@{#450530}, as in comment 26. 
(The bloat cases are as in comment 3, and in addition I've found that videos at http://www.nicovideo.jp also leak)

Is there a chance that other related commits in master need to be merged into this branch as well to fix this issue? 


Re comment #31. Do you know what process is leaking? You can check using chrome's task manager (hit shift+esc in chrome) to see if it's flash, the GPU process, or something else.

Our UMA measurements seem to show that GPU usage is back to normal in 57.0.2987.74, but plugin process memory usage is probably still elevated due to  bug 693124 .

Comment 33 by yybt...@gmail.com, Feb 28 2017

I ran a quick test using nicovideo (playing any video will do) and the top screen from Flower Knight Girl. 

It seems I had hardware acceleration turned OFF for Beta when the problem occurred in comment 31, and so this is probably not a GPU issue. (My apologies for the confusion here. )

On the contrary, turning hardware acceleration ON actually STOPS the memory bloat, for some reason....

test ver \ GPU | on | off
---------------|----|-------
current Beta   | ok | bloat
Canary@cmnt26  | ok?| ??
current Canary | ok | ok

So one possibility is that  bug 693124  fixes this issue, and that my checking in comment 26 (before that fix) happened to use GPU. 

However,  bug 693124  occurred REGARDLESS of whether GPU is used or not, and does not seem to fit the current case pattern, which is slightly unsettling, and may point to another issue (fixed in Canary and not in Beta yet)...

My fear is that the relevant fix may be in Canary, but not yet scheduled to be merged for M57 Beta. 
Should I wait for the next Beta to be released, and file another bug if this issue reproduces there, or should I try to start something before that? 
(The best would be to check if this problem persists with the about-to-be-fixed next Beta, if it is available somehow... )

(Since analysis shows no memory bloat in Chrome's task manager, but Process Explorer shows the chrome.exe --type=ppapi to have a Working Set memory bloat, and this is true for  bug 693124  as well, I would hope it is resolved by the same issue...) 

Ok, that makes sense.  bug 693124  should happen only when flash isn't using the GPU to draw the video, while this bug should happen when it is using the GPU. The reason you can sometimes see the leak from  bug 693124  even when the GPU is enabled (e.g. on flower knight girl) is that some flash content won't use the GPU for rendering even if it's enabled.

I'd recommend waiting until beta is released with the fix for  bug 693124  before trying to test this again. If the leak persists after that, then it's a problem.

Comment 35 by yybt...@gmail.com, Feb 28 2017

Thank you for the explanation; I see now that the fix for  bug 693124  is very likely to be the wanted fix. 
I'll wait for the beta release, and file another issue iff a leak still occurs there.  

Sign in to add a comment