Issue metadata
Sign in to add a comment
|
Regression: GPU memory leak in M57
Reported by
flipflop...@gmail.com,
Feb 7 2017
|
||||||||||||||||||||||
Issue descriptionUsing Chrome Beta 64 bit (Windows 10), I recently started having an issue where occasionally, when streaming video, Memory will continuously increase in a specific process (via Resource Monitor), causing Memory usage to reach ~99% (via Task Manager) making things go slow, auto-suspending other tabs and causing audio stutter. When watching Memory usage in the Windows Resource Monitor, I noticed that one specific chrome.exe process was continuously increasing in Working Set (KB) Memory. When I first checked and noticed it, this one process was at over 15,000,000 KB and rising (I have 20 GB of RAM). After ending the process, the problem would stop and the Memory usage would go back to normal (~45%). Image references linked at the bottom. After doing this I checked all my chrome tabs and discovered which tab was causing the Memory build up; it was a video I was streaming on http://www.crunchyroll.com. I noticed this because after ending the process, the video stopped playing and a message popped up saying "Shockwave Flash has crashed. Reload." After clicking reload, refreshing and even restarting my computer, the issue just reappears everytime, gradually increasing Memory usage in that one process. I should also note that I was using an Incognito Window and that after ending the process, all my other tabs continued to work as normal; only the Crunchyroll tab's behaviour changed. Also something interesting to note; when an episode ended and I moved onto the next one, the Working Set Memory continued to increase with the new episode, it didn't reset even though the tab had moved to a new URL (which is what caused it to keep building so high). Here are some image references: http://i.imgur.com/aMG28Av.png (Streaming; Resource monitor @20:59) http://i.imgur.com/xYvFep7.png (Streaming; Task manager @20:59) http://i.imgur.com/UstLCC8.png (Streaming; Resource monitor @21:37) http://i.imgur.com/ztthX1V.png (Streaming; Task manager @21:37) http://i.imgur.com/Y2aoWo6.png (After ending process; Resource monitor @21:38) http://i.imgur.com/NWIJtqa.png (After ending process; Task manager @21:38) http://i.imgur.com/uzJwyDD.png (After ending process; Shockwave Flash has crashed @21:38)
,
Feb 8 2017
I've encountered the same issue with version 57 (32 and 64 bit) when playing H.264 videos from https://vimeo.com (for example https://vimeo.com/187662096). The memory consumption of the gpu process will increase seemingly without limit. In version 56 this was not the case and the memory consumption stayed within reasonable limits. Videos with VP9 codec work fine, OGG videos have the same problem.
,
Feb 9 2017
I've encountered a similar problem with Chrome version 57.0.2987.37 beta (64-bit) with Flash-based contents (without video), so I'll try to fill in some details: I first noticed that my memory usage was being bloated, and managed to pinpoint a certain Flash-based browser game to be the culprit. (Memory increase in about-real-time (as gif): https://twitter.com/kitamiya/status/829708822033813504 ) It seems to be the case that: - Memory increase occurs on certain screens only; that is, those where there is some kind of motion effect; - On the relevant screens, no user input is necessary for the memory increase to continue; - Memory increase stops (but does not flush/compress) while viewing a separate tab and keeping the culprit tab alive; - Closing the culprit tab seems to initiate a memory flush/compress to free up the bloated memory properly; - Memory usage (as the Working Set) would increase to fill 95%ish of the available memory: https://pbs.twimg.com/media/C4O7ngbUcAIZLHZ.jpg . This seems to be a problem in memory allocation/flushing/compressing/gc with Flash, since running a simple Flash-based load test will trigger the same effect, although at a slightly slower pace: (e.g., running the test at http://wonderfl.net/c/yIH0 worked for this purpose)
,
Feb 13 2017
Comment #2 mentions the GPU process, and indeed we seem to have a significant memory regression in the GPU process at high percentiles: https://goto.google.com/ogsiv (link for Googlers only) Adding a bunch of relevant folks. Can we see if we can get a repro and then a memory-infra dump?
,
Feb 13 2017
Oh, and thanks for the report! :)
,
Feb 13 2017
Just to clarify things: It's not just flash, we experienced the massive memory consumption mainly with H.264 and OGG video files (in regards to comment #2, we work in the same team). See e.g. Vimeo or any other H.264 mp4 video file for reproduction.
,
Feb 13 2017
Hey Ryan, If I take the link you provided and switch it from Beta to Dev, we can see when this hit the Dev channel. It's a little hard to interpret if the issue started in 57.0.2970.0, or came after, but it was a big enough regression to be seen on the lower channels.
,
Feb 13 2017
,
Feb 13 2017
,
Feb 13 2017
+Bruce as FYI
,
Feb 13 2017
https://goo.gl/VBBYqq (Googlers only) shows that it is 57.0.2970.0 that introduced the regression. Since that was the first dev-channel release for 57, not sure how to generate a diff that makes sense.
,
Feb 13 2017
,
Feb 13 2017
,
Feb 13 2017
Let's get this resolved (obviously) in time for Stable. I'm going to assign an owner to get some traction on this.
,
Feb 13 2017
,
Feb 13 2017
https://chromium.googlesource.com/chromium/src/+log/57.0.2969.0..57.0.2970.0?pretty=fuller&n=10000 change log from 2969, and here's the one from the last dev before 2970, https://chromium.googlesource.com/chromium/src/+log/57.0.2950.4..57.0.2970.0?pretty=fuller&n=10000
,
Feb 13 2017
+jbauman given mention of Windows GPU process in c#2,#6 and assortment of DXVA changes in the change log.
,
Feb 13 2017
https://uma.googleplex.com/timeline_v2?sid=e86463ba0307c447e955ea25d0ec7773 Splitting by canary makes it look like 2956..2957 is the start of the regression: https://chromium.googlesource.com/chromium/src/+log/57.0.2956.0..57.0.2957.0?pretty=fuller&n=10000 This changes how memory is reported from the GPU shared memory buffers, +hajimehoshi in case this is just a reporting change. https://chromium.googlesource.com/chromium/src/+/6c8897721e2df9a0ec3677d0a13d677c457e5e1e
,
Feb 14 2017
I'm pretty sure this is caused by 7bbecdb7f45d60b81e2d2578944daaafed174d93
,
Feb 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ab60729c01d4e123052aa4fecdf3752dc57bf9c2 commit ab60729c01d4e123052aa4fecdf3752dc57bf9c2 Author: jbauman <jbauman@chromium.org> Date: Tue Feb 14 03:15:00 2017 Fix IMFSample leak in DXVAVideoDecodeAccelerator Since r439662 DXVAVideoDecodeAccelerator has been adding an extra reference to the IMFSample, causing a leak. Switch more code to use ScopedComPtr to fix this. BUG= 689678 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Review-Url: https://codereview.chromium.org/2693923003 Cr-Commit-Position: refs/heads/master@{#450218} [modify] https://crrev.com/ab60729c01d4e123052aa4fecdf3752dc57bf9c2/media/base/win/mf_helpers.cc [modify] https://crrev.com/ab60729c01d4e123052aa4fecdf3752dc57bf9c2/media/base/win/mf_helpers.h [modify] https://crrev.com/ab60729c01d4e123052aa4fecdf3752dc57bf9c2/media/gpu/media_foundation_video_encode_accelerator_win.cc
,
Feb 15 2017
Updating the Summary. Is there enough data to justify a Merge-Request?
,
Feb 15 2017
Yeah, the fix is in canary and seems like a good idea to merge.
,
Feb 15 2017
Your change meets the bar and is auto-approved for M57. Please go ahead and merge the CL to branch 2987 manually. Please contact milestone owner if you have questions. Owners: amineer@(clank), cmasso@(bling), ketakid@(cros), govind@(desktop) For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Feb 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1ad1e31fe09edd48255871b00db7cd8298d9e7b6 commit 1ad1e31fe09edd48255871b00db7cd8298d9e7b6 Author: John Bauman <jbauman@chromium.org> Date: Thu Feb 16 01:02:53 2017 Fix IMFSample leak in DXVAVideoDecodeAccelerator Since r439662 DXVAVideoDecodeAccelerator has been adding an extra reference to the IMFSample, causing a leak. Switch more code to use ScopedComPtr to fix this. BUG= 689678 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Review-Url: https://codereview.chromium.org/2693923003 Cr-Commit-Position: refs/heads/master@{#450218} (cherry picked from commit ab60729c01d4e123052aa4fecdf3752dc57bf9c2) Review-Url: https://codereview.chromium.org/2699533005 . Cr-Commit-Position: refs/branch-heads/2987@{#535} Cr-Branched-From: ad51088c0e8776e8dcd963dbe752c4035ba6dab6-refs/heads/master@{#444943} [modify] https://crrev.com/1ad1e31fe09edd48255871b00db7cd8298d9e7b6/media/base/win/mf_helpers.cc [modify] https://crrev.com/1ad1e31fe09edd48255871b00db7cd8298d9e7b6/media/base/win/mf_helpers.h [modify] https://crrev.com/1ad1e31fe09edd48255871b00db7cd8298d9e7b6/media/gpu/media_foundation_video_encode_accelerator_win.cc
,
Feb 16 2017
,
Feb 16 2017
Thank you for the fix. I've checked the problems mentioned in comment #3 via Canary, and most problems seem to be fixed. One problem still remains, however, and results in a quick memory bloat (5GB/min) as shown here: https://pbs.twimg.com/media/C4x5HFkUkAArpdJ.jpg:orig I'm unsure if this is a related memory issue, or separate, and also if this merits being reported as a separate issue... The problem may be slightly difficult to reproduce as well, since the relevant page requires user signup+login, and is in Japanese as well... I'd be happy to provide dumps and the like if necessary. Tested using: Chrome 58.0.3013.0 (Official Build) canary (64-bit) on Windows10 (Not reproducible in latest stable releases of Chrome/Firefox/IE) Steps for reproduction: 1. Open game from below URL: http://www.dmm.com/netgame/social/-/gadgets/=/app_id=738496/ (requires user login) (initial access may provide a cushion page) 2. Click on title page to enter game; (first play may initiate a tutorial in the likes of https://www.youtube.com/watch?v=FXvuvOgDz5g ) 3. Click on "ショップ" (shop menu) on the navigation on the left of the game screen: https://pbs.twimg.com/media/C4x_F2uVMAAFHUo.jpg:orig 4. Leave the resulting screen open to see memory bloat
,
Feb 16 2017
After working on comment #26, it seems this particular problem is GPU-independent (running Chrome with the GPU disabled using the flag "--disable-gpu" does not stop the memory bloating) so I've raised this portion as a separate issue: Regression: memory bloat while running Flash game (Flower Knight Girl) https://bugs.chromium.org/p/chromium/issues/detail?id=693124
,
Feb 23 2017
I feel like this should have been caught by https://chromeperf.appspot.com/report?sid=829d93daa5a935a7e7bae4d558adaffa97ca840abd9782e475cd5cb8e46e64d4 (media.tough_video_cases on windows 10). Maybe the test isn't running long enough?
,
Feb 23 2017
Actually the win-zenbook was affected - https://chromeperf.appspot.com/report?sid=dcc39649035ed99d76e262301b8dd3e5fd46ac3d3f22335592ed12ec1ddc0852&start_rev=433947&end_rev=452513 Do we not have alerts on this? Or was it just a small enough change to be in the noise?
,
Feb 23 2017
I filed another bug: crbug.com/695615
,
Feb 26 2017
I may be a bit confused, and so I'd appreciate clarification; as I understand it,
- the current Beta is 57.0.2987.74 based on refs/branch-heads/2987@{#632}
- the above fix has been merged into this @ refs/branch-heads/2987@{#535}
and thus, the current Beta should include the above fix.
However, I STILL SEE the memory bloats in the current Beta, in cases that were tested as fixed in the Canary 58.0.3013.0 based on master@{#450530}, as in comment 26.
(The bloat cases are as in comment 3, and in addition I've found that videos at http://www.nicovideo.jp also leak)
Is there a chance that other related commits in master need to be merged into this branch as well to fix this issue?
,
Feb 28 2017
Re comment #31. Do you know what process is leaking? You can check using chrome's task manager (hit shift+esc in chrome) to see if it's flash, the GPU process, or something else. Our UMA measurements seem to show that GPU usage is back to normal in 57.0.2987.74, but plugin process memory usage is probably still elevated due to bug 693124 .
,
Feb 28 2017
I ran a quick test using nicovideo (playing any video will do) and the top screen from Flower Knight Girl. It seems I had hardware acceleration turned OFF for Beta when the problem occurred in comment 31, and so this is probably not a GPU issue. (My apologies for the confusion here. ) On the contrary, turning hardware acceleration ON actually STOPS the memory bloat, for some reason.... test ver \ GPU | on | off ---------------|----|------- current Beta | ok | bloat Canary@cmnt26 | ok?| ?? current Canary | ok | ok So one possibility is that bug 693124 fixes this issue, and that my checking in comment 26 (before that fix) happened to use GPU. However, bug 693124 occurred REGARDLESS of whether GPU is used or not, and does not seem to fit the current case pattern, which is slightly unsettling, and may point to another issue (fixed in Canary and not in Beta yet)... My fear is that the relevant fix may be in Canary, but not yet scheduled to be merged for M57 Beta. Should I wait for the next Beta to be released, and file another bug if this issue reproduces there, or should I try to start something before that? (The best would be to check if this problem persists with the about-to-be-fixed next Beta, if it is available somehow... ) (Since analysis shows no memory bloat in Chrome's task manager, but Process Explorer shows the chrome.exe --type=ppapi to have a Working Set memory bloat, and this is true for bug 693124 as well, I would hope it is resolved by the same issue...)
,
Feb 28 2017
Ok, that makes sense. bug 693124 should happen only when flash isn't using the GPU to draw the video, while this bug should happen when it is using the GPU. The reason you can sometimes see the leak from bug 693124 even when the GPU is enabled (e.g. on flower knight girl) is that some flash content won't use the GPU for rendering even if it's enabled. I'd recommend waiting until beta is released with the fix for bug 693124 before trying to test this again. If the leak persists after that, then it's a problem.
,
Feb 28 2017
Thank you for the explanation; I see now that the fix for bug 693124 is very likely to be the wanted fix. I'll wait for the beta release, and file another issue iff a leak still occurs there. |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by ajha@chromium.org
, Feb 8 2017