Issue metadata
Sign in to add a comment
|
Regression: memory bloat while running Flash game (Flower Knight Girl)
Reported by
yybt...@gmail.com,
Feb 16 2017
|
||||||||||||||||||||||
Issue descriptionUserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3013.0 Safari/537.36 Steps to reproduce the problem: 1. Open game from below URL: http://www.dmm.com/netgame/social/-/gadgets/=/app_id=738496/ (page in Japanese, and requires user signup & login) (initial access may provide a cushion page) 2. Click on title page to enter game; (first play may initiate a tutorial in the likes of https://www.youtube.com/watch?v=FXvuvOgDz5g ) 3. Click on "ショップ" (shop menu) on the navigation on the left of the game screen: https://pbs.twimg.com/media/C4x_F2uVMAAFHUo.jpg:orig 4. Leave the resulting screen open to see memory bloat What is the expected behavior? Memory usage remains at a reasonable amount. What went wrong? Process memory (working set) bloats to fill 95%ish of all available memory. Bloating rate is rather fast, at about 5GB/min. (example after a minute: https://pbs.twimg.com/media/C4x5HFkUkAArpdJ.jpg:orig ) Did this work before? Yes 56.0.2924.87 Chrome version: 58.0.3013.0 Channel: canary OS Version: 10.0 Flash Version: Shockwave Flash 25.0 r0 Continuing from https://bugs.chromium.org/p/chromium/issues/detail?id=689678#c26 (issue is GPU-related). Raising as a new issue, since it seems this problem is GPU-independent, as below. Current Beta (57.0.2987.54) suffers from the same problem. Here are some additional points upon working on comment #26 of the previous issue: - "Record Allocation Timeline" in "Developer Tools > Memory" stops memory bloating, but only while the recording is being run - Starting Chrome with the flag "--no-sandbox" stops memory bloating - Starting Chrome with the flag "--disable-gpu-sandbox" does not stop memory bloating - * Starting Chrome with the flag "--disable-gpu" does not stop memory bloating * - recording "Performance" in "Developer Tools" shows a repetition of - Paint - Composite Layers - Update Layer Tree during the memory bloating - debug log with GPU log flags on shows massive amounts of "ERROR:gles2_cmd_decoder.cc(5204)" (I'm unsure if this is relevant, since disabling GPU still reproduces the problem...) I'd be happy to provide dumps and the like if necessary.
,
Feb 16 2017
I assumed the fix for 689678 was already in the current Canary, from the revision numbers: Current Beta:2987.54 (Revision 2987@{#516}) 689678 bugfix Cr-Commit-Position:2987@{#535}; master@{#450218} Current Canary:3013.0 (Revision master@{#450530}) In actual testing, most problematic parts of 689678 (like http://wonderfl.net/c/yIH0 ) were fixed with the above Canary, but the problem with this issue remains, so it's likely to be another (non-GPU?) issue.
,
Feb 16 2017
GLES is used for OpenGL, so it's likely still some flavor of a graphics issue. John, any thoughts?
,
Feb 16 2017
"ERROR:gles2_cmd_decoder.cc(5204)" seems to occur even with just the chrome://gpu/ tab open, so is probably a separate issue... I'm sorry for the mix-up. As a reference, attached is the actual log after just opening and closing Canary with chrome://gpu/ tab with command-line "C:\Users\<username>\AppData\Local\Google\Chrome SxS\Application\chrome.exe" --enable-logging --enable-gpu-client-logging --enable-gpu-service-logging --enable-gpu-debugging --enable-gpu-command-logging --gpu-startup-dialog
,
Feb 16 2017
Case in comment #4 reproduced in current Stable (56.0.2924.87) as well; attaching debug file from there. Since the problem for the current issue is *not* seen in the Stable, this is much likely to be a separate issue. (I wonder if it is worth raising as another ticket?)
,
Feb 17 2017
If it happens with --disable-gpu it's not a GPU problem.
,
Feb 19 2017
I noticed that regular game play also causes a memory bloat (on Canary 58.0.3017.0), though at a bit slower pace (4GB/5min); map advancement seems to trigger most of the memory leakage. Given the number of players (nearly a million as per http://www.dmm.com/netgame_s/flower/ ), this is much likely to be noticed if not fixed by Stable. If this is a game-side bug (that was hidden up to the current Stable and surfaced as of the current Beta for some reason...?), then the game devs should be alerted ASAP, so a diagnosis would be very much appreciated.
,
Feb 20 2017
I noticed that Canary seems to be leaking memory with other similar Flash-based games as well. The good news is that I've managed to find a Flash benchmark test that will reproduce the problem easier: http://www.snailsanimation.com/benchmark08_play.php The heavy test shows a significant memory leakage, and each new run adds to the bloat; two runs gave me 4GB's worth of a Working Set. The bad news is that this now seems to be a generic Flash issue, where one of those benchmark elements is leaking memory, and thus any Flash using that element would leak as well...?
,
Feb 20 2017
+Adobe folks
,
Feb 21 2017
I am adding M57 release block stable so that this doesn't slip through our radar.
,
Feb 21 2017
I've opened this for investigation as FLASH-4187487.
,
Feb 22 2017
,
Feb 22 2017
URGENT - PTAL ASAP. We're getting VERY close to M57 Stable promotion. And this issue is marked as M57 stable release blocker. Pls make sure to land the fix and get it merged into the release branch ASAP so it gets enough baking time in Beta (before Stable promotion). Know that this issue shouldn't block the release? Remove the ReleaseBlock-Stable label or move to M58. Thank you.
,
Feb 22 2017
Hey Bruce, Would you mind taking a look at this bug? Talking w/ the Adobe folks off thread, it doesn't appear to be an issue on the Flash side of things.
,
Feb 23 2017
Flower Knight Girl has been moved from dmm.com to https://www.nutaku.com/games/, except that I can't find it there. More seriously, I tried using the Flash benchmark at http://www.snailsanimation.com/benchmark08_play.php and it says that Flash is disabled (I'm using canary) and chrome://plugins/ says "This site can't be reached". So, I'm currently unable to test. I'll try installing beta on my secondary machine.
,
Feb 23 2017
Hey Bruce, Thanks for trying to help us get to the bottom of this. If you click on the "Click to enable Flash Player" gray placeholder it will trigger a prompt to enable Flash for the site.
,
Feb 23 2017
Uggh. Of course. I was right-clicking on the "Click to enable Flash Player" which doesn't work as well. Okay, I can repro it. Curiously enough the memory consumption does not go up, but address space does. For a 32-bit browser this causes the benchmark to crash quite quickly. One minute...
,
Feb 23 2017
I tried ETW tracing to see if some VirtualAlloc calls were causing the growth, but they are not. I then let the memory growth on the benchmark go to about 1 GB and then halted the benchmark and used vmmap to look at the process. It showed 796,644 KiB of Shareable memory. I was wrong when I said that it was not committed memory - it just isn't private committed memory. The shared memory is in the working set. The shareable memory in my test comes from 1,372 separate chunks, with the largest being a sub-allocated 64 MiB block, and then lots of smaller blocks ranging from 2 MiB down to 4 KiB. Looking at the ETW sampling data showed me this suspicious call stack: | | |- chrome_child.dll!ppapi::thunk::`anonymous namespace'::Map | | | |- chrome_child.dll!ppapi::proxy::PlatformImageData::Map | | | | |- chrome_child.dll!TransportDIB::GetPlatformCanvas | | | | | chrome_child.dll!skia::CreatePlatformCanvasWithSharedSection | | | | | |- KernelBase.dll!MapViewOfFile It seems pretty likely that that is the problem, so I'm looking for recent changes there.
,
Feb 23 2017
Suspect: https://codereview.chromium.org/2519083002 Shared memory grows quickly when running many flash games including this benchmark: http://www.snailsanimation.com/benchmark08_play.php
,
Feb 23 2017
,
Feb 23 2017
,
Feb 23 2017
We ran a bisect on Win10 x64 and got the following result. >python.exe bisect-builds.py -a win64 -b 451874 -g 433059 -f "C:\temp\pepflashplayer.dll" Downloading list of known revisions... (use --use-local-cache to cache and re-use the list of revisions) ... ... You are probably looking for a change made after 443361 (known good), but no later than 443393 (first known bad). CHANGELOG URL: https://chromium.googlesource.com/chromium/src/+log/0e84ee428168105852608f4a900081e58f3084a2..b5c9bcd8091d35691e61a939eb2aa8201baf57c8 Note: 433059 = Chrome Stable 56.0.2924.87 451874 = Chrome Canary 58.0.3020.0 pepflashplayer.dll = Flash Player 25.0.0.119
,
Feb 23 2017
We suspect these lines are the problem / completely nonsensical: HBITMAP hbitmap = static_cast<HBITMAP>(SelectObject(hdc, nullptr)); DeleteObject(hbitmap); SelectObject can't possibly know we're talking about HBITMAPS when we pass nullptr, can it?
,
Feb 23 2017
It looks like that bug was fixed at r443583.
,
Feb 23 2017
Memory leak is still present in today's canary, r452347.
,
Feb 23 2017
In std::unique_ptr<SkCanvas> CreatePlatformCanvasWithSharedSection it looks like these two lines are potentially leaky:
if (pixels)
return SkCanvas::MakeRasterDirect(info, pixels, row_bytes);
MakeRasterDirect has two code paths that return nullptr which would then leak the mapped pixels. Not sure if this is related but it should be fixed at some point I think.
,
Feb 23 2017
CreatePlatformCanvasWithSharedSection seems to call MapViewOfFile, but there's no call to UnmapViewOfFile. This should only be happening if GDI's not available due to the win32k lockdown.
,
Feb 23 2017
Yup (re: 27), that is the line we're pursuing. Just slogging through building the right version locally...
,
Feb 23 2017
(very) speculative fix https://codereview.chromium.org/2715823002/ Have not confirmed anything yet...
,
Feb 23 2017
What about the success case? I couldn't see where in the SkCanvas destructor we were calling UnmapViewOfFile. Or does the releaseProc get stored for later use? If so then the fix looks very plausible.
,
Feb 23 2017
The release_proc is attached to the bitmap, and is called by its destructor.
,
Feb 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ad02b4ef8f2fcb669de0e32f7e30edf998f859b9 commit ad02b4ef8f2fcb669de0e32f7e30edf998f859b9 Author: reed <reed@google.com> Date: Fri Feb 24 15:04:34 2017 call UnmapViewOfFile if the context failed BUG= 693124 Review-Url: https://codereview.chromium.org/2715823002 Cr-Commit-Position: refs/heads/master@{#452825} [modify] https://crrev.com/ad02b4ef8f2fcb669de0e32f7e30edf998f859b9/skia/ext/raster_handle_allocator_win.cc
,
Feb 24 2017
Thank you all for looking into this problem. As for the first bit of comment 15, my apologies, I completely forgot DMM games does these things to foreign-identified connections... I'll try to check that the original problem is fixed as well, as soon as the above fix makes it into Canary.
,
Feb 24 2017
+1 to bruce, who did all of the diagnosis (and verification)
,
Feb 25 2017
Chrome 58.0.3023.0 canary master@{#453044}
(includes above commit at master@{#452825} )
I checked with this Canary and can verify that:
1) re: comment 1: the shop menu does NOT cause a rapid memory bloat;
2) re: comment 7: normal play (esp. map advance) does NOT cause a rapid memory bloat;
3) re: comment 8: roughly playing around with other DMM Flash games seems ok as well.
(as for point 3, I DID find one game where repetitively switching menus caused the chrome.exe --type=ppapi process to gain 1GB/5min of the Working Set, but this is more likely a game-side bug (I reproduced it on Chrome Stable as well), so I'll report to them first...)
Thank you all again for all the work!
,
Feb 26 2017
Looks good in my tests also. Ship it!
,
Feb 26 2017
reed@, fix seems to be working in Canary per comment #35 and #36. Please request a merge to M57 if you think it is ok do so now. Thank you.
,
Feb 26 2017
Merging seems safe. The change is to 1 file, with very low chance of affecting anythnig but plugin case.
,
Feb 26 2017
,
Feb 26 2017
Your change meets the bar and is auto-approved for M57. Please go ahead and merge the CL to branch 2987 manually. Please contact milestone owner if you have questions. Owners: amineer@(clank), cmasso@(bling), ketakid@(cros), govind@(desktop) For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Feb 28 2017
Please merge your change to M57 branch 2987 by 5:00 PM PT Tuesday (02/28) so we can take it in for next week last M57 Desktop Beta release before Stable promotion. Thank you.
,
Feb 28 2017
I just tried git drover --branch 2987 --cherry-pick ad02b4ef8f2fcb669de0e32f7e30edf998f859b9 but it seemed to be sending 2941 files ... so I killed it. What did I do wrong?
,
Feb 28 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/94ec0a372422b9fa5a4d4c7260fadef1d93bd660 commit 94ec0a372422b9fa5a4d4c7260fadef1d93bd660 Author: Florin Malita <fmalita@chromium.org> Date: Tue Feb 28 15:15:40 2017 call UnmapViewOfFile if the context failed BUG= 693124 Review-Url: https://codereview.chromium.org/2715823002 Cr-Commit-Position: refs/heads/master@{#452825} (cherry picked from commit ad02b4ef8f2fcb669de0e32f7e30edf998f859b9) Review-Url: https://codereview.chromium.org/2719853007 . Cr-Commit-Position: refs/branch-heads/2987@{#715} Cr-Branched-From: ad51088c0e8776e8dcd963dbe752c4035ba6dab6-refs/heads/master@{#444943} [modify] https://crrev.com/94ec0a372422b9fa5a4d4c7260fadef1d93bd660/skia/ext/raster_handle_allocator_win.cc
,
Feb 28 2017
The drover cmd looks good (landed cleanly for me), maybe something wrong with your tree.
,
Mar 1 2017
Tested this issue on Windows-10 using chrome latest M57 #57.0.2987.88 by following steps mentioned in the original comment, But as mentioned in the comment #15 unable to play the game from the provided link of step-1. Navigated to http://www.snailsanimation.com/benchmark08_play.php as per comment #8 and performed testing and observed all the test has passed except Heavy test. While performing Heavy test unable to see much memory consumption from task manager. Could anyone let us know is the above steps is the right way to reproduce this issue? If not is there any test case or consistent repro steps available to verify this issue from Chrome-TE end. Thanks!
,
Mar 1 2017
Re: comment #46, to elaborate on what I briskly mentioned in comment #33: DMM games tends to block/redirect users identified as being a foreign connection; I believe they base this on the location of the user IP address. Quoting the English Wikia for the game ( http://flowerknight.wikia.com/wiki/Flower_Knight_Girl:_FAQ#Troubleshooting ) > Troubleshooting > Q: Why can’t I play the game? > A: DMM only allows Japan-based players to play their game, so you need to be in Japan or fool the server into thinking you’re in Japan, via proxies or cookies. The R18 version doesn't require you to have a Japanese IP address or even just the right cookies, but you do have to disable your ad blocker if you use one. In other words, playing the R18 version requires less effort than playing the safe version. So to reproduce from outside Japan, you'll have to either 1) use VPN/proxies or the sort to seem to be using a Japanese IP address; or 2) test on the R18 version (which uses the same basic code, and reproduces this issue) at http://www.dmm.co.jp/netgame/social/-/gadgets/=/app_id=329993/ but this version is NSFW; even the tutorial will play one scene that is NSFW, so please be warned.
,
Mar 1 2017
Add to comment #47: These steps may be useful to trick the server for method 1 above: http://flowerknight.wikia.com/wiki/Tutorial:_Get_Started
,
Mar 1 2017
Thank you Bruce. Verified the issue on Windows 7 and 10, please find steps followed below : Steps followed : 1. Install and launch chrome 57.0.2987.88 2. Open task manager or process explorer and check the Chrome working set memory consumption 3. Visit "http://www.snailsanimation.com/benchmark08_play.php" and start test Observed behavior : There wasn't any spike in Memory w.r.t Chrome Working set. Note : Myself and Bruce have verified.
,
Mar 2 2017
Based on Comment#49,Marking the bug as Fixed. |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by laforge@google.com
, Feb 16 2017Components: Internals>GPU Internals>Plugins>Flash
Labels: M-57
Owner: jbau...@chromium.org
Status: Assigned (was: Unconfirmed)