New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Regression: memory bloat while running Flash game (Flower Knight Girl)

Reported by yybt...@gmail.com, Feb 16 2017

Issue description

UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3013.0 Safari/537.36

Steps to reproduce the problem:
1. Open game from below URL:
 http://www.dmm.com/netgame/social/-/gadgets/=/app_id=738496/
 (page in Japanese, and requires user signup & login)
 (initial access may provide a cushion page)

2. Click on title page to enter game;
 (first play may initiate a tutorial in the likes of https://www.youtube.com/watch?v=FXvuvOgDz5g )

3. Click on "ショップ" (shop menu) on the navigation on the left of the game screen:
 https://pbs.twimg.com/media/C4x_F2uVMAAFHUo.jpg:orig

4. Leave the resulting screen open to see memory bloat 


What is the expected behavior?
Memory usage remains at a reasonable amount. 

What went wrong?
Process memory (working set) bloats to fill 95%ish of all available memory. 
Bloating rate is rather fast, at about 5GB/min.
 (example after a minute: https://pbs.twimg.com/media/C4x5HFkUkAArpdJ.jpg:orig )


Did this work before? Yes 56.0.2924.87

Chrome version: 58.0.3013.0  Channel: canary
OS Version: 10.0
Flash Version: Shockwave Flash 25.0 r0


Continuing from https://bugs.chromium.org/p/chromium/issues/detail?id=689678#c26 (issue is GPU-related). 
Raising as a new issue, since it seems this problem is GPU-independent, as below. 

Current Beta (57.0.2987.54) suffers from the same problem. 

Here are some additional points upon working on comment #26 of the previous issue: 

- "Record Allocation Timeline" in "Developer Tools > Memory" stops memory bloating, but only while the recording is being run

- Starting Chrome with the flag "--no-sandbox" stops memory bloating
- Starting Chrome with the flag "--disable-gpu-sandbox" does not stop memory bloating
- * Starting Chrome with the flag "--disable-gpu" does not stop memory bloating * 

- recording "Performance" in "Developer Tools" shows a repetition of 
  - Paint
  - Composite Layers
  - Update Layer Tree
  during the memory bloating

- debug log with GPU log flags on shows massive amounts of "ERROR:gles2_cmd_decoder.cc(5204)" 
  (I'm unsure if this is relevant, since disabling GPU still reproduces the problem...)

I'd be happy to provide dumps and the like if necessary.

 

Comment 1 by laforge@google.com, Feb 16 2017

Cc: pbomm...@chromium.org hajimehoshi@chromium.org rsch...@chromium.org brucedaw...@chromium.org benhenry@chromium.org gov...@chromium.org ericde@chromium.org vmi...@chromium.org jbau...@chromium.org jsc...@chromium.org lafo...@chromium.org dalecur...@chromium.org
Components: Internals>GPU Internals>Plugins>Flash
Labels: M-57
Owner: jbau...@chromium.org
Status: Assigned (was: Unconfirmed)
We should see if the fix for 689678 resolves this issue as well, or if there isn't possibly another issue.

Comment 2 by yybt...@gmail.com, Feb 16 2017

I assumed the fix for 689678 was already in the current Canary, from the revision numbers:

Current Beta:2987.54  (Revision  2987@{#516})
689678 bugfix Cr-Commit-Position:2987@{#535}; master@{#450218}
Current Canary:3013.0 (Revision               master@{#450530})

In actual testing, most problematic parts of 689678 (like http://wonderfl.net/c/yIH0 ) were fixed with the above Canary, but the problem with this issue remains, so it's likely to be another (non-GPU?) issue.

Comment 3 by laforge@google.com, Feb 16 2017

GLES is used for OpenGL, so it's likely still some flavor of a graphics issue.

John, any thoughts?

Comment 4 by yybt...@gmail.com, Feb 16 2017

"ERROR:gles2_cmd_decoder.cc(5204)" seems to occur even with just the chrome://gpu/ tab open, so is probably a separate issue... I'm sorry for the mix-up.

As a reference, attached is the actual log after just opening and closing Canary with chrome://gpu/ tab with command-line
"C:\Users\<username>\AppData\Local\Google\Chrome SxS\Application\chrome.exe" --enable-logging --enable-gpu-client-logging --enable-gpu-service-logging --enable-gpu-debugging --enable-gpu-command-logging --gpu-startup-dialog
chrome_debug.20170217-3.log
3.0 MB View Download

Comment 5 by yybt...@gmail.com, Feb 16 2017

Case in comment #4 reproduced in current Stable (56.0.2924.87) as well; attaching debug file from there.

Since the problem for the current issue is *not* seen in the Stable, this is much likely to be a separate issue. 
(I wonder if it is worth raising as another ticket?)
chrome_debug.stable.20170217.log
445 KB View Download
Components: -Internals>GPU
Owner: ----
Status: Available (was: Assigned)
If it happens with --disable-gpu it's not a GPU problem.

Comment 7 by yybt...@gmail.com, Feb 19 2017

I noticed that regular game play also causes a memory bloat (on Canary 58.0.3017.0), though at a bit slower pace (4GB/5min); map advancement seems to trigger most of the memory leakage. 

Given the number of players (nearly a million as per http://www.dmm.com/netgame_s/flower/ ), this is much likely to be noticed if not fixed by Stable. 

If this is a game-side bug (that was hidden up to the current Stable and surfaced as of the current Beta for some reason...?), then the game devs should be alerted ASAP, so a diagnosis would be very much appreciated. 

Comment 8 by yybt...@gmail.com, Feb 20 2017

I noticed that Canary seems to be leaking memory with other similar Flash-based games as well.

The good news is that I've managed to find a Flash benchmark test that will reproduce the problem easier: 
http://www.snailsanimation.com/benchmark08_play.php

The heavy test shows a significant memory leakage, and each new run adds to the bloat; two runs gave me 4GB's worth of a Working Set. 

The bad news is that this now seems to be a generic Flash issue, where one of those benchmark elements is leaking memory, and thus any Flash using that element would leak as well...?

Comment 9 by laforge@google.com, Feb 20 2017

Cc: jecl...@adobe.com smori...@adobe.com
+Adobe folks
Labels: -Pri-2 ReleaseBlock-Stable Pri-1
I am adding M57 release block stable so that this doesn't slip through our radar. 

Comment 11 by jecl...@adobe.com, Feb 21 2017

I've opened this for investigation as FLASH-4187487.
Cc: bbudge@chromium.org ihf@chromium.org

URGENT - PTAL ASAP.

We're getting VERY close to M57 Stable promotion. And 
this issue is marked as M57 stable release blocker. Pls make sure to land the fix and get it merged into the release branch ASAP so it gets enough baking time in Beta (before Stable promotion).

Know that this issue shouldn't block the release?  Remove the ReleaseBlock-Stable label or move to M58.

Thank you.

Comment 14 by laforge@google.com, Feb 22 2017

Hey Bruce,

Would you mind taking a look at this bug?  Talking w/ the Adobe folks off thread, it doesn't appear to be an issue on the Flash side of things.
Flower Knight Girl has been moved from dmm.com to https://www.nutaku.com/games/, except that I can't find it there.

More seriously, I tried using the Flash benchmark at http://www.snailsanimation.com/benchmark08_play.php and it says that Flash is disabled (I'm using canary) and chrome://plugins/ says "This site can't be reached". So, I'm currently unable to test.

I'll try installing beta on my secondary machine.

Comment 16 by laforge@google.com, Feb 23 2017

Hey Bruce,

Thanks for trying to help us get to the bottom of this.  If you click on the "Click to enable Flash Player" gray placeholder it will trigger a prompt to enable Flash for the site.
Uggh. Of course.

I was right-clicking on the "Click to enable Flash Player" which doesn't work as well.

Okay, I can repro it. Curiously enough the memory consumption does not go up, but address space does. For a 32-bit browser this causes the benchmark to crash quite quickly. One minute...
I tried ETW tracing to see if some VirtualAlloc calls were causing the growth, but they are not.

I then let the memory growth on the benchmark go to about 1 GB and then halted the benchmark and used vmmap to look at the process. It showed 796,644 KiB of Shareable memory. I was wrong when I said that it was not committed memory - it just isn't private committed memory. The shared memory is in the working set. The shareable memory in my test comes from 1,372 separate chunks, with the largest being a sub-allocated 64 MiB block, and then lots of smaller blocks ranging from 2 MiB down to 4 KiB.

Looking at the ETW sampling data showed me this suspicious call stack:

  |    |    |- chrome_child.dll!ppapi::thunk::`anonymous namespace'::Map
  |    |    |    |- chrome_child.dll!ppapi::proxy::PlatformImageData::Map
  |    |    |    |    |- chrome_child.dll!TransportDIB::GetPlatformCanvas
  |    |    |    |    |    chrome_child.dll!skia::CreatePlatformCanvasWithSharedSection
  |    |    |    |    |    |- KernelBase.dll!MapViewOfFile

It seems pretty likely that that is the problem, so I'm looking for recent changes there.

Owner: reed@google.com
Suspect:

https://codereview.chromium.org/2519083002

Shared memory grows quickly when running many flash games including this benchmark:

http://www.snailsanimation.com/benchmark08_play.php

Comment 20 by reed@chromium.org, Feb 23 2017

Cc: mtklein@chromium.org

Comment 21 by reed@chromium.org, Feb 23 2017

Cc: bunge...@chromium.org

Comment 22 by vi...@adobe.com, Feb 23 2017

We ran a bisect on Win10 x64 and got the following result.

>python.exe bisect-builds.py -a win64 -b 451874 -g 433059 -f "C:\temp\pepflashplayer.dll"
Downloading list of known revisions... (use --use-local-cache to cache and re-use the list of revisions)
...
...
You are probably looking for a change made after 443361 (known good), but no later than 443393 (first known bad).
CHANGELOG URL:
https://chromium.googlesource.com/chromium/src/+log/0e84ee428168105852608f4a900081e58f3084a2..b5c9bcd8091d35691e61a939eb2aa8201baf57c8

Note:
433059 = Chrome Stable 56.0.2924.87
451874 = Chrome Canary 58.0.3020.0
pepflashplayer.dll = Flash Player 25.0.0.119
We suspect these lines are the problem / completely nonsensical:

   HBITMAP hbitmap = static_cast<HBITMAP>(SelectObject(hdc, nullptr));
   DeleteObject(hbitmap);

SelectObject can't possibly know we're talking about HBITMAPS when we pass nullptr, can it?
It looks like that bug was fixed at r443583.
Memory leak is still present in today's canary, r452347.
In std::unique_ptr<SkCanvas> CreatePlatformCanvasWithSharedSection it looks like these two lines are potentially leaky:

    if (pixels)
      return SkCanvas::MakeRasterDirect(info, pixels, row_bytes);

MakeRasterDirect has two code paths that return nullptr which would then leak the mapped pixels. Not sure if this is related but it should be fixed at some point I think.

CreatePlatformCanvasWithSharedSection seems to call MapViewOfFile, but there's no call to UnmapViewOfFile. This should only be happening if GDI's not available due to the win32k lockdown.

Comment 28 by reed@chromium.org, Feb 23 2017

Yup (re: 27), that is the line we're pursuing. Just slogging through building the right version locally...

Comment 29 by reed@chromium.org, Feb 23 2017

(very) speculative fix https://codereview.chromium.org/2715823002/

Have not confirmed anything yet...
What about the success case? I couldn't see where in the SkCanvas destructor we were calling UnmapViewOfFile. Or does the releaseProc get stored for later use?

If so then the fix looks very plausible.

Comment 31 by reed@chromium.org, Feb 23 2017

The release_proc is attached to the bitmap, and is called by its destructor.
Project Member

Comment 32 by bugdroid1@chromium.org, Feb 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ad02b4ef8f2fcb669de0e32f7e30edf998f859b9

commit ad02b4ef8f2fcb669de0e32f7e30edf998f859b9
Author: reed <reed@google.com>
Date: Fri Feb 24 15:04:34 2017

call UnmapViewOfFile if the context failed

BUG= 693124 

Review-Url: https://codereview.chromium.org/2715823002
Cr-Commit-Position: refs/heads/master@{#452825}

[modify] https://crrev.com/ad02b4ef8f2fcb669de0e32f7e30edf998f859b9/skia/ext/raster_handle_allocator_win.cc

Comment 33 by yybt...@gmail.com, Feb 24 2017

Thank you all for looking into this problem. 

As for the first bit of comment 15, my apologies, I completely forgot DMM games does these things to foreign-identified connections... 
I'll try to check that the original problem is fixed as well, as soon as the above fix makes it into Canary. 

Comment 34 by reed@google.com, Feb 24 2017

+1 to bruce, who did all of the diagnosis (and verification)

Comment 35 by yybt...@gmail.com, Feb 25 2017

Chrome 58.0.3023.0 canary master@{#453044}
(includes above commit at master@{#452825} )

I checked with this Canary and can verify that: 
1) re: comment 1: the shop menu does NOT cause a rapid memory bloat; 
2) re: comment 7: normal play (esp. map advance) does NOT cause a rapid memory bloat;
3) re: comment 8: roughly playing around with other DMM Flash games seems ok as well. 

(as for point 3, I DID find one game where repetitively switching menus caused the chrome.exe --type=ppapi process to gain 1GB/5min of the Working Set, but this is more likely a game-side bug (I reproduced it on Chrome Stable as well), so I'll report to them first...)

Thank you all again for all the work!

Status: Assigned (was: Available)
Looks good in my tests also. Ship it!
reed@, fix seems to be working in Canary per comment #35 and #36. Please request a merge to M57 if you think it is ok do so now. Thank you.

Comment 38 by reed@chromium.org, Feb 26 2017

Cc: hcm@chromium.org
Merging seems safe. The change is to 1 file, with very low chance of affecting anythnig but plugin case.

Comment 39 by hcm@chromium.org, Feb 26 2017

Labels: Merge-Request-57
Project Member

Comment 40 by sheriffbot@chromium.org, Feb 26 2017

Labels: -Merge-Request-57 Hotlist-Merge-Approved Merge-Approved-57
Your change meets the bar and is auto-approved for M57. Please go ahead and merge the CL to branch 2987 manually. Please contact milestone owner if you have questions.
Owners: amineer@(clank), cmasso@(bling), ketakid@(cros), govind@(desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 41 Deleted

Please merge your change to M57 branch 2987 by 5:00 PM PT Tuesday (02/28) so we can take it in for next week last M57 Desktop Beta release before Stable promotion. Thank you.

Comment 43 by reed@chromium.org, Feb 28 2017

I just tried 

git drover --branch 2987 --cherry-pick ad02b4ef8f2fcb669de0e32f7e30edf998f859b9

but it seemed to be sending 2941 files ... so I killed it.

What did I do wrong?
Project Member

Comment 44 by bugdroid1@chromium.org, Feb 28 2017

Labels: -merge-approved-57 merge-merged-2987
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/94ec0a372422b9fa5a4d4c7260fadef1d93bd660

commit 94ec0a372422b9fa5a4d4c7260fadef1d93bd660
Author: Florin Malita <fmalita@chromium.org>
Date: Tue Feb 28 15:15:40 2017

call UnmapViewOfFile if the context failed

BUG= 693124 

Review-Url: https://codereview.chromium.org/2715823002
Cr-Commit-Position: refs/heads/master@{#452825}
(cherry picked from commit ad02b4ef8f2fcb669de0e32f7e30edf998f859b9)

Review-Url: https://codereview.chromium.org/2719853007 .
Cr-Commit-Position: refs/branch-heads/2987@{#715}
Cr-Branched-From: ad51088c0e8776e8dcd963dbe752c4035ba6dab6-refs/heads/master@{#444943}

[modify] https://crrev.com/94ec0a372422b9fa5a4d4c7260fadef1d93bd660/skia/ext/raster_handle_allocator_win.cc

The drover cmd looks good (landed cleanly for me), maybe something wrong with your tree.
Cc: brajkumar@chromium.org
Labels: Needs-Feedback
Tested this issue on Windows-10 using chrome latest M57 #57.0.2987.88 by following steps mentioned in the original comment, But as mentioned in the comment #15 unable to play the game from the provided link of step-1. 

Navigated to http://www.snailsanimation.com/benchmark08_play.php as per comment #8 and performed testing and observed all the test has passed except Heavy test. While performing Heavy test unable to see much memory consumption from task manager.

Could anyone let us know is the above steps is the right way to reproduce this issue? If not is there any test case or consistent repro steps available to verify this issue from Chrome-TE end.

Thanks!

Comment 47 by yybt...@gmail.com, Mar 1 2017

Re: comment #46, to elaborate on what I briskly mentioned in comment #33: DMM games tends to block/redirect users identified as being a foreign connection; I believe they base this on the location of the user IP address. 

Quoting the English Wikia for the game ( http://flowerknight.wikia.com/wiki/Flower_Knight_Girl:_FAQ#Troubleshooting )

> Troubleshooting
> Q: Why can’t I play the game?
> A: DMM only allows Japan-based players to play their game, so you need to be in Japan or fool the server into thinking you’re in Japan, via proxies or cookies. The R18 version doesn't require you to have a Japanese IP address or even just the right cookies, but you do have to disable your ad blocker if you use one. In other words, playing the R18 version requires less effort than playing the safe version.

So to reproduce from outside Japan, you'll have to either

1) use VPN/proxies or the sort to seem to be using a Japanese IP address; or 
2) test on the R18 version (which uses the same basic code, and reproduces this issue) at
   http://www.dmm.co.jp/netgame/social/-/gadgets/=/app_id=329993/
   but this version is NSFW; even the tutorial will play one scene that is NSFW, so please be warned. 

Comment 48 by yybt...@gmail.com, Mar 1 2017

Add to comment #47: 

These steps may be useful to trick the server for method 1 above: 
http://flowerknight.wikia.com/wiki/Tutorial:_Get_Started
Labels: TE-Verified-57.0.2987.88
Thank you Bruce.

Verified the issue on Windows 7 and 10, please find steps followed below :

Steps followed :
1. Install and launch chrome 57.0.2987.88
2. Open task manager or process explorer and check the Chrome working set memory consumption
3. Visit "http://www.snailsanimation.com/benchmark08_play.php" and start test

Observed behavior :
There wasn't any spike in Memory w.r.t Chrome Working set.

Note : Myself and Bruce have verified.
Status: Fixed (was: Assigned)
Based on Comment#49,Marking the bug as Fixed.

Sign in to add a comment