New issue
Advanced search Search tips

Issue 763471 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Oct 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 1
Type: Bug



Sign in to add a comment

GPU memory leak, both malloc and IOKit

Project Member Reported by erikc...@chromium.org, Sep 8 2017

Issue description

AMD FirePro D500 3072 MB
chrome: 63.0.3208.0
macOS: 10.12.6

Attaching vmmap and memory-infra trace. There's ~2GB of malloc, ~200MB in GPU MDP, 6GB in IOKit.


 
trace_gpu_trace (1).json.gz
2.8 MB Download
Cc: primiano@chromium.org ccameron@chromium.org vmi...@chromium.org sullivan@chromium.org etienneb@chromium.org
Components: Internals>GPU
Labels: -Pri-3 M-63 Pri-1
This memory leak probably first appeared on Wed Sept 6.

I'm able to reproduce the leak by: opening 5 tabs to different pages, and holding down <ctr> + <tab> to cycle through the tabs quickly.

The "test" file is the vmmap of the process. memlog.*.json.gz is the symbolized trace. memlog.*BACKUP is the unsymbolized trace.

GPU team: This should be enough information to repro and diagnose the issue?

etienneb: Most of the frames in the trace look reasonable, but some appear very suspicious. Bugs in symbolization? Also, we appear to have pruned ~98% of all allocs. :(

primiano, sullivan: I'm surprised this wasn't caught by our local memory benchmarks. Maybe there's some area we're lacking coverage? [e.g. switching tabs]
Screen Shot 2017-09-08 at 4.11.57 PM.png
230 KB View Download
memlog_92541.json.gz
61.1 KB Download
memlog_92541.json.gz.BACKUP
60.7 KB Download
The symbolized traces suggest a leak of Framebuffers and textures. I build ToT with some minimal logging and confirmed that framebuffers leak when cycling through tabs.
Screen Shot 2017-09-08 at 4.23.15 PM.png
276 KB View Download
Screen Shot 2017-09-08 at 4.23.23 PM.png
178 KB View Download
Possibly "Use GPU Discardable in Image Decode Cache"? https://chromium-review.googlesource.com/c/chromium/src/+/544102
I reverted the CL locally and confirmed that it passes my super-hacky smoke test.
Owner: ericrk@chromium.org
Status: Assigned (was: Untriaged)
Cc: perezju@chromium.org nednguyen@chromium.org
Re benchmarks, hmm I eyeballed some of the desktop benchmarks and there doesn't seem to be any bump, at least in the few random pages I picked [1]

We should indeed investigate more and figure out why this was not caught.
vmiura/erikchen: I need some more context:
1) Is this GPU specific? Does it happen only on AMD GPUs? If so do we have any coverage in the waterfall?
2) What triggers this, just cycling through tabs?

+perezju/+nednguyen: do you know if we have any coverage for multitab / tab switching on desktop (and also on mobile wouldn't hurt)?
I see a multitab_misc.typical24 story in the dashboard [2] but the last datapoint was in july and I cannot find any reference in the codebase, which makes it sound like it has been deleted at some point in the past.


[1] https://chromeperf.appspot.com/report?sid=afc9bb7b5193c5794500e7b3e116f00c11145a663e8436f4815fbd86e13d1f1f
[2] https://chromeperf.appspot.com/report?sid=d1d0ae9a3f8ffd9e79728bad0b8219e7ae9c7b250a6c96ccbb8ee7602ba3b220
Cc: -nednguyen@chromium.org nedngu...@google.com
multitab:misc:typical24 was failing & disabled on Mac. 

We have datapoint on other platform:

Win & Linux:
https://chromeperf.appspot.com/report?sid=734cbc4cdf85759fa990a82d140082fa4e46a6effb65fab36d0d84c23f67e9ff

We don't have a multitab story for mobile.

Comment 8 by ericrk@chromium.org, Sep 11 2017

Thanks for the test case. I'll take a look today. Probably something I missed in the GPU discardable patch.

Comment 9 by ericrk@chromium.org, Sep 11 2017

Re #6, once I understand why my patch caused a leak, we can identify why this wasn't caught by the Win/Linux multitab stories.
Project Member

Comment 10 by bugdroid1@chromium.org, Sep 11 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bc1b89a7d0fd2965b10544651d3be31e77f9ff64

commit bc1b89a7d0fd2965b10544651d3be31e77f9ff64
Author: Eric Karl <ericrk@chromium.org>
Date: Mon Sep 11 20:27:00 2017

Revert "Use GPU Discardable in Image Decode Cache"

This reverts commit 6a0f206ccbe0b1e81f53a79f37474a4ba3c760e8.

Appeared to be causing a memory leak. See
 crbug.com/763471 

TBR=isherman@chromium.org,vmpstr@chromium.org,piman@chromium.org,ericrk@chromium.org

Bug:  763471 
Cq-Include-Trybots: master.tryserver.blink:linux_trusty_blink_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I84e987fe848670041280b53543119c1de2f12821
Reviewed-on: https://chromium-review.googlesource.com/661074
Commit-Queue: Eric Karl <ericrk@chromium.org>
Reviewed-by: Eric Karl <ericrk@chromium.org>
Cr-Commit-Position: refs/heads/master@{#501020}
[modify] https://crrev.com/bc1b89a7d0fd2965b10544651d3be31e77f9ff64/cc/tiles/gpu_image_decode_cache.cc
[modify] https://crrev.com/bc1b89a7d0fd2965b10544651d3be31e77f9ff64/cc/tiles/gpu_image_decode_cache.h
[modify] https://crrev.com/bc1b89a7d0fd2965b10544651d3be31e77f9ff64/cc/tiles/gpu_image_decode_cache_unittest.cc
[modify] https://crrev.com/bc1b89a7d0fd2965b10544651d3be31e77f9ff64/cc/trees/layer_tree_host_impl.cc
[modify] https://crrev.com/bc1b89a7d0fd2965b10544651d3be31e77f9ff64/cc/trees/layer_tree_host_unittest.cc
[modify] https://crrev.com/bc1b89a7d0fd2965b10544651d3be31e77f9ff64/cc/trees/layer_tree_settings.h
[modify] https://crrev.com/bc1b89a7d0fd2965b10544651d3be31e77f9ff64/content/renderer/gpu/render_widget_compositor.cc
[modify] https://crrev.com/bc1b89a7d0fd2965b10544651d3be31e77f9ff64/gpu/command_buffer/service/service_discardable_manager.cc
[modify] https://crrev.com/bc1b89a7d0fd2965b10544651d3be31e77f9ff64/gpu/command_buffer/service/service_discardable_manager.h
[modify] https://crrev.com/bc1b89a7d0fd2965b10544651d3be31e77f9ff64/gpu/command_buffer/service/service_discardable_manager_unittest.cc
[modify] https://crrev.com/bc1b89a7d0fd2965b10544651d3be31e77f9ff64/tools/metrics/histograms/enums.xml
[modify] https://crrev.com/bc1b89a7d0fd2965b10544651d3be31e77f9ff64/tools/metrics/histograms/histograms.xml

Re #6 - this leak appears to be triggered by the color-conversion path in the image decode controller. This path is most frequently (always?) used on mac, which is why the impact was not seen on other platforms (and why the tab cycling bots likely missed it). Working on a fix.
Would it be reasonable to try to add some perf/memory coverage for this path? 
Erikchen: We have coverage for that path (according to #11) with the multitab test, but the test was failing on Mac ( issue 742475 )
Status: Fixed (was: Assigned)
This has been resolved, other than  issue 742475 , which is reducing coverage. We can keep working on the coverage issues there.
Project Member

Comment 15 by bugdroid1@chromium.org, Oct 27 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7ee1854e75b7c03734e4cdac8df48a221989d5fa

commit 7ee1854e75b7c03734e4cdac8df48a221989d5fa
Author: Eric Karl <ericrk@chromium.org>
Date: Fri Oct 27 23:10:54 2017

Make GPUImageDecodeCache unit tests check for GL cleanup

Currently, the GPUImageDecodeCache unit tests don't check for GL leaks.
This change causes them to verify that textures/framebuffers/
renderbuffers are cleaned up.

R=vmpstr@chromium.org

Bug:  763471 
Cq-Include-Trybots: master.tryserver.blink:linux_trusty_blink_rel
Change-Id: I8fa830ee3060951268ee433cfeb9f3038fd7c95a
Reviewed-on: https://chromium-review.googlesource.com/726487
Commit-Queue: Eric Karl <ericrk@chromium.org>
Reviewed-by: vmpstr <vmpstr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#512342}
[modify] https://crrev.com/7ee1854e75b7c03734e4cdac8df48a221989d5fa/cc/test/test_context_provider.h
[modify] https://crrev.com/7ee1854e75b7c03734e4cdac8df48a221989d5fa/cc/test/test_web_graphics_context_3d.cc
[modify] https://crrev.com/7ee1854e75b7c03734e4cdac8df48a221989d5fa/cc/test/test_web_graphics_context_3d.h
[modify] https://crrev.com/7ee1854e75b7c03734e4cdac8df48a221989d5fa/cc/tiles/gpu_image_decode_cache_unittest.cc

Sign in to add a comment