Investigate ZRAM usage on low-end Android |
||
Issue description$subject
,
May 22 2018
Test scenario:
- Load a page in Chrome on Gobo (1GB), O MR1.
- Go back to homescreen
- Make the shell take a large amount of RAM (adb shell, then: a=$(seq 1 30000000))
$ adb shell dumpsys meminfo
[...]
Total PSS by process:
344,021K: sh (pid 29868)
[...]
43,215K: Previous
22,741K: com.google.android.apps.chrome (pid 24452 / activities)
20,474K: com.google.android.apps.chrome:sandboxed_process0 (pid 24484)
[...]
Total RAM: 958,920K (status critical)
Free RAM: 257,939K ( 27,563K cached pss + 67,332K cached kernel + 163,044K free)
Used RAM: 830,540K ( 752,828K used pss + 77,712K kernel)
Lost RAM: -99,925K
ZRAM: 44,840K physical used for 210,196K in swap ( 719,184K total swap)
Tuning: 128 (large 256), oom 184,320K, restore limit 61,440K (low-ram)
Swapped out memory for the browser process:
$ adb shell dumpsys meminfo com.google.android.apps.chrome
Applications Memory Usage (in Kilobytes):
Uptime: 414159115 Realtime: 609530968
** MEMINFO in pid 24452 [com.google.android.apps.chrome] **
Pss Private Private SwapPss Heap Heap Heap
Total Dirty Clean Dirty Size Alloc Free
------ ------ ------ ------ ------ ------ ------
Native Heap 2428 2404 0 4494 12800 6334 6465
Dalvik Heap 1832 1820 0 206 3487 2632 855
Dalvik Other 1190 1188 0 86
Stack 120 120 0 284
Ashmem 0 0 0 0
Other dev 29 0 24 0
.so mmap 1319 64 20 181
.apk mmap 2310 84 1504 2048
.dex mmap 3205 0 796 12
.oat mmap 683 0 0 0
.art mmap 1447 748 0 211
Other mmap 128 0 120 4
GL mtrack 608 608 0 0
Unknown 182 180 0 386
TOTAL 23393 7216 2464 7912 16287 8966 7320
[...]
Renderer process:
$ adb shell dumpsys meminfo com.google.android.apps.chrome:sandboxed_process0
Applications Memory Usage (in Kilobytes):
Uptime: 414251911 Realtime: 609623764
** MEMINFO in pid 24484 [com.google.android.apps.chrome:sandboxed_process0] **
Pss Private Private SwapPss Heap Heap Heap
Total Dirty Clean Dirty Size Alloc Free
------ ------ ------ ------ ------ ------ ------
Native Heap 1066 1040 0 2654 6400 3800 2599
Dalvik Heap 140 124 0 98 1465 969 496
Dalvik Other 110 108 0 27
Stack 88 88 0 212
Ashmem 40 40 0 0
Other dev 12 0 8 0
.so mmap 687 48 4 112
.apk mmap 2366 104 1540 2028
.dex mmap 953 0 112 4
.oat mmap 281 0 0 0
.art mmap 817 252 4 60
Other mmap 7 0 0 4
Unknown 310 304 0 8319
TOTAL 20395 2108 1668 13518 7865 4769 3095
[...]
Some memory was swapped out for both of these processes.
Browser:
$ tools/android/native_lib_memory/parse_smaps.py --pid 24452 | head -n 50
INFO:devil.utils.cmd_helper:[host]> /media/lizeb/code/clankium/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s A6R9K17A17901790 shell '( cat /proc/24452/smaps );echo %$?'
Total Swap Size (kB) = 11696
/data/app/com.google.android.apps.chrome-GCnF_tlZNhH2urhPGxxXSQ==/base.apk r--p: 1960 kB (Total Size: 2052 kB)
/vendor/lib/egl/libGLES_mali.so r--p: 692 kB (Total Size: 816 kB)
[anon:libc_malloc] rw-p: 504 kB (Total Size: 2048 kB)
[anon:libc_malloc] rw-p: 444 kB (Total Size: 1024 kB)
[anon:libc_malloc] rw-p: 388 kB (Total Size: 512 kB)
/dev/ashmem/dalvik-large object space allocation (deleted) rw-p: 384 kB (Total Size: 388 kB)
[anon:libc_malloc] rw-p: 380 kB (Total Size: 1024 kB)
[anon:libc_malloc] rw-p: 372 kB (Total Size: 512 kB)
[anon:libc_malloc] rw-p: 360 kB (Total Size: 512 kB)
[anon:libc_malloc] rw-p: 348 kB (Total Size: 512 kB)
[anon:libc_malloc] rw-p: 324 kB (Total Size: 512 kB)
[anon:libc_malloc] rw-p: 280 kB (Total Size: 512 kB)
[anon:libc_malloc] rw-p: 268 kB (Total Size: 512 kB)
[anon:libc_malloc] rw-p: 256 kB (Total Size: 512 kB)
[anon:libc_malloc] rw-p: 244 kB (Total Size: 1024 kB)
[anon:libc_malloc] rw-p: 224 kB (Total Size: 512 kB)
[anon:libc_malloc] rw-p: 204 kB (Total Size: 2048 kB)
[anon:libc_malloc] rw-p: 200 kB (Total Size: 1024 kB)
[anon:libc_malloc] rw-p: 168 kB (Total Size: 512 kB)
[anon:.bss] rw-p: 136 kB (Total Size: 1380 kB)
/data/dalvik-cache/arm/system@framework@boot-framework.art rw-p: 112 kB (Total Size: 4916 kB)
/system/lib/libskia.so r--p: 92 kB (Total Size: 148 kB)
/data/app/com.google.android.apps.chrome-GCnF_tlZNhH2urhPGxxXSQ==/base.apk rw-p: 88 kB (Total Size: 96 kB)
/system/lib/libstagefright.so r--p: 88 kB (Total Size: 88 kB)
[anon:libc_malloc] rw-p: 84 kB (Total Size: 512 kB)
/system/lib/libmedia.so r--p: 64 kB (Total Size: 68 kB)
[anon:.bss] rw-p: 56 kB (Total Size: 76 kB)
/dev/ashmem/dalvik-LinearAlloc (deleted) rw-p: 52 kB (Total Size: 128 kB)
/dev/ashmem/dalvik-large object space allocation (deleted) rw-p: 52 kB (Total Size: 56 kB)
/system/lib/libexif.so r--p: 48 kB (Total Size: 48 kB)
/dev/ashmem/dalvik-large object space allocation (deleted) rw-p: 48 kB (Total Size: 52 kB)
/system/lib/libpdfium.so r--p: 44 kB (Total Size: 44 kB)
[stack:24516] rw-p: 36 kB (Total Size: 1004 kB)
/dev/ashmem/dalvik-large object space allocation (deleted) rw-p: 36 kB (Total Size: 40 kB)
/dev/ashmem/dalvik-large object space allocation (deleted) rw-p: 36 kB (Total Size: 44 kB)
/dev/ashmem/dalvik-LinearAlloc (deleted) rw-p: 36 kB (Total Size: 128 kB)
/system/lib/libcrypto.so r--p: 36 kB (Total Size: 40 kB)
[stack] rw-p: 36 kB (Total Size: 8188 kB)
/system/lib/libicuuc.so r--p: 32 kB (Total Size: 44 kB)
/dev/ashmem/dalvik-large object space allocation (deleted) rw-p: 28 kB (Total Size: 32 kB)
/system/lib/libmedia_omx.so r--p: 28 kB (Total Size: 28 kB)
/dev/ashmem/dalvik-zygote space (deleted) rw-p: 24 kB (Total Size: 664 kB)
/dev/ashmem/dalvik-large object space allocation (deleted) rw-p: 24 kB (Total Size: 28 kB)
/dev/ashmem/dalvik-large object space allocation (deleted) rw-p: 24 kB (Total Size: 28 kB)
/dev/ashmem/dalvik-large object space allocation (deleted) rw-p: 24 kB (Total Size: 28 kB)
/data/dalvik-cache/arm/system@framework@boot-ext.art rw-p: 20 kB (Total Size: 224 kB)
/data/dalvik-cache/arm/system@framework@boot-mediatek-telephony-base.art rw-p: 20 kB (Total Size: 64 kB)
/system/lib/libandroid_runtime.so r--p: 20 kB (Total Size: 60 kB)
Renderer:
$ tools/android/native_lib_memory/parse_smaps.py --pid 24484 | head -n 50
INFO:devil.utils.cmd_helper:[host]> /media/lizeb/code/clankium/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s A6R9K17A17901790 shell '( cat /proc/24484/smaps );echo %$?'
Total Swap Size (kB) = 17540
/data/app/com.google.android.apps.chrome-GCnF_tlZNhH2urhPGxxXSQ==/base.apk r--p: 1940 kB (Total Size: 2052 kB)
rw-p: 812 kB (Total Size: 1824 kB)
[anon:libc_malloc] rw-p: 752 kB (Total Size: 1024 kB)
/vendor/lib/egl/libGLES_mali.so r--p: 692 kB (Total Size: 816 kB)
rw-p: 508 kB (Total Size: 512 kB)
rw-p: 508 kB (Total Size: 512 kB)
rw-p: 508 kB (Total Size: 512 kB)
rw-p: 508 kB (Total Size: 512 kB)
rw-p: 504 kB (Total Size: 512 kB)
rw-p: 444 kB (Total Size: 2016 kB)
r-xp: 416 kB (Total Size: 416 kB)
[anon:libc_malloc] rw-p: 412 kB (Total Size: 512 kB)
rw-p: 408 kB (Total Size: 1984 kB)
/dev/ashmem/dalvik-large object space allocation (deleted) rw-p: 384 kB (Total Size: 388 kB)
rw-p: 380 kB (Total Size: 512 kB)
[anon:libc_malloc] rw-p: 368 kB (Total Size: 512 kB)
r-xp: 356 kB (Total Size: 484 kB)
rw-p: 348 kB (Total Size: 1696 kB)
[anon:libc_malloc] rw-p: 328 kB (Total Size: 512 kB)
[anon:libc_malloc] rw-p: 320 kB (Total Size: 512 kB)
rw-p: 272 kB (Total Size: 1936 kB)
[anon:libc_malloc] rw-p: 260 kB (Total Size: 512 kB)
[anon:libc_malloc] rw-p: 224 kB (Total Size: 512 kB)
rw-p: 208 kB (Total Size: 512 kB)
[anon:libc_malloc] rw-p: 200 kB (Total Size: 512 kB)
[anon:libc_malloc] rw-p: 192 kB (Total Size: 3584 kB)
rw-p: 188 kB (Total Size: 192 kB)
[anon:.bss] rw-p: 168 kB (Total Size: 1380 kB)
[anon:libc_malloc] rw-p: 164 kB (Total Size: 1024 kB)
r-xp: 132 kB (Total Size: 132 kB)
r-xp: 132 kB (Total Size: 132 kB)
r-xp: 132 kB (Total Size: 132 kB)
rw-p: 120 kB (Total Size: 640 kB)
rw-p: 112 kB (Total Size: 120 kB)
rw-p: 100 kB (Total Size: 120 kB)
rw-p: 96 kB (Total Size: 120 kB)
rw-p: 92 kB (Total Size: 120 kB)
/system/lib/libskia.so r--p: 92 kB (Total Size: 148 kB)
/data/app/com.google.android.apps.chrome-GCnF_tlZNhH2urhPGxxXSQ==/base.apk rw-p: 88 kB (Total Size: 96 kB)
/system/lib/libstagefright.so r--p: 88 kB (Total Size: 88 kB)
rw-p: 80 kB (Total Size: 120 kB)
[stack:24554] rw-p: 68 kB (Total Size: 1008 kB)
/system/lib/libmedia.so r--p: 68 kB (Total Size: 68 kB)
rw-p: 56 kB (Total Size: 120 kB)
rw-p: 52 kB (Total Size: 120 kB)
/dev/ashmem/dalvik-large object space allocation (deleted) rw-p: 52 kB (Total Size: 56 kB)
/dev/ashmem/dalvik-LinearAlloc (deleted) rw-p: 52 kB (Total Size: 128 kB)
rw-p: 48 kB (Total Size: 176 kB)
rw-p: 48 kB (Total Size: 120 kB)
Some remarks:
- Swap overestimates the swapped-out memory as some is shared, hence the discrepencies between "dumpsys meminfo" and parse_smaps.py.
- Some annotated entries:
- Browser process:
- /data/app/com.google.android.apps.chrome-GCnF_tlZNhH2urhPGxxXSQ==/base.apk r--p: 1960 kB (Total Size: 2052 kB)
This is the relocations, as this is read-only and mapped from the .so, yet dirty memory. Almost the entirety of relocations have been swapped out?
- /vendor/lib/egl/libGLES_mali.so r--p: 692 kB (Total Size: 816 kB)
Relocations for libGLES_mali.so? It seems large.
- [anon:libc_malloc] rw-p: 504 kB (Total Size: 2048 kB)
malloc()
- /dev/ashmem/dalvik-LinearAlloc (deleted) rw-p: 52 kB (Total Size: 128 kB)
Young generation for the Java GC?
- Renderer process:
- rw-p: 812 kB (Total Size: 1824 kB)
Either some mapping from Chrome, or a third-party library.
- r-xp: 416 kB (Total Size: 416 kB)
V8 generated code, as it is anonymous, yet executable, in the renderer process.
- rw-p: 508 kB (Total Size: 512 kB)
PartitionAlloc, or the V8 heap?
Preliminary observations:
- Swapping out relocations is likely to hurt performance when Chrome comes back to the foreground. It is also an indication that Chrome doesn't do a lot of background processing.
- libGLES_mali.so seems to have a lot of relocations?
- Swapping out the young generation in dalvik is also likely to cause jank at the next minor garbage collection
- V8 executable code can be paged out. Depending on what gets removed, this could be an issue or not.
- The nice annotation we have for libc_malloc is likely not an option for us, as it comes from task_mmu.c in the kernel (https://github.com/torvalds/linux/blob/master/fs/proc/task_mmu.c)
,
May 23 2018
What does meminfo com.google.android.apps.chrome say before the shell loop? I'm interested in how much swapping happens just from the churn in chrome.
,
May 23 2018
Repeated the same test, this time loading cnn.com on Gobo, then going to the homescreen, and letting the device there, sitting idle. Browser process: $ adb shell dumpsys meminfo com.google.android.apps.chrome Applications Memory Usage (in Kilobytes): Uptime: 479417688 Realtime: 674789540 ** MEMINFO in pid 24452 [com.google.android.apps.chrome] ** Pss Private Private SwapPss Heap Heap Heap Total Dirty Clean Dirty Size Alloc Free ------ ------ ------ ------ ------ ------ ------ Native Heap 7599 7580 0 4052 23040 9645 13394 Dalvik Heap 1942 1816 0 1594 5209 3923 1286 Dalvik Other 1421 1420 0 177 Stack 268 268 0 152 Ashmem 121 4 0 0 Other dev 33 0 28 0 .so mmap 4782 384 560 161 .apk mmap 7596 396 964 1736 .ttf mmap 105 0 0 0 .dex mmap 4192 0 1284 12 .oat mmap 1635 0 8 0 .art mmap 1532 984 0 134 Other mmap 891 0 776 4 GL mtrack 648 648 0 0 Unknown 446 444 0 174 TOTAL 41407 13944 3620 8196 28249 13568 14680 Renderer: $ adb shell dumpsys meminfo com.google.android.apps.chrome:sandboxed_process3 Applications Memory Usage (in Kilobytes): Uptime: 479469181 Realtime: 674841034 ** MEMINFO in pid 1454 [com.google.android.apps.chrome:sandboxed_process3] ** Pss Private Private SwapPss Heap Heap Heap Total Dirty Clean Dirty Size Alloc Free ------ ------ ------ ------ ------ ------ ------ Native Heap 2638 2616 0 6940 14336 8538 5797 Dalvik Heap 423 272 0 38 1467 971 496 Dalvik Other 113 112 0 27 Stack 112 112 0 248 Ashmem 126 64 0 0 Other dev 12 0 8 0 .so mmap 1389 52 0 88 .apk mmap 22113 500 14536 1632 .ttf mmap 276 0 140 0 .dex mmap 1254 0 132 4 .oat mmap 551 0 24 0 .art mmap 908 296 4 16 Other mmap 46 4 0 0 Unknown 20834 20828 0 54651 TOTAL 114439 24856 14844 63644 15803 9509 6293 So, quite a lot of swapping for the renderer. Details of swapped-out areas in the renderer (biggest areas): Total Swap Size (kB) = 67856 rw-p: 3208 kB (Total Size: 3220 kB) rw-p: 1864 kB (Total Size: 2016 kB) rw-p: 1560 kB (Total Size: 1904 kB) /data/app/com.google.android.apps.chrome-GCnF_tlZNhH2urhPGxxXSQ==/base.apk r--p: 1548 kB (Total Size: 2052 kB) [anon:libc_malloc] rw-p: 1512 kB (Total Size: 2560 kB) rw-p: 1300 kB (Total Size: 1412 kB) rw-p: 1272 kB (Total Size: 1808 kB) rw-p: 1260 kB (Total Size: 1292 kB) [anon:libc_malloc] rw-p: 1188 kB (Total Size: 2560 kB) rw-p: 1172 kB (Total Size: 2016 kB) rw-p: 1144 kB (Total Size: 1376 kB) rw-p: 1132 kB (Total Size: 2016 kB) rw-p: 1076 kB (Total Size: 1760 kB) rw-p: 1024 kB (Total Size: 1024 kB) rw-p: 1016 kB (Total Size: 2016 kB) rw-p: 980 kB (Total Size: 1632 kB) rw-p: 872 kB (Total Size: 2016 kB) [anon:libc_malloc] rw-p: 868 kB (Total Size: 1024 kB) rw-p: 860 kB (Total Size: 2000 kB) [anon:libc_malloc] rw-p: 852 kB (Total Size: 1024 kB) rw-p: 844 kB (Total Size: 1424 kB) rw-p: 832 kB (Total Size: 1024 kB) rw-p: 788 kB (Total Size: 1024 kB) rw-p: 776 kB (Total Size: 1024 kB) rw-p: 772 kB (Total Size: 2016 kB) rw-p: 764 kB (Total Size: 1536 kB) [anon:libc_malloc] rw-p: 740 kB (Total Size: 1024 kB) rw-p: 716 kB (Total Size: 2000 kB) /vendor/lib/egl/libGLES_mali.so r--p: 704 kB (Total Size: 816 kB) rw-p: 648 kB (Total Size: 1024 kB) rw-p: 636 kB (Total Size: 1024 kB) rw-p: 616 kB (Total Size: 1024 kB) rw-p: 564 kB (Total Size: 2016 kB) [anon:libc_malloc] rw-p: 516 kB (Total Size: 2048 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 512 kB (Total Size: 512 kB) rw-p: 500 kB (Total Size: 512 kB) r-xp: 484 kB (Total Size: 484 kB) r-xp: 484 kB (Total Size: 484 kB) rw-p: 480 kB (Total Size: 512 kB) rw-p: 480 kB (Total Size: 512 kB) r-xp: 480 kB (Total Size: 484 kB) rw-p: 448 kB (Total Size: 532 kB) So yes, quite a lot of our allocations got swapped out.
,
May 23 2018
I have an interesting graph to share (use google@ account): https://docs.google.com/spreadsheets/d/11gmoQs7lq9a7-TjQX8ciQmaVpLxJAk49XzmtFa1UXUA/edit?usp=sharing This is ToT Chrome (~2 months old) playing long Youtube video on 512MiB Android Go device (gobo_512). This use case pushes Chrome to the limit and used to crash ( issue 831305 ), but we made it better by lowering demuxer limits and implementing pressure polling (issue 813909). I.e. playing a video (especially high res one, like 720p) on 512MiB Go device is relatively easy way to see how Chrome + Android behave when there is no memory. For example kswapd takes ~60% of CPU, everything becomes really slow to the point of Android complaining about long lock contentions. Video plays smoothly though :) Also keep in mind that it's not only Chrome to blame, playing a video creates media.codec process that uses ~30MiB of EGL memory, see b/77695661.
,
Jun 18 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e34d38aa50ce3bcc887c44be480cb18ca522a3d6 commit e34d38aa50ce3bcc887c44be480cb18ca522a3d6 Author: Benoit Lize <lizeb@chromium.org> Date: Mon Jun 18 09:19:02 2018 tools: Dump a process' heap, and analyze it. Adds a tool to dump the content of all anonymous mapping in a given process, and one to show statistics about the content. From a process, this shows: - Total anonymous mappings size - For each page, whether it's present XOR swapped - The total number of 0 pages - Pages with identical non-zero content - Compressed size of non-zero pages. The compression ratio is derived using zlib, which is likely to be overestimating the compression ratio, as it's better than ZRAM's compression algorithm. Tested on Android and Linux. Sample output (cnn.com on Android Go): Total pages = 36532 (149.64MB) Total zero pages = 15191 (41.58%) Total present zero pages = 754 (3.09MB) Total size of non-zero pages = 87412736 (87.41MB) Total compressed size = 28994296 (33.17%) Duplicated non-zero pages = 812 Max non-zero pages with the same content = 387 Swapped pages = 231 (946.18kB) Non-present pages = 14667 (60.08MB) Freed = 5232960 (5.23MB) TBR: brucedawson@chromium.org Change-Id: Iba991f8a57d01591e72359163711064297ea8658 Bug: 845459 Reviewed-on: https://chromium-review.googlesource.com/1087059 Commit-Queue: Benoit L <lizeb@chromium.org> Reviewed-by: Benoit L <lizeb@chromium.org> Reviewed-by: Bruce Dawson <brucedawson@chromium.org> Reviewed-by: Egor Pasko <pasko@chromium.org> Reviewed-by: agrieve <agrieve@chromium.org> Cr-Commit-Position: refs/heads/master@{#567960} [modify] https://crrev.com/e34d38aa50ce3bcc887c44be480cb18ca522a3d6/BUILD.gn [add] https://crrev.com/e34d38aa50ce3bcc887c44be480cb18ca522a3d6/tools/dump_process_memory/BUILD.gn [add] https://crrev.com/e34d38aa50ce3bcc887c44be480cb18ca522a3d6/tools/dump_process_memory/OWNERS [add] https://crrev.com/e34d38aa50ce3bcc887c44be480cb18ca522a3d6/tools/dump_process_memory/analyze_dumps.py [add] https://crrev.com/e34d38aa50ce3bcc887c44be480cb18ca522a3d6/tools/dump_process_memory/dump_process.cc |
||
►
Sign in to add a comment |
||
Comment 1 by bugdroid1@chromium.org
, May 22 2018