New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 793537 link

Starred by 2 users
Status: Closed
Owner:
gurcheta...@chromium.org
Closed: Aug 24
Cc:
█ marc...@chromium.org
█ dbehr@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

Investigate Chrome OS cached RAM

Project Member Reported by gurcheta...@chromium.org, Dec 9 2017 
To reproduce on Eve:

cd /proc/sys/vm

stop ui
echo 3 > drop_caches
cat /proc/meminfo

You should see something like:

MemTotal:        8059992 kB
MemFree:         7765120 kB
MemAvailable:    7630444 kB
Buffers:            2092 kB
Cached:           107860 kB

Now do this:

start ui
cat /proc/meminfo 

MemTotal:        8059992 kB
MemFree:         6633320 kB
MemAvailable:    6987872 kB
Buffers:           42152 kB
Cached:           882248 kB

echo 3 > drop_caches
cat /proc/meminfo

MemTotal:        8059992 kB
MemFree:         6912356 kB
MemAvailable:    6993704 kB
Buffers:            1136 kB
Cached:           652516 kB

We expect the "Cached" value to go from 900MB --> 100MB.  It only drops to 600MB?  Why?  Is Chrome holding on to something?  marcheu@ started investigation of this here:

https://docs.google.com/document/d/1Tn7zrqoijqBvwO34Z-7TYcVzQnxXO8mr2pEOxVenPlw/edit#heading=h.iucjaj7l8jtc

Comment 1 by gurcheta...@chromium.org, Dec 15 2017 

I examined the memory used by the page cache while the UI was running.  I installed the 
vmtouch tool (https://packages.gentoo.org/packages/dev-util/vmtouch) tool on Falco and wrote
a small wrapper script around it.  The vmtouch tool reports how much of the file
is resident in memory (https://hoytech.com/vmtouch/).  There's another tool called fincore -- they use similar methods [mincore] and get the same results.  Here are the results on Falco [simpler since it does not have ARC++]:

/home/ --> 20M
   /home/.shadow/ --> 20M 
/mnt/ --> 22M
   /home/mnt/home/.shadow/ (bind mount to /home/.shadow, so mostly ignored) --> 20M
/opt/ --> 90M
   /opt/google/chrome --> 90M
      /opt/google/chrome/chrome --> 68M
      /opt/google/chrome/chrome_200_percent.pak --> 8M 
/sys/ --> 512M (ignored)
   /sys/devices/pci0000:00/0000:00:02.0/resource2 --> 256M (ignored)
   /sys/devices/pci0000:00/0000:00:02.0/resource2_wc --> 256M (ignored)
/usr/ --> 32M
   /usr/bin --> 5M 
   /usr/lib64 --> 20M
   /usr/sbin --> 5M 
   /usr/lib64 --> 19M

Discounting the eCryptfs bind mounts (we used ext4 encryption on newer devices anyways) and the PCI files, I can only account for 144M of the cached memory.  However, after echo 3 > /proc/sys/vm/drop_caches on Falco, /proc/meminfo reports about 210M.  

After stopping the ui and doing echo 3 > /proc/sys/vm/drop_caches, there was only 48M of cache usage (even though /proc/sys/vm/min_filelist_kbytes is 100M). 
 I could only account for about 29M with Chrome off:

Examining file bin
  Resident Pages: 214/1443  856K/5M  14.8%
Examining file boot
  Resident Pages: 0/1880  0/7M  0%
Examining file build
  Resident Pages: 0/0  0/0  
Examining file debugd
  Resident Pages: 0/0  0/0  
Examining file dev
  Resident Pages: 5/34  20K/136K  14.7%
Examining file etc
  Resident Pages: 16/1217  64K/4M  1.31%
Examining file home
  Resident Pages: 29/39650  116K/154M  0.0731%
Examining file lib
  Resident Pages: 4/3824  16K/14M  0.105%
Examining file lib64
  Resident Pages: 855/2369  3M/9M  36.1%
Examining file lost+found
  Resident Pages: 0/0  0/0  
Examining file media
  Resident Pages: 0/0  0/0  
Examining file mnt
  Resident Pages: 1001/2329042  3M/8G  0.043%
Examining file opt
  Resident Pages: 83/82299  332K/321M  0.101%
Examining file postinst
  Resident Pages: 0/0  0/0  
Examining file proc
  Resident Pages: 0/0  0/0  
Examining file root
  Resident Pages: 2/2176  8K/8M  0.0919%
Examining file run
  Resident Pages: 137/137  548K/548K  100%
Examining file sbin
  Resident Pages: 208/2062  832K/8M  10.1%
Examining file sys
  Resident Pages: 131233/131233  512M/512M  100%
Examining file tmp
  Resident Pages: 50/50  200K/200K  100%
Examining file usr
  Resident Pages: 5332/470892  20M/1G  1.13%
Examining file var
  Resident Pages: 30/18625  120K/72M  0.161%

So where's the missing 60M when Chrome is on and 20M when Chrome is off coming from?
vmtouch-all.sh
148 bytes View Download

Comment 2 by gurcheta...@chromium.org, Dec 16 2017

Cc: dbehr@chromium.org
I get this result when examining the memory when the UI is stopped, indicating frecon has 17M in the pagecache:


localhost ~ # stop ui
localhost ~ # echo 3 > /proc/sys/vm/drop_caches 
localhost ~ # cat /proc/meminfo 
MemTotal:        3986012 kB
MemFree:         3830080 kB
Buffers:             852 kB
Cached:            46784 kB
SwapCached:            0 kB
Active:            49216 kB
Inactive:          38092 kB
Active(anon):      39844 kB
Inactive(anon):    21916 kB
Active(file):       9372 kB
Inactive(file):    16176 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       5838884 kB
SwapFree:        5838884 kB
Dirty:               624 kB
Writeback:             0 kB
AnonPages:         39892 kB
Mapped:            23936 kB
Shmem:             21916 kB
Slab:              22704 kB
SReclaimable:      10564 kB
SUnreclaim:        12140 kB
KernelStack:        1088 kB
PageTables:         2352 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     7831888 kB
Committed_AS:     235696 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      360920 kB
VmallocChunk:   34359374660 kB
AnonHugePages:         0 kB
DirectMap4k:       36328 kB
DirectMap2M:     3049472 kB
DirectMap1G:     1048576 kB
localhost ~ # ps -ef | grep "frecon"
root       285     1  0 16:10 ?        00:00:00 frecon --daemon --clear 0xfffefefe --dev-mode --enable-osc --enable-vts --pre-create-vts --frame-interval 25 --image-hires /usr/share/chromeos-assets/images_200_percent/boot_splash_frame01.png --image-hires /usr/share/chromeos-assets/images_200_percent/boot_splash_frame02.png --image-hires /usr/share/chromeos-assets/images_200_percent/boot_splash_frame03.png --image-hires /usr/share/chromeos-assets/images_200_percent/boot_splash_frame04.png --image-hires /usr/share/chromeos-assets/images_200_percent/boot_splash_frame05.png --image-hires /usr/share/chromeos-assets/images_200_percent/boot_splash_frame06.png --image-hires /usr/share/chromeos-assets/images_200_percent/boot_splash_frame07.png --image-hires /usr/share/chromeos-assets/images_200_percent/boot_splash_frame08.png --image-hires /usr/share/chromeos-assets/images_200_percent/boot_splash_frame09.png --image-hires /usr/share/chromeos-assets/images_200_percent/boot_splash_frame10.png --image-hires /usr/share/chromeos-assets/images_200_percent/boot_splash_frame11.png --image-hires /usr/share/chromeos-assets/images_200_percent/boot_splash_frame12.png --image-hires /usr/share/chromeos-assets/images_200_percent/boot_splash_frame13.png --image /usr/share/chromeos-assets/images_100_percent/boot_splash_frame01.png --image /usr/share/chromeos-assets/images_100_percent/boot_splash_frame02.png --image /usr/share/chromeos-assets/images_100_percent/boot_splash_frame03.png --image /usr/share/chromeos-assets/images_100_percent/boot_splash_frame04.png --image /usr/share/chromeos-assets/images_100_percent/boot_splash_frame05.png --image /usr/share/chromeos-assets/images_100_percent/boot_splash_frame06.png --image /usr/share/chromeos-assets/images_100_percent/boot_splash_frame07.png --image /usr/share/chromeos-assets/images_100_percent/boot_splash_frame08.png --image /usr/share/chromeos-assets/images_100_percent/boot_splash_frame09.png --image /usr/share/chromeos-assets/images_100_percent/boot_splash_frame10.png --image /usr/share/chromeos-assets/images_100_percent/boot_splash_frame11.png --image /usr/share/chromeos-assets/images_100_percent/boot_splash_frame12.png --image /usr/share/chromeos-assets/images_100_percent/boot_splash_frame13.png
root       286   285  0 16:10 ?        00:00:00 [frecon] <defunct>
root      2930  2663  0 16:11 pts/0    00:00:00 grep --colour=auto frecon
localhost ~ # kill 285
localhost ~ # echo 3 > /proc/sys/vm/drop_caches 
localhost ~ # cat /proc/meminfo 
MemTotal:        3986012 kB
MemFree:         3851964 kB
Buffers:             404 kB
Cached:            29896 kB
SwapCached:            0 kB
Active:            46324 kB
Inactive:          20884 kB
Active(anon):      36968 kB
Inactive(anon):     5404 kB
Active(file):       9356 kB
Inactive(file):    15480 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       5838884 kB
SwapFree:        5838884 kB
Dirty:               676 kB
Writeback:             0 kB
AnonPages:         37092 kB
Mapped:            23652 kB
Shmem:              5404 kB
Slab:              22348 kB
SReclaimable:      10580 kB
SUnreclaim:        11768 kB
KernelStack:        1048 kB
PageTables:         2236 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     7831888 kB
Committed_AS:     215380 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      360920 kB
VmallocChunk:   34359374660 kB
AnonHugePages:         0 kB
DirectMap4k:       36328 kB
DirectMap2M:     3049472 kB
DirectMap1G:     1048576 kB

Comment 3 by gurcheta...@chromium.org, Dec 20 2017 

It looks like the DRM dumb buffers backed by shm are getting counted in the pagecache in the case of Frecon:

2017-12-19T18:55:38.122666-08:00 NOTICE kernel: [   74.567997] dentry_kill -- the path is /drm mm object
2017-12-19T18:55:38.123911-08:00 NOTICE kernel: [   74.568772] before: 9012, after: 8981
2017-12-19T18:55:38.123929-08:00 NOTICE kernel: [   74.568784] Pid: 280, comm: agetty Tainted: G        WC   3.8.11 #35 
2017-12-19T18:55:38.123932-08:00 NOTICE kernel: [   74.568797] Call Trace:
2017-12-19T18:55:38.123933-08:00 NOTICE kernel: [   74.568811]  [<ffffffff9f6d0319>] zone_page_state_add+0xfc/0x132
2017-12-19T18:55:38.123935-08:00 NOTICE kernel: [   74.568829]  [<ffffffff9f6d10fb>] __dec_zone_state+0x4b/0x55
2017-12-19T18:55:38.123937-08:00 NOTICE kernel: [   74.568846]  [<ffffffff9f6d1128>] __dec_zone_page_state+0x23/0x25
2017-12-19T18:55:38.123939-08:00 NOTICE kernel: [   74.568862]  [<ffffffff9f6ba977>] __delete_from_page_cache+0x4a/0xbc
2017-12-19T18:55:38.123941-08:00 NOTICE kernel: [   74.568878]  [<ffffffff9f6baa27>] delete_from_page_cache+0x3e/0x60
2017-12-19T18:55:38.123944-08:00 NOTICE kernel: [   74.568897]  [<ffffffff9f6c47e0>] truncate_inode_page+0x65/0x6c
2017-12-19T18:55:38.123946-08:00 NOTICE kernel: [   74.568917]  [<ffffffff9f6cd46e>] shmem_undo_range+0x1ea/0x522
2017-12-19T18:55:38.123948-08:00 NOTICE kernel: [   74.568935]  [<ffffffff9f6cd7bc>] shmem_truncate_range+0x16/0x33
2017-12-19T18:55:38.123950-08:00 NOTICE kernel: [   74.568953]  [<ffffffff9f6cd846>] shmem_evict_inode+0x6d/0x10c
2017-12-19T18:55:38.123952-08:00 NOTICE kernel: [   74.568970]  [<ffffffff9f70b4e5>] evict+0xac/0x15f
2017-12-19T18:55:38.123954-08:00 NOTICE kernel: [   74.568985]  [<ffffffff9f70babb>] iput+0x12d/0x136
2017-12-19T18:55:38.123956-08:00 NOTICE kernel: [   74.568999]  [<ffffffff9f708127>] dentry_kill+0x1c1/0x238
2017-12-19T18:55:38.123957-08:00 NOTICE kernel: [   74.569015]  [<ffffffff9f708290>] dput+0xf2/0xff
2017-12-19T18:55:38.123959-08:00 NOTICE kernel: [   74.569030]  [<ffffffff9f6f7409>] __fput+0x1c3/0x1d9
2017-12-19T18:55:38.123961-08:00 NOTICE kernel: [   74.569046]  [<ffffffff9f6f742d>] ____fput+0xe/0x10
2017-12-19T18:55:38.123963-08:00 NOTICE kernel: [   74.569062]  [<ffffffff9f64e479>] task_work_run+0x7d/0x93
2017-12-19T18:55:38.123965-08:00 NOTICE kernel: [   74.569078]  [<ffffffff9f636fbc>] do_exit+0x3af/0x8e7
2017-12-19T18:55:38.123967-08:00 NOTICE kernel: [   74.569094]  [<ffffffff9f638230>] do_group_exit+0x42/0xb0
2017-12-19T18:55:38.123968-08:00 NOTICE kernel: [   74.569112]  [<ffffffff9f64530d>] get_signal_to_deliver+0x541/0x560
2017-12-19T18:55:38.123970-08:00 NOTICE kernel: [   74.569128]  [<ffffffff9f6017bd>] do_signal+0x43/0x533
2017-12-19T18:55:38.123972-08:00 NOTICE kernel: [   74.569145]  [<ffffffff9fad0fc0>] ? _raw_spin_unlock+0xe/0x10
2017-12-19T18:55:38.123974-08:00 NOTICE kernel: [   74.569161]  [<ffffffff9f70f1a7>] ? mntput+0x2d/0x2f
2017-12-19T18:55:38.123976-08:00 NOTICE kernel: [   74.569177]  [<ffffffff9f6f7412>] ? __fput+0x1cc/0x1d9
2017-12-19T18:55:38.123978-08:00 NOTICE kernel: [   74.569192]  [<ffffffff9f601cd6>] do_notify_resume+0x29/0x5b
2017-12-19T18:55:38.123980-08:00 NOTICE kernel: [   74.569209]  [<ffffffff9fad1f88>] int_signal+0x12/0x17
2017-12-19T18:55:38.123987-08:00 NOTICE kernel: [   74.569377] dentry_kill -- the path is /drm mm object
2017-12-19T18:55:38.125595-08:00 NOTICE kernel: [   74.570133] before: 8020, after: 7989
2017-12-19T18:55:38.125611-08:00 NOTICE kernel: [   74.570144] Pid: 280, comm: agetty Tainted: G        WC   3.8.11 #35 
2017-12-19T18:55:38.125614-08:00 NOTICE kernel: [   74.570157] Call Trace:
2017-12-19T18:55:38.125616-08:00 NOTICE kernel: [   74.570171]  [<ffffffff9f6d0344>] zone_page_state_add+0x127/0x132
2017-12-19T18:55:38.125618-08:00 NOTICE kernel: [   74.570189]  [<ffffffff9f6d10fb>] __dec_zone_state+0x4b/0x55
2017-12-19T18:55:38.125619-08:00 NOTICE kernel: [   74.570207]  [<ffffffff9f6d1128>] __dec_zone_page_state+0x23/0x25
2017-12-19T18:55:38.125621-08:00 NOTICE kernel: [   74.570222]  [<ffffffff9f6ba977>] __delete_from_page_cache+0x4a/0xbc
2017-12-19T18:55:38.125623-08:00 NOTICE kernel: [   74.570239]  [<ffffffff9f6baa27>] delete_from_page_cache+0x3e/0x60
2017-12-19T18:55:38.125627-08:00 NOTICE kernel: [   74.570257]  [<ffffffff9f6c47e0>] truncate_inode_page+0x65/0x6c
2017-12-19T18:55:38.125628-08:00 NOTICE kernel: [   74.570274]  [<ffffffff9f6cd46e>] shmem_undo_range+0x1ea/0x522
2017-12-19T18:55:38.125631-08:00 NOTICE kernel: [   74.570292]  [<ffffffff9f6cd7bc>] shmem_truncate_range+0x16/0x33
2017-12-19T18:55:38.125633-08:00 NOTICE kernel: [   74.570309]  [<ffffffff9f6cd846>] shmem_evict_inode+0x6d/0x10c
2017-12-19T18:55:38.125635-08:00 NOTICE kernel: [   74.570326]  [<ffffffff9f70b4e5>] evict+0xac/0x15f
2017-12-19T18:55:38.125636-08:00 NOTICE kernel: [   74.570342]  [<ffffffff9f70babb>] iput+0x12d/0x136
2017-12-19T18:55:38.125638-08:00 NOTICE kernel: [   74.570356]  [<ffffffff9f708127>] dentry_kill+0x1c1/0x238
2017-12-19T18:55:38.125640-08:00 NOTICE kernel: [   74.570371]  [<ffffffff9f708290>] dput+0xf2/0xff
2017-12-19T18:55:38.125642-08:00 NOTICE kernel: [   74.570387]  [<ffffffff9f6f7409>] __fput+0x1c3/0x1d9
2017-12-19T18:55:38.125644-08:00 NOTICE kernel: [   74.570403]  [<ffffffff9f6f742d>] ____fput+0xe/0x10
2017-12-19T18:55:38.125646-08:00 NOTICE kernel: [   74.570419]  [<ffffffff9f64e479>] task_work_run+0x7d/0x93
2017-12-19T18:55:38.125648-08:00 NOTICE kernel: [   74.570435]  [<ffffffff9f636fbc>] do_exit+0x3af/0x8e7
2017-12-19T18:55:38.125650-08:00 NOTICE kernel: [   74.570451]  [<ffffffff9f638230>] do_group_exit+0x42/0xb0
2017-12-19T18:55:38.125652-08:00 NOTICE kernel: [   74.570469]  [<ffffffff9f64530d>] get_signal_to_deliver+0x541/0x560
2017-12-19T18:55:38.125654-08:00 NOTICE kernel: [   74.570486]  [<ffffffff9f6017bd>] do_signal+0x43/0x533
2017-12-19T18:55:38.125656-08:00 NOTICE kernel: [   74.570502]  [<ffffffff9fad0fc0>] ? _raw_spin_unlock+0xe/0x10
2017-12-19T18:55:38.125658-08:00 NOTICE kernel: [   74.570518]  [<ffffffff9f70f1a7>] ? mntput+0x2d/0x2f
2017-12-19T18:55:38.125673-08:00 NOTICE kernel: [   74.570533]  [<ffffffff9f6f7412>] ? __fput+0x1cc/0x1d9
2017-12-19T18:55:38.125676-08:00 NOTICE kernel: [   74.570549]  [<ffffffff9f601cd6>] do_notify_resume+0x29/0x5b
2017-12-19T18:55:38.125678-08:00 NOTICE kernel: [   74.570566]  [<ffffffff9fad1f88>] int_signal+0x12/0x17

Comment 4 by gurcheta...@chromium.org, Aug 24

Status: Closed (was: Assigned)
We got nowhere with this investigation.  It happens!
►

Sign in to add a comment