New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 623499 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Dec 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 2
Type: Bug

Blocking:
issue 542584



Sign in to add a comment

[memory-infra] Get OS memory information on Windows

Project Member Reported by petrcermak@chromium.org, Jun 27 2016

Issue description

Memory-Infra on Windows doesn't use memory usage reported by the OS (private memory, shared memory, etc.) yet. As a result, the corresponding charts are flat: https://chromeperf.appspot.com/report?sid=cd105b396af1f05e3153586a5241ba4b7505bf3b2aa44dcd017ef7a364e27722.

Starting point: https://cs.chromium.org/chromium/src/components/tracing/common/process_metrics_memory_dump_provider.cc?l=252
 
Catching up very slowly. Bruce, here's the deal: in memory-infra we have two categories of probes:

1. Probes that come from codebase instrumentation (v8 heaps, partition alloc stats, skia stats, etc etc). We call this "memory reported by Chrome" in chromeperf
These work on all OS right now, including Windows. 
PRO: they are pretty stable (non-noisy) metrics and allow us to catch or investigate bugs.
CON: as they come from manual instrumentations, they don't give us full coverage of the codebase (we might miss instrumentation). 
As a partial mitigation we have the "malloc" catch-all bucket ("winheap" in windows) which capture all the remaining heap stuff.
Still, that is not a fully inclusive metric as one can use memory without using the heap (at least on Linux/Android)

2. What we call this "memory reported by OS". Think to this as a sort of "control group" which gives us feedback on the coverage we get by 1.
Today we are missing this in Windows. This bug is about this.

On Linux/Android specifically we track:
- PSS (Proportional Set Size) which is the sum of (private pages) + (shared_pages / number of sharers). That accounts any kind of memory (anonymous and not, dirty and not) and is one of our "grand totals" to have a overall summary. Has the nice property that can be added sanely cross processes.
- Private dirty: just the dirty pages (mostly == to anonymous memory + file mmaped and written)
- Total resident [not a big deal]: similar to PSS, without the "/number of sharers part". The only reason being of this is that the kernel gives us also the peak value, which is useful to spot where critical situations happen.
- Swapped [not a big deal]: the amount of memory moved which is not resident anymore and moved to the swap file.

The question now is: what are the closest translations of these for Windows? How should we track them (code-wise)?
From what I remember from your presentation (Life of a Windows memory page) there isn't a 1:1 mapping of those in windows land, because the memory model is not identical.
It is fine. What we want to ultimately achieve here is having one (or a bunch) of known total counters that Windows folks are familiar with and that we are confident catches most of the memory usage.

Comment 2 by ssid@chromium.org, Jul 20 2016

Blocking: 542584
Blocking: 632021
* PSS is not a commonly used Windows concept, but I have written code to calculate it before. The sharing count is not perfect - the maximum count for number-of-sharers is seven - but in practice it works quite well.
* Private dirty is something I'm not sure how to get on Windows, but I will investigate.
* Total resident is easy to get
* Swapped is a value that I would love to get but I have never found a way to do it.

So, PSS and Total resident are easy and should be identical or very similar in meaning to Linux.
Private dirty and swapped may not be possible.

It should be fairly straightforward to fill in the base::trace_event::ProcessMemoryMaps structure, as long as the interpretations of the fields are well documented.

Cc: brucedaw...@chromium.org
Owner: stanisc@chromium.org
I'll take this. 
Thanks for the thorough explanation Bruce. Yeah I remember that 7 (it's the ShareCount of PSAPI_WORKING_SET_EX_BLOCK, right?) , at some point I was looking at attempting the PSS calc on windows but eventually never got time to finish it. But looks like you have that code already :)

Thanks a lot for the help stanisc@. The impl for Linux/Android is in components/tracing/common/process_metrics_memory_dump_provider.cc, happy to give help for the plumbing / reviews any memory-infra related question.

Cc: -petrcermak@chromium.org
Cc: -petrcermak@chromium.org
Owner: chengx@chromium.org
 Issue 530236  has been merged into this issue.
Blocking: -632021
Status: Started (was: Assigned)
Project Member

Comment 13 by bugdroid1@chromium.org, Dec 12 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b172892773fe15a10508f733ca0bc831240a7ec6

commit b172892773fe15a10508f733ca0bc831240a7ec6
Author: chengx <chengx@chromium.org>
Date: Mon Dec 12 06:55:56 2016

Add function to compute proportional set size for OS_WIN

Memory-Infra on Windows doesn't use memory usage reported by the OS (proportional set size, private dirty, etc.) yet.

This CL added functionality to calculate proportional set size for a process.

BUG= 623499 

Review-Url: https://codereview.chromium.org/2549803003
Cr-Commit-Position: refs/heads/master@{#437827}

[modify] https://crrev.com/b172892773fe15a10508f733ca0bc831240a7ec6/base/process/process_metrics.h
[modify] https://crrev.com/b172892773fe15a10508f733ca0bc831240a7ec6/base/process/process_metrics_win.cc
[modify] https://crrev.com/b172892773fe15a10508f733ca0bc831240a7ec6/components/tracing/common/process_metrics_memory_dump_provider.cc

Status: Fixed (was: Started)
The CL as stated in comment #12 calculates Proportional Set Size (PSS) for a process, rather than generating full mmaps. It is enough as far as the bots and chromeperf are concerned.

The extra value of having the full mmaps is when developers pull the trace from the bots to investigate a regression and they happen to have more details. This can be a possible follow up for the future.

I have checked the Chrome Performance Dashboard. The PSS numbers begin to show up and look reasonable. So I will mark this bug as fixed for now.
Components: Internals>Instrumentation>Memory

Sign in to add a comment