Turn on alerts for system_health.memory_desktop regressions [windows/mac]. |
|||||
Issue description
This is a tracking bug.
While it's easy to flip the switch and get alerts flowing, that would inundate us with high-noise alerts that provide little actionable information. We need to do the following:
1) Incrementally turn on alerts, and reduce sensitivities until we have comparable coverage to android. We want to eventually alert on each tuple {process, source [chrome/os], memory dump provider, user story}.
2) Make sure there's a process in place that guarantees that alerts are acted on in a timely fashion. In the short term, we'll probably set up a private rotation where we examine all alerts ourselves. Eventually, when there's a high signal:noise ratio, we'd like to hand this off to the performance sheriffs.
3) Audit the existing metrics being reported by the waterfall for accuracy, and consistent semantics with the same metrics for android devices.
4) Ensure that existing memory dump providers provide useful information on desktop. Add new ones as necessary.
5) Measure and reduce noise in emitted metrics. [Define noise as ratio of alerts to real regressions]
,
Feb 1 2017
Issue 686208 is not really a blocker for this bug. The system health benchmarks don't use the heap profiler. the heap profiler is a more invasive mode which can be used when doing local debugging/investigations (we are discussing these days how to scale it to real world) but NOT benchmarks. The benchmaks get the data from the general instrumentation we have (i.e. the columns that show when you just open chrome://Tracing and tick memory-infra WITHOUT passing --enable-heap-profiling)
,
Feb 1 2017
Not that (5) will likely require the native heap profiler.
,
Feb 1 2017
*Note
,
Feb 1 2017
Ah, yeah if the noise ends up coming from malloc / OilPan / partition alloc, yes.
,
Feb 1 2017
We already know the noise comes from malloc: https://bugs.chromium.org/p/chromium/issues/detail?id=684721#c3
,
Feb 14 2017
Um looking at the discussion at https://bugs.chromium.org/p/chromium/issues/detail?id=684721#c3 The noise from malloc seems to be only 500KiB on browser process and 300KiB on renderer process. That seems like a fairly low noise. Do we really want to turn on heap profiler in telemetry to reduce this noise? The noise in malloc GPU might be a characteristic of GPU process. From observations GPU process malloc comes from very few allocation sites. I think what we want here is dump provider for GPU process malloc rather than heap profiler in telemetry.
,
Feb 14 2017
I looked at the data at https://bugs.chromium.org/p/chromium/issues/detail?id=684721#c10 in previous comment.
,
Feb 14 2017
> The noise from malloc seems to be only 500KiB on browser process and 300KiB on renderer process. That seems like a fairly low noise. Noise of 300KiB makes it very hard to bisect regressions less than 2*300KiB. > Do we really want to turn on heap profiler in telemetry to reduce this noise? I never suggested that. I suggested that we would want to use the native heap profiler [locally] to figure out why there's so much noise.
,
Mar 30 2017
,
Jan 16
,
Jan 16
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by ajwong@chromium.org
, Feb 1 2017