This actually causes a lot of data to be uploaded to dashboard. Something around ~150 metrics per page since it uploads memory metrics for all processes including sum, count, std, pct for each of the metrics. 

Hector, can you help us figure out why the memory dumps are not being associated with the browser process?

Sample trace: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/browse_media_youtube_2017-10-01_09-40-44_81228.html

In the devtools console, the browser process's memoryDumps is empty, so splitGlobalDumpsByBrowserName allocates the one global memory dump to 'unknown_browser':
document.body.querySelector('tr-ui-timeline-view').model.getOrCreateHelper(tr.model.helpers.ChromeModelHelper).browserHelpers[0].process.memoryDumps
document.body.querySelector('tr-ui-timeline-view').model.globalMemoryDumps

We don't hit this line [1] for the browser process because we don't have any data in the trace for the browser process, not sure why this is yet will look more tomorrow. 


[1]: https://github.com/catapult-project/catapult/blob/master/tracing/tracing/extras/importer/trace_event_importer.html#L2682

tl;dr: A new MDM deadlock consistently reproduces with v8.browsing_desktop

So the trace is malformed, specifically it doesn't contain the 'periodic_interval' event for the browser process which actually injects the memory dump data into the trace.

For the browser process we see this event:
https://cs.chromium.org/chromium/src/base/trace_event/memory_dump_manager.cc?sq=package:chromium&dr=CSs&l=529

But not this one:
https://cs.chromium.org/chromium/src/base/trace_event/memory_dump_manager.cc?sq=package:chromium&dr=CSs&l=794

We do see the MemoryDumpManager::InvokeOnMemoryDump event for some MemoryDumpProviders.

Together this suggests that at some point the browser MemoryDumpManager deadlocks - a problem we have encountered in the past.

From these graphs: https://chromeperf.appspot.com/report?sid=00376f9d952b9c77f4447d1e6b4b5d0deb3513d46793c5961344cf41edc1b452 we can see this first happened on 2017-09-07 and occuring intermittently until 2017-09-24 (the last observed non-deadlocking data point) at which point it started happening consistantly.

There are two special things about this configuration:
1) we take light dumps
2) disabled-by-default-memory-infra.v8.code_stats is turned on

Given this does not occur for the system_health.memory_mobile benchmarks I suspect one of these is involved.
 

Exposing of this deadlock could have something to do with https://chromium-review.googlesource.com/c/chromium/src/+/703234 maybe?

We do know that there exists a data race in the enabling of MDM (see the bug for that CL) and we landed a temporary fix for stable branch point (i.e. the CL itself). The proper fix is described in that bug.

I don't know if it matters but when "disabled-by-default-memory-infra.v8.code_stats" is enabled, we collect code statistics from V8. Collecting code statistics involves scanning V8 heap. So this is considerably more expensive than collecting other stats from V8. 

Sorry took me awhile to get back to this.

Re: #7 hmm, I think in this case the problem is in the browser process so I don't think it could be the same bug.
Re: #8 Yeah could be, I think we have much coverage of memory-infra.v8.code_stats on the memory-infra side so it could well be hitting some odd path.

While looking into this again I noticed the problems seem to be confined to the ref stories (browse_media_youtube_ref etc)?

On the non-reference build it seems to be working correctly: https://chromeperf.appspot.com/report?sid=4ac237bd0a722e208d5eea3a86760e2d669d8810399d571b8da36ebea59ca8b8

If that is the case then almost certainly this is an old bug that (possibly accidently) got fixed at some point.

We could hack around this in tracing/dashboard but it seems neater to update the reference build, I'll try and find the bug for that and block this on it.

If anybody sees this on a non-ref build let me know and I'll try to repo.