Excessive memory consumption when sending messages to a worker
Reported by
spaceca...@prezi.com,
Jun 21 2016
|
|||
Issue descriptionUserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 Steps to reproduce the problem: 1. Download the 3 attached files. 2. Select any 'method' by editing 'main.js' 3. Open index.html, it will try to start a worker so needs a local webserver or use '--allow-file-access-from-files' No matter which method from the attached 4 is used for populating a ByteArray used in postMessage() the memory consumption periodically grows to ~2GB as reported by the Activity Monitor. Profiling tools (Timeline, Heap Snapshot) only reports ~1GB. We would like to implement our background job system in terms of a worker using the least amount of dynamic memory allocation as possible to avoid GC stalls. The main thread calls the worker per frame. The application is implemented in C++ and compiled with Emscripten. What is the expected behavior? No excessive memory allocation using method #2 or #3 What went wrong? Initial idea was to use method #0 but as reported in issue 169705 , this is not possible, so we wanted to use a preallocated buffer (method #2) as a target buffer for message serialization but this also shows excessive memory consumption. Even tried copying byte-by-byte in method #3 to rule out the a possible optimization by sharing the underlying buffer of TypedArray.set(). Did this work before? N/A Chrome version: 51.0.2704.103 Channel: stable OS Version: OS X 10.9.5 Flash Version: Shockwave Flash 22.0 r0
,
Jun 22 2016
This test case is allocating C-heap memory with each postMessage call, because the typed array in the message is being copied. This memory is only reclaimed when garbage collection occurs on the worker's JavaScript context. The correct way to implement a producer-consumer queue with a web worker that uses a bounded amount of memory is to transfer an ArrayBuffer from the main thread to the worker (using the postMessage(message, [transferables]) form -- see https://developer.mozilla.org/en-US/docs/Web/API/Worker/postMessage ), and transfer it back from the worker to the main thread for refilling using another postMessage call. A double-buffering technique can be used so that the main thread always has a buffer ready to fill. I'm closing this as WontFix. The primitives exist in the platform to avoid unbounded memory consumption in cases like this.
,
Jun 22 2016
Sure, the mentioned technique would be the best way, and that's what I wanted to use originally, but IE doesn't have Transferable and if possible we wanted to avoid platform specifics in our architecture if possible. Also Firefox doesn't show this behaviour. I expected the C-heap memory to be deallocated much earlier as it's not referenced in the worker after the onmessage callback finishes, so it's not purely unbounded memory consumption. How come that the memory grows so much and no collection runs?
,
Jun 22 2016
We're currently reworking the way c memory is handled, see issue 621829 But in general, I agree with Ken: there's an API for that
,
Jun 22 2016
Yeah I found this issue, will keep an eye on it, thanks. In theory I do agree as well, in practice it's not just black and white; as said we have to support every major browser and they behave differently (as expected?). We will implement specific paths for using Transferables, but hope this issue will be fixed in the future with the mentioned rework.
,
Jun 22 2016
|
|||
►
Sign in to add a comment |
|||
Comment 1 by spaceca...@prezi.com
, Jun 21 201638.8 KB
38.8 KB View Download
4.2 KB
4.2 KB View Download