Implement MessagePort in AudioWorklet |
||||
Issue description
,
Nov 1 2017
> our WebAssembly build of Csound Do you have any link for it? I would love to take a look.
,
Nov 1 2017
Our last stable build was at: https://github.com/csound/csound/releases/download/6.09.1/csound-wasm-6.09.1.zip I use a dev build for http://live.csound.com, which has source code and pre-compiled WASM build at: https://github.com/kunstmusik/csound-live-code Currently, we use WASM with ScriptProcessorNode. I was doing some experiments with Chrome Canary (Version 64.0.3255.0 (Official Build) canary (64-bit)) and trying to setup an AudioWorklet for Csound at: https://github.com/kunstmusik/csound-audioworklet (The code here is plainly wrong at the moment; I was having trouble in figuring out what is the best way to load the WASM binary data to eventually have it instantiated within the AudioWorklet, and I did not manage to pass the binary data either as message via port or through constructor.) I'm very interested--as is, I believe, the rest of the Csound community--to get an AudioWorklet of Csound working that can support our live coding and other use cases that we currently support with the ScriptProcessorNode/WASM build we provide.
,
Nov 1 2017
Very exciting to see this. I have been working closely with Faust team for their transition to AudioWorklet, and they did not seem to have the problem you have. See here: http://faust.grame.fr/modules-worklet/ If loading WASM binary is one time thing (upon construction), you might want to try o = createObjectURL(string) and pass it with worklet.addModule(o). If you need bi-directional communication for the interaction after the instantiation, then you have to wait for MessagePort to land in Chrome Canary. Anyhow, CSound with AudioWorklet looks really interesting and we definitely want to support the use case!
,
Nov 1 2017
I had been studying the Faust examples prior to this and it looked to me they translate the binary WASM file into JS code. For example, in: http://faust.grame.fr/modules-worklet/RandomFlute-processor.js there is a getBinaryCodeRandomFlute() function that returns a UInt8Array with the WASM contents encoded in hex. Near the end of the file, the content is loaded with the call to WebAssembly.instantiate(). I had tried to work out something similarly by loading the WASM binary using Fetch API in the main thread, then passing to the worklet: https://github.com/kunstmusik/csound-audioworklet/blob/master/index.html#L14-L31 But passing as an option via constructor did not work (I wasn't sure if that was supported to add constructor arguments when instantiating Processor via AudioWorkletNode?). I'm not quite sure I understand though how one would use addModule() with the WASM binary data. Could you elaborate? Thanks very much for your assistance and interest!
,
Nov 1 2017
|options| for AudioWorkletNode does not handle "data transfer" from the node to the processor. It's for the initialization of AudioParam values and the channel management. Also I believe it's worth noting that the design of AWN/AWP/MP was not perfectly tailored for the WASM and SAB. If you have to transfer WASM binary data directly to the processor, you still have to use MessagePort. I was merely suggesting to use createObjectURL() and addModule() as a workaround until MP is available. Think of addModule() is a way of passing data to the AudioWorkletGlobalScope since you can call it as many times as you want. FWIW, I am working on it at this very moment: https://chromium-review.googlesource.com/c/chromium/src/+/747621
,
Nov 1 2017
I suspected my understanding of |options| was incorrect, but I wasn't sure; thanks for confirming this. :) As for addModule(), I'm looking at the WebAudio Spec and do not see addModule() described for passing data. (I did not see it in CSS Houdini Worklet addModule() link either.) At this point though, it looks like MessagePort is on its way and may be the better solution. I'm happy to wait to test that in Canary when it lands.
,
Nov 1 2017
addModule() is just a way of passing JS code and get it evaluated in the WorkletGlobalScope. So you can utilize it however you want like what a Faust example does. Some folks already started to some experiments on addModule() along with createObjectURL() and I was simply pointing out possibilities/workarounds.
,
Nov 1 2017
I'd love to try this out, but as I mentioned, I don't see any documentation explaining how to do this. I also do not see any examples of this. (As mentioned in Comment 5, Faust embeds the binary content into the JS file as JS; I see no example where they use createObjectURL()). If you could point me to somewhere I could look to see how addModule() could be used together with createObjectURL(), I'd be happy to experiment. Otherwise, I don't understand how passing the binary WASM data via addModule is supposed to work.
,
Nov 1 2017
Just to note, I hope my comments don't come off as critical; I'm grateful for the advice/suggestions and the work you're doing on this, I just don't understand how to put the addModule/createObjectURL stuff together. Thanks!
,
Nov 2 2017
I was directed to the WebAudio Slack and read through archives to understand addModule/createObjectURL path. I'll give it a try to see if I can get something working until the MessagePort lands, thanks!
,
Nov 2 2017
I'll try to add an example of addModule/createObjectURL in the AudioWorklet demo collection. Glad you found the thread in the slack channel!
,
Nov 2 2017
I've tried a few different strategies to try to get the binary data over to the AudioWorklet but it's been a struggle. I can create an object URL for the binary wasm blob, but it seems there's no way to load the data from the worklet as there appears to be no Fetch or XMLHttpRequest to grab data from URL. I've tried doing text encoding and transferring that way but it's not a good solution as it looks to stress the JS runtime too much. I'm investigating just pre-encoding the data to JS file, similar to what the Faust examples do. (Side question: is async fetching from URLs supposed to be supported in AudioWorklet, or is that intentionally left out? Maybe it's a moot point though as once MessagePort is in, all of the paths probably won't be necessary.)
,
Nov 2 2017
https://fetch.spec.whatwg.org/#headers-class According to the spec, it is not exposed in Worklet. So I raised an issue: https://github.com/WebAudio/web-audio-api/issues/1439
,
Nov 2 2017
I've subscribed to the Github issue, thanks!
One other note: I tried doing multiple window.audioWorklet.addModule() calls, using code like:
window.audioWorklet.addModule('libcsound.wasm.as.js').then(() =>
window.audioWorklet.addModule('CsoundProcessor.js')).then(() => {
let context = new AudioContext();
let csNode = new AudioWorkletNode(context, "Csound");
csNode.connect(context.destination);
console.log("TESTING");
});
Where the first .js file had code like:
let libcsoundWasm = new Uint8Array([......]);
and the second .js file had code that tried to reference libcsoundWasm. I found the JS in the second file would give "Uncaught ReferenceError: libcsoundWasm is not defined". Is it the case that a variable defined in one addModule call should be available from code in a latter addVariable call?
,
Nov 2 2017
That's interesting. If you have a full example of that I suggest you to file a bug against Worker/Worklet and cc me (hongchan@).
,
Nov 8 2017
,
Nov 8 2017
,
Nov 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/40f38e8327da8200478f33e060a56a7f192ad93f commit 40f38e8327da8200478f33e060a56a7f192ad93f Author: Hongchan Choi <hongchan@chromium.org> Date: Fri Nov 17 19:43:47 2017 Implement MessagePort on AudioWorkletNode/AudioWorkletProcessor NOTE: This CL will land after https://chromium-review.googlesource.com/c/chromium/src/+/769330. This is to expose MessagePort object in AudioWorkletNode and AudioWorkletProcessor for the bi-directional communication. Spec: https://webaudio.github.io/web-audio-api/#instantiation-of-AudioWorkletNode-and-AudioWorkletProcessor Bug: 774564 Change-Id: I543735fc141460a8c356b52e2d461a82d75422fb Reviewed-on: https://chromium-review.googlesource.com/747621 Commit-Queue: Hongchan Choi <hongchan@chromium.org> Reviewed-by: Kentaro Hara <haraken@chromium.org> Reviewed-by: Hiroki Nakagawa <nhiroki@chromium.org> Cr-Commit-Position: refs/heads/master@{#517517} [add] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/LayoutTests/http/tests/webaudio/audio-worklet/message-port.html [add] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/LayoutTests/http/tests/webaudio/audio-worklet/port-processor.js [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/core/dom/MessageChannel.h [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/core/dom/MessagePort.idl [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/core/dom/events/Event.idl [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/core/dom/events/EventTarget.idl [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/core/events/MessageEvent.idl [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/core/workers/WorkletGlobalScope.h [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletGlobalScope.cpp [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletGlobalScope.h [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletGlobalScopeTest.cpp [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletMessagingProxy.cpp [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletMessagingProxy.h [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletNode.cpp [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletNode.h [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletNode.idl [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletProcessor.cpp [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletProcessor.h [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletProcessor.idl [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletProcessorDefinition.cpp [modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletProcessorDefinition.h
,
Nov 17 2017
|
||||
►
Sign in to add a comment |
||||
Comment 1 by steve...@gmail.com
, Nov 1 2017