New issue
Advanced search Search tips

Issue 774564 link

Starred by 5 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2017
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocked on:
issue 782696

Blocking:
issue 469639



Sign in to add a comment

Implement MessagePort in AudioWorklet

Project Member Reported by hongchan@chromium.org, Oct 13 2017

Issue description

Comment 1 by steve...@gmail.com, Nov 1 2017

I was interested to get our WebAssembly build of Csound working as an AudioWorklet and found the lack of MessagePort support blocked the design I wanted to explore.  I will be happy to test MessagePort once it is implemented.
> our WebAssembly build of Csound

Do you have any link for it? I would love to take a look.

Comment 3 by steve...@gmail.com, Nov 1 2017

Our last stable build was at: 

https://github.com/csound/csound/releases/download/6.09.1/csound-wasm-6.09.1.zip

I use a dev build for http://live.csound.com, which has source code and pre-compiled WASM build at:

https://github.com/kunstmusik/csound-live-code

Currently, we use WASM with ScriptProcessorNode. I was doing some experiments with Chrome Canary (Version 64.0.3255.0 (Official Build) canary (64-bit)) and trying to setup an AudioWorklet for Csound at:

https://github.com/kunstmusik/csound-audioworklet

(The code here is plainly wrong at the moment; I was having trouble in figuring out what is the best way to load the WASM binary data to eventually have it instantiated within the AudioWorklet, and I did not manage to pass the binary data either as message via port or through constructor.)

I'm very interested--as is, I believe, the rest of the Csound community--to get an AudioWorklet of Csound working that can support our live coding and other use cases that we currently support with the ScriptProcessorNode/WASM build we provide. 
Very exciting to see this.

I have been working closely with Faust team for their transition to AudioWorklet, and they did not seem to have the problem you have. See here: http://faust.grame.fr/modules-worklet/

If loading WASM binary is one time thing (upon construction), you might want to try o = createObjectURL(string) and pass it with worklet.addModule(o). If you need bi-directional communication for the interaction after the instantiation, then you have to wait for MessagePort to land in Chrome Canary.

Anyhow, CSound with AudioWorklet looks really interesting and we definitely want to support the use case!

Comment 5 by steve...@gmail.com, Nov 1 2017

I had been studying the Faust examples prior to this and it looked to me they translate the binary WASM file into JS code. For example, in:

http://faust.grame.fr/modules-worklet/RandomFlute-processor.js

there is a getBinaryCodeRandomFlute() function that returns a UInt8Array with the WASM contents encoded in hex. Near the end of the file, the content is loaded with the call to WebAssembly.instantiate(). 

I had tried to work out something similarly by loading the WASM binary using Fetch API in the main thread, then passing to the worklet:

https://github.com/kunstmusik/csound-audioworklet/blob/master/index.html#L14-L31

But passing as an option via constructor did not work (I wasn't sure if that was supported to add constructor arguments when instantiating Processor via AudioWorkletNode?).  

I'm not quite sure I understand though how one would use addModule() with the WASM binary data. Could you elaborate? 

Thanks very much for your assistance and interest! 
|options| for AudioWorkletNode does not handle "data transfer" from the node to the processor. It's for the initialization of AudioParam values and the channel management.

Also I believe it's worth noting that the design of AWN/AWP/MP was not perfectly tailored for the WASM and SAB.

If you have to transfer WASM binary data directly to the processor, you still have to use MessagePort. I was merely suggesting to use createObjectURL() and addModule() as a workaround until MP is available. Think of addModule() is a way of passing data to the AudioWorkletGlobalScope since you can call it as many times as you want.

FWIW, I am working on it at this very moment:
https://chromium-review.googlesource.com/c/chromium/src/+/747621

Comment 7 by steve...@gmail.com, Nov 1 2017

I suspected my understanding of |options| was incorrect, but I wasn't sure; thanks for confirming this. :)  

As for addModule(), I'm looking at the WebAudio Spec and do not see addModule() described for passing data. (I did not see it in CSS Houdini Worklet addModule() link either.) 

At this point though, it looks like MessagePort is on its way and may be the better solution. I'm happy to wait to test that in Canary when it lands. 


addModule() is just a way of passing JS code and get it evaluated in the WorkletGlobalScope. So you can utilize it however you want like what a Faust example does.

Some folks already started to some experiments on addModule() along with createObjectURL() and I was simply pointing out possibilities/workarounds.

Comment 9 by steve...@gmail.com, Nov 1 2017

I'd love to try this out, but as I mentioned, I don't see any documentation explaining how to do this.  I also do not see any examples of this. (As mentioned in Comment 5, Faust embeds the binary content into the JS file as JS; I see no example where they use createObjectURL()). 

If you could point me to somewhere I could look to see how addModule() could be used together with createObjectURL(), I'd be happy to experiment.  Otherwise, I don't understand how passing the binary WASM data via addModule is supposed to work. 
Just to note, I hope my comments don't come off as critical; I'm grateful for the advice/suggestions and the work you're doing on this, I just don't understand how to put the addModule/createObjectURL stuff together. Thanks!
I was directed to the WebAudio Slack and read through archives to understand addModule/createObjectURL path.  I'll give it a try to see if I can get something working until the MessagePort lands, thanks!
I'll try to add an example of addModule/createObjectURL in the AudioWorklet demo collection. Glad you found the thread in the slack channel!
I've tried a few different strategies to try to get the binary data over to the AudioWorklet but it's been a struggle. I can create an object URL for the binary wasm blob, but it seems there's no way to load the data from the worklet as there appears to be no Fetch or XMLHttpRequest to grab data from URL.  I've tried doing text encoding and transferring that way but it's not a good solution as it looks to stress the JS runtime too much.  I'm investigating just pre-encoding the data to JS file, similar to what the Faust examples do.  

(Side question: is async fetching from URLs supposed to be supported in AudioWorklet, or is that intentionally left out?  Maybe it's a moot point though as once MessagePort is in, all of the paths probably won't be necessary.)
https://fetch.spec.whatwg.org/#headers-class

According to the spec, it is not exposed in Worklet. So I raised an issue:

https://github.com/WebAudio/web-audio-api/issues/1439
I've subscribed to the Github issue, thanks!  

One other note: I tried doing multiple window.audioWorklet.addModule() calls, using code like:

    window.audioWorklet.addModule('libcsound.wasm.as.js').then(() => 
      window.audioWorklet.addModule('CsoundProcessor.js')).then(() => {
        let context = new AudioContext();
        let csNode = new AudioWorkletNode(context, "Csound");

        csNode.connect(context.destination);

        console.log("TESTING");
      });

Where the first .js file had code like:

let libcsoundWasm = new Uint8Array([......]);

and the second .js file had code that tried to reference libcsoundWasm.  I found the JS in the second file would give "Uncaught ReferenceError: libcsoundWasm is not defined".  Is it the case that a variable defined in one addModule call should be available from code in a latter addVariable call?


That's interesting. If you have a full example of that I suggest you to file a bug against Worker/Worklet and cc me (hongchan@).
Blockedon: 782696
Labels: M-64
Project Member

Comment 19 by bugdroid1@chromium.org, Nov 17 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/40f38e8327da8200478f33e060a56a7f192ad93f

commit 40f38e8327da8200478f33e060a56a7f192ad93f
Author: Hongchan Choi <hongchan@chromium.org>
Date: Fri Nov 17 19:43:47 2017

Implement MessagePort on AudioWorkletNode/AudioWorkletProcessor

NOTE: This CL will land after https://chromium-review.googlesource.com/c/chromium/src/+/769330.

This is to expose MessagePort object in AudioWorkletNode and
AudioWorkletProcessor for the bi-directional communication.

Spec: https://webaudio.github.io/web-audio-api/#instantiation-of-AudioWorkletNode-and-AudioWorkletProcessor

Bug:  774564 
Change-Id: I543735fc141460a8c356b52e2d461a82d75422fb
Reviewed-on: https://chromium-review.googlesource.com/747621
Commit-Queue: Hongchan Choi <hongchan@chromium.org>
Reviewed-by: Kentaro Hara <haraken@chromium.org>
Reviewed-by: Hiroki Nakagawa <nhiroki@chromium.org>
Cr-Commit-Position: refs/heads/master@{#517517}
[add] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/LayoutTests/http/tests/webaudio/audio-worklet/message-port.html
[add] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/LayoutTests/http/tests/webaudio/audio-worklet/port-processor.js
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/core/dom/MessageChannel.h
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/core/dom/MessagePort.idl
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/core/dom/events/Event.idl
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/core/dom/events/EventTarget.idl
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/core/events/MessageEvent.idl
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/core/workers/WorkletGlobalScope.h
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletGlobalScope.cpp
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletGlobalScope.h
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletGlobalScopeTest.cpp
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletMessagingProxy.cpp
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletMessagingProxy.h
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletNode.cpp
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletNode.h
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletNode.idl
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletProcessor.cpp
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletProcessor.h
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletProcessor.idl
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletProcessorDefinition.cpp
[modify] https://crrev.com/40f38e8327da8200478f33e060a56a7f192ad93f/third_party/WebKit/Source/modules/webaudio/AudioWorkletProcessorDefinition.h

Status: Fixed (was: Started)

Sign in to add a comment