2 I don't recommend since they've been tried in the past and did not work.

3 lgtm.

4 is probably not possible due to all the things the audio service does in addition to playing audio. I.e. that audio might be looped back through WebRTC for echo cancellation, or diverted to some other output. There's a whole virtual layer of things happening you'd preclude. There's also permissions to consider. I.e. browser process is the arbitor of what devices can actually be connected to. If you're allowing renderer processes to arbitrary start audio input streams, you've got a privacy problem.

To elaborate on 2, we got regular reports of glitching on all platforms prior to the select() based approach currently in use.

> 4 is probably not possible due to all the things the audio service does in addition to playing audio. I.e. that audio might be looped back through WebRTC for echo cancellation, or diverted to some other output. There's a whole virtual layer of things happening you'd preclude.

Where is this code located? As far as I can tell AudioOutputDevice calls AudioOutputController::OnMoreData(), which then calls AudioSyncReader::RequestMoreData(), which reads data directly from the renderer. I don't see any processing in these layers. In the renderer process AudioDeviceThread is responsible for sending audio stream data to the browser. The change I'm proposing is to update AudioDeviceThread and AudioOutputDeviceThreadCallback to talk directly to system audio APIs on Fuchsia.

If I understand it correctly the current implementation of the echo canceler in WebRTC uses loopback from the same renderer process, I'm not proposing to change any of that code.

It's a good point about limiting access to the audio APIs, but this is solvable. We don't want to give renderer access to fuchsia::media::Audio, but we can open media::AudioOut in the browser and then send it to the renderer, which means that the renderer will be able to output only to that one stream.

Can provide more tomorrow. +Olka But there's way more, we would not want fuschia to diverge like you're thinking. It'd be a nightmare to maintain. See services/audio

Please note that media::AudioOutputController will be deprecated and deleted in favor of audio::OutputController when we fully launch the audio service.

4 - Will the audio server be running in the renderer process? Or what exactly does "send directly" mean and what mechanisms will be used? Do you mean to replace current "pull" audio data transfer by "push" or by asynchronous polls? In this case, who will be driving the clock? Could you point me to AudioOut code?


>>If I understand it correctly the current implementation of the echo canceler in WebRTC uses loopback from the same renderer process, I'm not proposing to change any of that code.

At least on desktop, WebRTC audio processing will be moved to the audio service.

The audio service takes care of tab audio capture (used by Cast) and muting, and the plan is to introduce cross-tab echo cancellation (cancelling all the audio playback from the microphone input).

1 - What are platform limitations in terms of the number of threads and memory? And what are expected usage patterns for audio?

> 4 - Will the audio server be running in the renderer process?

No. By "audio server" I meant the fuchsia system service that mixes audio streams and sends them to appropriate audio devices. It always runs in a separate process.

> Or what exactly does "send directly" mean and what mechanisms will be used? 

FIDL, Fuchsia's IPC mechanism

> Do you mean to replace current "pull" audio data transfer by "push" or by asynchronous polls? In this case, who will be driving the clock? Could you point me to AudioOut code?

Here is AudioOut interface definition: https://fuchsia.googlesource.com/garnet/+/master/public/fidl/fuchsia.media/audio_out.fidl
Current version uses push model.

> At least on desktop, WebRTC audio processing will be moved to the audio service.

> The audio service takes care of tab audio capture (used by Cast) and muting, and the plan is to introduce cross-tab echo cancellation (cancelling all the audio playback from the microphone input).

Do I understand it correctly that this wouldn't cancel audio streams that come from other applications? (which would require loopback from the system audio server).

Will audio service talk directly to the system audio output API or will it still need IPC to the browser process?


> 1 - What are platform limitations in terms of the number of threads and memory? And what are expected usage patterns for audio?

This is Fuchsia-specific (sorry if it wasn't clear from the bug).

Thanks for the link!

> Do I understand it correctly that this wouldn't cancel audio streams that come from other applications? (which would require loopback from the system audio server).

There are discussions regarding cancelling all system audio as well: it would be very useful for video conferencing. But it does not have to happen in Chrome if the platform provides an option to clean up input streams itself. For example, Mac has a native echo canceller which supports system-wide playback cancellation.

> Will audio service talk directly to the system audio output API or will it still need IPC to the browser process?

Stream setup happens like that:
Renderer -> Browser: request a stream.
Browser: check permissions.
Browser -> Service: request a stream.
Service -> Platform: setup a stream.
Service -> Browser: return the stream.
Browser -> Renderer: return the stream.

After that stream control and data flow are Renderer -> Service -> Platform, bypassing the browser.

>> 1 - What are platform limitations in terms of the number of threads and memory? And what are expected usage patterns for audio?

>This is Fuchsia-specific (sorry if it wasn't clear from the bug).

Yes, exactly - I'm asking about Fuchsia specifics to understand why a single thread for all AudioOutputStreamFuchsia is chosen.

> After that stream control and data flow are Renderer -> Service -> Platform, bypassing the browser.

Thanks for the info! Will the Service mix all output streams before passing them to the platform? 


> Yes, exactly - I'm asking about Fuchsia specifics to understand why a single thread for all AudioOutputStreamFuchsia is chosen.

Because it's simpler. Also because I made incorrect assumption that OnMoreData() doesn't block.

> Will the Service mix all output streams before passing them to the platform?
No plans for this right now (thought it was discussed to try as an experiment). Streams may (and often do) have different latencies and sample rates, and the platform usually does mixing itself (and may be doing it more energy-efficient).

-
3 - We've had a similar problem on Mac where all the streams for a given audio device are polled by the platform on the same thread. That's why the timeout there is half the buffer size duration instead of 20 ms.