=>WebAudio, but +some webrtc folk who looked into similar issues in the past.

I believe this has something to do with ScriptProcessorNode. The node is supposed to be deprecated.

https://cs.chromium.org/chromium/src/third_party/WebKit/Source/modules/webaudio/ScriptProcessorNode.cpp?sq=package:chromium&l=176

I think m_doubleBufferIndex is somehow not changing fast enough. So you get to hear the same buffer twice, which explains the stutters.

This issue is forcing us to tell our users to use Firefox or IE11 - audio recording is unusable for our users if the main thread has a bit of load (as low as 10% CPU).

Any idea when this will be triaged (and ideally fixed ... ;-)?

Please try this WebRTC demo

https://webrtc.github.io/samples/src/content/getusermedia/audio/

on the same machine as you have issues (use a headset to avoid echo). Does it sound bad as well?

* The issue does not occur with the original https://webrtc.github.io/samples/src/content/getusermedia/audio/ sample
* The issue also does not occur with the modified sample (attached) that generates load on the main javascript thread.

Note that this sample does *not* record audio into a javascript buffer.

The ChromeStutteryAudioRepro.zip *does* record audio bytes into javascript arrays.  Somewhere on this data flow path apparently the audio blocks get duplicated.

Deleted: ChromeStutteryAudioRepro_samples-gh-pages.zip 57.3 KB

ChromeStutteryAudioRepro_samples-gh-pages.zip 57.3 KB Download

Created a repro based on  https://webrtc.github.io/samples/src/content/getusermedia/audio/ (see attached).

This adds a script processor node and puts load on the main javascript thread.  Effect is duplicate audio blocks.

I understand script processor node is deprecated - by when will audio workers be available?

Deleted: ChromeStutteryAudioRepro_webrtcsamples_withscriptprocessor.zip 58.4 KB

ChromeStutteryAudioRepro_webrtcsamples_withscriptprocessor.zip 58.4 KB Download

If you overload the main javascript thread, there's not much that can be done if the ScriptProcessorNode doesn't finish in time for the audio thread that can't wait for data.  You'll either get (silence) glitches or duplicates.

AudioWorklets are being actively worked on, but we have no delivery schedule.

Receiving *duplicates* of recorded audio is an obvious bug: If the audio thread cannot push audio into the ScriptProcessorNode in time, then it should *drop* those samples.

There is no situation where it makes sense to send the *same* block twice to the ScriptProcessorNode.
(Note that this is in an audio recording scenario where the recorded samples are processed - the duplicates are on the input to the ScriptProcessorNode.)

I understand that the AudioWorklets https://bugs.chromium.org/p/chromium/issues/detail?id=469639 will solve this issue, but work on those seems to be going slowly ...

If you're just recording audio, can you use the MediaRecorder API (https://developer.mozilla.org/en-US/docs/Web/API/MediaStream_Recording_API)?

I tried the stuttery repro case on my Z620 (a relatively fast machine) running Linux.  I didn't hear any stuttering.

What kind of machine are you running this on?

andreas@

So 'dropping samples' is not a bug? It depends on your perspective. The duplicated block might sound better than a completely silent gap (more unnoticeable). Note that the spec does not mandate the behavior of underflow buffer in the ScriptProcessorNode and I doubt that more work will be done to specify the behavior.

@rtoy:
* We need to process audio in javascript (silence detection, SNR computation, etc.), so we need access to the audio bytes and process them in real time.
* I can reproduce on a Dell laptop with Intel i5 CPU.  It really depends on having "sufficient" load on the Javascript.

@hongchan: My understanding of the audio pipeline may be wrong, but I thought it works somewhat like this:
1) audio is received from the OS in a background thread and queued for the ScriptProcessorNode.
2) At some point in time, the main Javascript thread will notice that the ScriptProcessorNode has work pending and will execute its code (that in turn can process the next queued audio block).

Thus, on an overloaded system that cannot process the audio in real time, I would expect to
a) either see some audio queue on the ScriptProcessorNode growing (and processing lagging behind audio capture until the system load is reduced and it can catch up),
b) or see that some samples are skipped (because they could not be processed in real time, immediately after capture).

What I do not see as any kind of meaningful behavior is having the ScriptProcessor Node process the *same* audio again when the main JavaScript thread is too slow processing the audio in time.

Note that I do not understand how an *underflow* could happen - in the scenario I'm describing the main Javascript thread is busy and therefore apparently too slow in audio processing - so we are in an *overflow* situation.

This issue would not be so pressing if AudioWorklets were already available, but there does not seem to be much progress on https://bugs.chromium.org/p/chromium/issues/detail?id=469639

andreas@

It is unfortunate that this feature behaves differently from your expectation, but your ideas are quite far from how WebAudio engine works. WebAudio engine is real-time system. The realtimeness certainly can be broken if your task exceeds the performance of your system. What you are asking is for us to handle those 'broken' cases in favor of your use case.

> a) either see some audio queue on the ScriptProcessorNode growing (and processing lagging behind audio capture until the system load is reduced and it can catch up)

So accumulating 'missed frames' in the dynamically growing internal queue? Then the data for the next rendering quantum will be delayed and this change will certainly break other WebAudio apps on the web.

> b) or see that some samples are skipped

How do you define those 'some samples'? On what basis? Even if you have some idea for criteria, does it make sense for other generic use cases?

FWIW, this is the buffer *underflow* situation because the main thread failed to fill up the buffer in time.

Lastly, AudioWorklet might not magically solve the issue you're having here. Its primary benefit is the sample-accurate synchronization with other WebAudio components, not "moar performance". Although the code will run on the different thread than the main thread, the buffer underflow still can happen if the task is too big for the render time limit.

hongchan@

Ok, I understand the reason for an underflow now - wasn't aware that the main processing thread is responsible for grabbing audio blocks from the sound card and forwarding them to the ScriptProcessorNode.

I agree that a quick fix will not be easily possible given such a model - apparently Firefox and Edge implemented the audio pipeline differently, because there we are not experiencing this issue.

Concerning AudioWorklet: I am quite certain that this will improve performance, unless I misunderstood the concept.
- According to my understanding, with ScriptProcessorNode the *entire* Javascript processing on the web page (*both* audio processing code *and* code doing other stuff such as modifying the page content) must be executed in time for the audio blocks to be delivered in real time.
- With AudioWorklet (again according to my understanding of the concept), only the audio processing code must be executable in real time.
    Other Javascript code on the page will not influence the audio-specific performance.  At least that is my interpretation of the "rendering thread" concept https://webaudio.github.io/web-audio-api/#dfn-rendering-thread ).

Note that we do not have a performance problem with the audio processing per se - we only see issues if there's both the audio processing code active *and* some other processing active on on the page (e.g. in jQuery or some other framework that is manipulating the page).

Can you confirm my assumption that AudioWorklets will allow us to decouple audio processing from latency on the control thread/main Javascript processing thread?

andreas@

> According to my understanding, with ScriptProcessorNode the *entire* Javascript processing on the web page (*both* audio processing code *and* code doing other stuff such as modifying the page content) must be executed in time for the audio blocks to be delivered in real time.

This is incorrect. There is no hard contact like 'MUST be executed in time' and it is still possible to fail if the AudioWorklet's JS code is too expensive. To put it nicely, we give up a bit of 'realtimeness' to guarantee the sample accurate synchronization with other native AudioNodes. JS developers are still responsible for balancing the load appropriately.

> With AudioWorklet (again according to my understanding of the concept), only the audio processing code must be executable in real time.

I am not sure what you meant by this.

> Other Javascript code on the page will not influence the audio-specific performance.  At least that is my interpretation of the "rendering thread" concept https://webaudio.github.io/web-audio-api/#dfn-rendering-thread).

This is correct. Note that even with native audio nodes rendering audio can fail (glitch) if there are too many AudioNodes in the graph. The AudioWorkletNode will behave exactly the same.

> we only see issues if there's both the audio processing code active *and* some other processing active on on the page

Yes, this is a common scenario where you get glitch out of ScriptProcessorNode.

> Can you confirm my assumption that AudioWorklets will allow us to decouple audio processing from latency on the control thread/main Javascript processing thread?

This is correct; decoupling JS audio code out of the main thread + sample-accurate synchronization. That's the primary benefit of it. However, let me remind you again it will not be a magical escape hatch for everything.

Just want to add that while there is clearly room for improvement in Chrome's ScriptProcessorNode (as shown by Firefox and Edge), we have limited resources, and I think we would rather spend those resources implementing AudioWorklet than fixing the deprecated ScriptProcessorNode.

I agree that AudioWorklet is preferable over ScriptProcessorNode.

Just FYI attached a repro that also plays back the "stutter" effect and makes the scripting load tunable.  It seems that approx. 30ms of uninterrupted JS can trigger the effect.

Deleted: ChromeStutteryAudioRepro_webrtcsamples_withscriptprocessor2.zip 58.9 KB

ChromeStutteryAudioRepro_webrtcsamples_withscriptprocessor2.zip 58.9 KB Download

Increasing the buffer size and moving audio processing to a web worker allowed us to work around the "stutter effect" in essentially all use cases.

We will switch to AudioWorkerNode when it is available, but results are acceptable for now with this workaround.

IMO this issue can be closed.

andreas@

Glad to hear that. For your use case, I still believe the worker-based solution is better than using AudioWorklet, because you're not feeding the audio data back into the graph.

Closing this issue as WontFix per comment #23.