New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 734290 link

Starred by 6 users

Issue metadata

Status: WontFix
Owner:
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows , Chrome , Mac
Pri: 3
Type: Bug


Show other hotlists

Hotlists containing this issue:
Hotlist-1


Sign in to add a comment

getUserMedia onaudioprocess sometimes drops tiny amounts of audio

Reported by douggerh...@gmail.com, Jun 17 2017

Issue description

UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36

Steps to reproduce the problem:
1. Get an audio stream using getUserMedia
2. Hook that stream up to a ScriptProcessor's onaudioprocess
3. Save the raw audio data passed to onaudioprocess

What is the expected behavior?
Audio passed is not missing data.

What went wrong?
In certain cases, the data provided to the browser is missing small snippets of time – often ranging from 10 to 100ms. Over the course of a long (1hr) recording this can occur dozens of times, and add up to multiple seconds of lost time.

We've documented this occurring in a small percentage of recordings over the last 12 months in Chrome Stable, across Windows, macOS and ChromeOS. While it seems vaguely correlated with processor power (slower machines seem to experience this more frequently) we've come up with no way of reliably reproducing this. That said, users who experience it seem to experience it repeatedly in more than one recording, and users who don't are typically in the clear for all of their recordings. We've never succeeded in duplicating it by trying to compete for processor cycles or anything of that nature.

As part of our work to diagnose & mitigate this issue, we have tracked the time delta between sequential pairs of onaudioprocess calls. Our audio buffer is configured to be 372ms long, and typically those calls occur within a few milliseconds of this value. When audio gets dropped, however, the onaudioprocess call is *late* by the amount of missing audio. Because we're recording each local side of a WebRTC conversation, and because our recordings stop and start simultaneously, we know that these two recordings should be almost identical in length. In cases where one participant is experiencing the audio loss issue, we've found that adding back small snippets of silence where we noted late onaudioprocess calls (each of a duration corresponding to how late the call was) restores the duration of their recording to the precise expected duration.

Did this work before? No 

Chrome version: 59.0.3071.104  Channel: stable
OS Version: OS X 10.12.5
Flash Version: 

I've filed this as a separate bug on the suggestion of grunell@, however I do think this is almost certainly the same issue as 637558.
 
Labels: Needs-Triage-M59
Cc: grunell@chromium.org
Components: Internals>Media>Audio
Labels: TE-NeedsTriageHelp
As per the original comment adding @grunell for more updates on this issue.

Cc: olka@chromium.org solenberg@chromium.org hongchan@chromium.org maxmorin@chromium.org ossu@chromium.org rtoy@chromium.org
Components: Blink>WebAudio Blink>GetUserMedia>Mic
Labels: -TE-NeedsTriageHelp -Needs-Triage-M59 OS-Chrome OS-Windows
Owner: hongchan@chromium.org
Status: Assigned (was: Unconfirmed)
I suspect this could be a WebAudio issue.

hongchan@, can you take a look at this and see what you think?

Comment 4 by rtoy@chromium.org, Jun 23 2017

What buffer size is being used for the ScriptProcessorNode?  That's important. I'm guessing the size if relatively small because you say that slower machines seem to have this problem more often.
Actually when the main thread is busy, the behavior of ScriptProcessor's double buffer switching mechanism becomes non-deterministic. For example, the rendering thread can switch the buffer when the main thread is still writing data into it.

> While it seems vaguely correlated with processor power (slower machines seem to experience this more frequently) we've come up with no way of reliably reproducing this.

This is exactly why I call this behavior is "non-deterministic". When your machine is fast enough to fill the buffer with the main thread, you get the result you expect. However, the main thread can stop or stall for various reasons (e.g. GC or UI modal dialog). When that happens, all bets are off.

FWIW, the sample-accurate audio capturing via ScriptProcessorNode is not supported and the node itself was deprecated from the spec a while ago for this very reason.

Here are my suggestions:

1. If you only need the recording of microphone input, consider using MediaRecorder.
2. Decrease the processing load in the main thread and increase the buffer size of ScriptProcessorNode.
3. Use AudioWorkletNode when it is available.
Hey rtoy@, hongchan@, thanks for the replies here. 

I should have included the details on our buffer size, because that's indeed something we investigated. When we initially ran into this, we were using a buffer of 8192, however we increased that quite a while back to 16384 and we're still seeing the issue at that buffer size. There's no way to get a buffer larger than that, is there?

I do appreciate the suggestions here. We've done our best here to move as much out of the main thread as possible – all our audio encoding is done in a web worker, for example.

What's been really puzzling us here is this: given two machines which are largely identical (same hardware, same OS, same Chrome version, same microphone), one machine may suffer from this issue repeatedly while the other never does. If this happens to a machine once it seems highly predictive that it'll happen again, rather than being evenly distributed across a bunch of similar machines.
> What's been really puzzling us here is this: given two machines which are largely identical (same hardware, same OS, same Chrome version, same microphone), one machine may suffer from this issue repeatedly while the other never does.

The problem being non-deterministic makes almost impossible to predict it, let alone to solve it. If balancing the load on the main thread with a bigger buffer size does not help the issue, please consider using MediaRecorder. Perhaps looking at your trace profiling result (to see what's happening in the main thread) along with CPU usage per thread might be helpful.
@hongchan – thanks, we'll indeed see what we can figure out with the profiler.

One other wrinkle that I'd love to get your thoughts on: currently we get two separate MediaStreams from two separate getUserMedia calls (to the same audio device). One is fed into WebRTC (and hooked up to a separate ScriptProcessorNode which generates data for a live VU meter), and the second is fed into the ScriptProcessorNode discussed here for encoding. Any thoughts on whether that could interact negatively with an issue like this?
> currently we get two separate MediaStreams from two separate getUserMedia calls (to the same audio device)

I don't think this is ideal, but should be orthogonal to the issue we're having if the implementation does the right thing. However, I would reconfigure the graph like this:

                                          +-> DestinatioNode (for live monitoring output)
gUM -> MediaStreamSourceNode -> GainNode -+-> AnalyzerNode (then VUMeter rendering)
                                          +-> ScriptProcessorNode (encoding)

One less ScriptProcessorNode and one less MediaStreamSourceNode. If you're using a particular VU metering algorithm AnalyzerNode might not be what you want, but the native node is fast and does have no main thread business.
Hey hongchan@, I wanted to follow up with an additional wrinkle we've seen with some of our users. 

We've had one report from a user on a powerful (i7-7700HQ) Windows laptop who was experiencing this bug regularly, but it appeared to resolve itself after they shut off Chrome's hardware acceleration.

Any thoughts on how hardware accel could be interacting with the bug?
douggerhardt@

No. I have no clue on how they can be related. If I have to guess, shutting off GPU accel might affect Chrome's thread priority so it changes how the main thread behaves. (gets more chance to run the JS code somehow.)
Labels: -Pri-2 Needs-Feedback Pri-3
However we approach/address this issue, it will be a band-aid workaround. The fix might affect other users in different use cases. Also this is why ScriptProcessorNode is being deprecated from the spec.

The only suggestion I can make is to lower the workload of the main thread and minimize the chance of GC as much as possible.
Hey hongchan@, we've seen a massive increase in the frequency and severity of this issue which appears to be related to the release of Chrome 60. We're now seeing some cases where onaudioprocess is firing late enough with such frequency that 5-10% of an audio recording is simply not being captured. It sounds like something in 60 has introduced a regression that has exacerbated this problem severely.

I definitely understand the desire to avoid breaking something here for other people, but at this point the API unfortunately becoming unusable.
douggerhardt@

Thanks for the report. M60 introduced the new rendering thread for AudioWorklet change and the regression is being tracked in other bug entries. It affects the entire rendering chain of WebAudio, including ScriptProcessorNode. However, the problem with ScriptProcessorNode might be worse because now it has 2 thread hops instead of 1.
hongchan@ – we've taken your suggestion and undertook the extensive work to port our audio encoding over to MediaRecorder, however we're now seeing the same data-loss issue occur in MediaRecorder as well. Please see #761840. 
douggerhardt@

Please compare your app between M60 and M62. From M62, the new threading model is behind the experimental flag. If M62 doesn't affect your app, that means it's not WebAudio's problem. (FWIW, MediaRecorder is outside of WebAudio codebase.)

Could you provide the tracing data of your application? Then we can take a better guess on where the glitches come from. Please make sure untick all the options except for "audio" and "webaudio" when recording the trace.
What flag is this new threading model associated with?  I'd be happy to test this out if I knew the flag to try.
AudioWorklet has been available behind the flag:
https://googlechromelabs.github.io/web-audio-samples/audio-worklet/

This will give you the better recording within AudioWorkletProcessor.
Status: WontFix (was: Assigned)
AudioWorklet has shipped by default in M66. Please replace ScriptProcessorNode with it it if the problem persists.

Sign in to add a comment