New issue
Advanced search Search tips

Issue 866248 link

Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: Bug



Sign in to add a comment

Simultaneous audio capture and speech recognition does not recognize speech

Project Member Reported by phistuck@gmail.com, Jul 21

Issue description

UserAgent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36

Steps to reproduce the problem:
1. Go to https://jsfiddle.net/Lx51bz2t/5/ on an Android device. For example, LG G 4.
Alternatively, go to https://jsfiddle.net/Lx51bz2t/6/
2. Approve the audio capture permission.

Essentially -
// Capture audio.
navigator.mediaDevices.getUserMedia({audio: true}).then(
stream =>
{
 // Recognize speech.
 const speech = new webkitSpeechRecognition();
 speech.continuous = true;

 // Restart the recognition as long as we record.
 speech.addEventListener("end", () => speech.start());

 speech.start();

 // Record stream.
 const recorder = new MediaRecorder(stream);
 recorder.start();
});

What is the expected behavior?
Simultaneous audio capture, speech recognition and recording without beeps.

What went wrong?
LG G 4 plays rapid beeps repeatedly and does not transcribe almost at all. I once got "no no no no no no" somehow.

Did this work before? N/A 

Does this work in other browsers? N/A

Chrome version: 67.0.3396.99  Channel: stable
OS Version: 6.1 (Windows 7, Windows Server 2008 R2)
Flash Version: 

It works pretty well on desktop. No beeping.
 
Components: Blink>GetUserMedia Internals>Media>Capture Blink>MediaRecording
Labels: -OS-Windows OS-Android
Status: Untriaged (was: Unconfirmed)
Use case - a voice recorder that also transcribes the recording in real time (no other choice, cannot transcribe a pre-recorded file using Web Speech).
The native Google Keep Android application does that successfully, no reason why the Chrome (or the web) could not.
phistuck@ I was told in the past that encoding (into Opus) would throw
off any speech recognition so, off the bat could you try instantiating
the MediaRecorder like:

 const recorder = new MediaRecorder(stream, { mimeType: "audio/webm;codecs=pcm" });

and see if the recognition gets any better?

OTOH we should diagnose if the beeps are natural or introduced in the 
capture or the recording parts. Do they show up if you just record ?
(I.e. minus the speech reco part). 
#2 - no, this is definitely coming due to the speech recognition. My guess is that it beeps as an indication that speech recognition has started and because I also capture (or record, I have not diagnosed it yet, I was focused on making my main goal work which is recording the captured audio. Speech recognition was just a bonus), it immediately stops the recognition and because I keep restarting it when it stops, it just beeps again to indicate it starts again.

In case PCM works, recording to PCM is not a great option, it will weigh a lot and I will have to compress it somehow anyway.
On the other hand, worst case, I guess I can compress it later with web audio and media recorder. Still, on a mobile device, taking large amounts of memory (either on disk or not) is not a great way to roll.
Components: -Internals>Media>Capture -Blink>GetUserMedia
Labels: Needs-Feedback
NextAction: 2018-08-21
phistuck@ we're in a bit of a conundrum: Opus might throw off the Speech
recognition, and PCM would be too bandwidth-hungry :-/
Could you try PCM nonetheless and see if that improves the recognitinon
plz? 
Status: Unconfirmed (was: Untriaged)
Summary: Simultaneous audio capture and speech recognition does not recognize speech (was: Simultaneous audio capture, speech recognition and recording beeps and does not recognize speech)
MediaRecorder is redundant here (it may add up, but it is not necessary).
It is capturing audio combined with speech recognition that creates the conflict.

https://jsfiddle.net/Lx51bz2t/14/

// Capture audio.
navigator.mediaDevices.getUserMedia({audio: true}).then(
stream =>
{
 // Recognize speech.
 const speech = new webkitSpeechRecognition();
 speech.continuous = true;

 // Restart the recognition as long as we record.
 speech.addEventListener("end", () => speech.start());

 speech.start();
});
Speech recognition alone works fine -
https://jsfiddle.net/Lx51bz2t/17/


I confirm the problem on LG G3, too.
Components: -Blink>MediaRecording Blink>GetUserMedia>Mic
Labels: -Needs-Feedback
NextAction: ----
Owner: guidou@chromium.org
Status: Untriaged (was: Unconfirmed)
Adjusting Components. 

guidou@ could you please route to the audio folks plz?
Status: Assigned (was: Untriaged)
[Triage] Setting to assigned since it has an owner.
One more data point that sort of suggests that this combination is not greatly supported - on Windows 7, the tab sometimes crashes when I capture, recognize and record (might not be necessary) simultaneously (perhaps only when the Developer Tools feature is open? Not sure). Once I removed the speech recognition from the mix, it either never crashed, or crashed once in a very long while.

Also, LG G3 software details -
Android 6.0
Security patch level - 2016-09-01
Baseband - MPSS.DI.2.0.1.c1.13-00111-M8974AAAAANPZM-1.44479.1
Kernel - 3.4.0
Build - MRA58K
Software - V30b-425-01

Sign in to add a comment