Captured stream from muted MediaElement unleashes the sound.
Reported by
tristan....@gmail.com,
Sep 11
|
||||||||||||||||||
Issue descriptionChrome Version : 71.0.3549.0 (Official Build) canary (64-bit) and stable 69 OS Version: OS X 10.12 URLs (if applicable) : https://jsfiddle.net/nh30m71g/ Other browsers tested: Add OK or FAIL after other browsers where you have tested this issue: Safari: N/A Firefox: OK IE/Edge: N/A What steps will reproduce the problem? 1. Create a MediaStream from a muted MediaElement with MediaElement#captureStream 2. Set this stream as the srcObject of an other MediaElement, which doesn't have autoplay 3. Enjoy the earing What is the expected result? There should be no sound at this moment. What happens instead of that? We ear sound Here is a live demo using a video https://jsfiddle.net/nh30m71g/ and the same with the final line commented https://jsfiddle.net/nh30m71g/1/ Being a MediaElement bug, this also affects <audio> element. Note that this bug can probably be abused to workaround the new audio autoplay policies.
,
Sep 11
,
Sep 12
,
Sep 12
,
Sep 13
Tested this issue on Mac using chrome reported version-71.0.3549.0 as per the URL's provided in C#0. Observations: ------------ 1. Able to hear background noise when user navigates to https://jsfiddle.net/nh30m71g/ (without playing video too) 2. No audio & no chance to play video when user navigates to https://jsfiddle.net/nh30m71g/1/ Please find the attached screencast and confirm is this the issue you are talking about in chrome? Thanks..!
,
Sep 13
@jmukthavaram@chromium.org Yes that's it. Maybe a sound only fiddle would be better (less data to load)? https://jsfiddle.net/aLfxh512 (add /1 for the silent, commented version). (Ps: I'm not sure how to clear Needs-Feedback label on monorail)
,
Sep 13
Thank you for providing more feedback. Adding the requester to the cc list. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Sep 16
I had time to run a bisect. The ChangeLog URL is: https://chromium.googlesource.com/chromium/src/+log/416c510f0d7085609c95c27402656a4b2a9bf4fb..f520c02822a0b662d9198509117fe8c633352222 So I'd guess commit 94e83cf5a5c9353ab54ba31b82de6c6320917d49 relates to this. cc guidou@chromium.org
,
Sep 17
Able to reproduce issue on reported chrome version 71.0.3549.0 & on latest chrome 71.0.3554.0 using Windows 10,Ubuntu 14.04 and Mac 10.13.6. As per comment #8 providing bisect information below . CHANGELOG URL: https://chromium.googlesource.com/chromium/src/+log/416c510f0d7085609c95c27402656a4b2a9bf4fb..f520c02822a0b662d9198509117fe8c633352222 suspect: https://chromium.googlesource.com/chromium/src/+/954ed9ceb26067a267165e440bd863f74820ddef Reviewed-on: https://chromium-review.googlesource.com/632579 @Guido Urdaneta: Please confirm the issue and help in re-assigning if it is not related to your change.Adding RBS label for M-69 feel free to change it if not required. Thanks!
,
Sep 17
I don't think https://chromium-review.googlesource.com/632579 is related to this bug since it's for the most part a dead-code removal. Also, it landed in M62, in which case this would not be a (recent) regression. Assigning to emircan@, who owns the capture-from-element code for triaging.
,
Sep 17
Assigning to chfremer@ since emircan@ and I are OOO. I'll take a look at this again when I'm back. Removing ReleaseBlock since the bisection points to a range in M62, although this needs verification.
,
Sep 17
I confirmed the bisect. I am getting the same results, and with https://jsfiddle.net/aLfxh512 it is really quick and easy to repro (thanks!). Since this does not seem to be a recent regression, I think it is okay to wait until emircan@ and guidou@ are back.
,
Sep 17
,
Sep 17
tristan.fraipont@gmail.com: Thanks a lot for the bistect, and thanks chfremer@ for confirming it. I'll take care of this next week.
,
Sep 24
,
Sep 24
I investigated the issue and the root cause of the problem is not r498039, but the fact that captureStream() is capturing data from a muted element. Before r498039 the media element was notified about track changes only when they were caused by JavaScript calls addTrack() and removeTrack(), whereas after r498039 the notification is sent when a track is added by other means, such as when the system adds an audio track to the stream returned by captureStream(). This extra notification, which I don't think is a bug, is what's making the element play. I wrote a simple patch that restores notifications to the way they were before (https://chromium-review.googlesource.com/c/chromium/src/+/1240123), but this doesn't fix the underlying issue which is that the stream is capturing audio data from a muted element. The audio data can still be obtained, for example, by sending the stream to a peer connection. I modified the jsfiddle so that the audio data of the stream is analyzed with Web Audio. https://jsfiddle.net/6dy8uf3m/ If the data is not silence, a message is printed in the console. This fiddle fails before and after r498039, and also with my patch (which I do not plan to land).
,
Sep 24
Assigning to emircan@ since he owns the element capture code.
,
Nov 19
emircan@ are you looking into this?
,
Nov 20
I haven't had a chance. Will take a look now.
,
Dec 12
HtmlAudioElementCapturerSource() is responsible for the audio capture portion[0]. We have access to media::WebAudioSourceProviderImpl there but it doesn't expose any indication that audio is muted. We can resolve this issue from the capturer side with any of these two ways: 1) media::WebAudioSourceProviderImpl exposes state 2) media::WebAudioSourceProviderImpl does not call WebAudioSourceProviderImpl::TeeFilter::Render(which calls back the capturer set by SetCopyAudioCallback()) in a muted state because nothing should be captured then. dalecurtis@ can you take a look as well and advise/triage. [0] https://cs.chromium.org/chromium/src/content/renderer/media_capture_from_element/html_audio_element_capturer_source.cc?sq=package:chromium&dr=CSs&g=0&l=26
,
Dec 12
,
Dec 12
Defer to what y'all want. Either solution is easy. So just let me know which you'd prefer, 1) is the only way if you want volume applied versus just being a binary muted/unmuted.
,
Dec 12
Let's go with 1) then.
,
Dec 12
Okay, I'd just expose a const volume() getter on WASPI then. Do you want to just add that in a CL modifying HtmlAudioElementCapturerSource() to use it? I'm not sure what modifications to make there and won't have time to get to that this week.
,
Dec 12
Sure, I can do that.
,
Dec 13
Might the right time to quote the specs and let you know that there was an unrelated bug, previous from this one, exactly in here.
So at the end of [0] the specs say:
> Whether a media element is actively rendering content (e.g., to a screen or audio device) has no effect on the content of captured streams. Muting the audio on a media element does not cause the capture to produce silence, nor does hiding a media element cause captured video to stop. Similarly, the audio level or volume of the media element does not affect the volume of captured audio.
>
> Captured audio from an element with an effective playback rate other than 1.0 MUST be time-stretched. An unplayable playback rate causes the captured audio track to become muted.
So the correct workflow would be (not sure how it will render...)
original mediaSource
=> apply playBack state
=> route to MediaStream
=> apply volume state
=> output to MediaElement sink (e.g system audio)
So the routing to MediaStream should be made from the source as it is after the playBack state is set (either rate or play/pause state), but before the volume is set.
IIRC when I made the bisect for this bug, the previous versions had an incorrect behavior here in that they did route the source to the MedaiStream after the volume state has been applied to it.
Note that Firefox's behavior is also wrong here[1].
[0][https://w3c.github.io/mediacapture-fromelement/#dom-htmlmediaelement-capturestream]
[1][https://bugzilla.mozilla.org/show_bug.cgi?id=966247]
,
Dec 14
Thanks for the pointers on #26. I am deferring this to mcasas@ who worked on the realted portions of the spec.
,
Jan 2
#26, from your reading of the Spec, it seems like the pseudo
flow diagram should be something like the following:
/------ WebAudioSourceProviderImpl ------/
<audio in>--> playback rate -+-> volume --> mute yes/no -->...
\
------<MediaStreamAudioTrack>------>
/---HtmlAudioElementCapturerSource--/
Where HtmlAudioElementCapturerSource should create the MSAudioTrack
with the playback rate applied, but without the volume/mute appplied
to the original audio in, is this your understanding as well?
I'm not sure if WebAudioSourceProviderImpl follows an internal
model that is amenable to the diagram above, i.e. I know it applies
the volume/mute (via its internal TeeFilter class), but I haven't
seen the playback rate.
,
Jan 3
#28 Yes, that's how I read the specs too. |
||||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||||
Comment 1 by vamshi.kommuri@chromium.org
, Sep 11