Add a method to inform media tracks about their type of content |
||||||||||||
Issue description
Change description:
Optimal methods for treating and encoding audio/video data are different depending on content type.
For audio processing in webrtc/; noise reduction, gain and level controller and others are optimized for speech. These would be possible to turn off for music or use other modes.
For video, text content is sensitive to quantization, whileas movies and games are usually intelligible even with high quantization artifacts.
For WebRTC Chrome uses "screencast" settings for anything desktop/tab capture. This optimizes encoding settings for text content in web sites/presentations (drops frames to preserve low quantization), but these settings are often inappropriate for screensharing YouTube videos or video games.
When a web app can be aware of the content it can help inform encoder/processing decision and thresholds. Game livestreaming services would be likely to stream realtime content (video games). It's not always possible (without perfect content detection) to determine which settings are appropriate for which scenario.
Changes to API surface:
* MediaStreamTrack:
Add a writable optional contentTypeHint to audio/video tracks to inform ingestion points about its content.
* Audio:
contentTypeHint: {"speech", "music"}
* Video:
contentTypeHint: {"text", "video"}
Links:
Public standards discussion: https://github.com/w3c/mediacapture-main/issues/391
Support in other browsers:
Internet Explorer: No
Firefox: No
Safari: No
,
Oct 6 2016
,
Oct 6 2016
,
Dec 2 2016
There's a spec draft for this feature which just moved over to WICG at https://github.com/WICG/mst-content-hint.
,
Dec 2 2016
,
Dec 2 2016
,
Dec 13 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a592cf9a183f3caa8f9df64c962277d57c9d5662 commit a592cf9a183f3caa8f9df64c962277d57c9d5662 Author: pbos <pbos@chromium.org> Date: Tue Dec 13 21:30:14 2016 Implement MediaStreamTrack contentHint skeleton. No sinks yet handle MediaStreamTrack content hints, but they are exposed (as a runtime-enabled feature) and passed to audio/video sinks inside Chromium. BUG= chromium:653531 R=esprehn@chromium.org, guidou@chromium.org, jam@chromium.org Review-Url: https://codereview.chromium.org/2501623003 Cr-Commit-Position: refs/heads/master@{#438279} [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/content/public/renderer/media_stream_sink.h [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/content/renderer/media/media_stream_audio_track.cc [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/content/renderer/media/media_stream_audio_track.h [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/content/renderer/media/media_stream_center.cc [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/content/renderer/media/media_stream_center.h [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/content/renderer/media/media_stream_track.h [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/content/renderer/media/media_stream_video_track.cc [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/content/renderer/media/media_stream_video_track.h [add] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/third_party/WebKit/LayoutTests/fast/mediastream/MediaStreamTrack-contentHint.html [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/third_party/WebKit/LayoutTests/webexposed/global-interface-listing-expected.txt [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/third_party/WebKit/Source/modules/mediastream/MediaStreamTrack.cpp [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/third_party/WebKit/Source/modules/mediastream/MediaStreamTrack.h [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/third_party/WebKit/Source/modules/mediastream/MediaStreamTrack.idl [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/third_party/WebKit/Source/platform/RuntimeEnabledFeatures.in [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/third_party/WebKit/Source/platform/exported/WebMediaStreamTrack.cpp [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/third_party/WebKit/Source/platform/mediastream/MediaStreamCenter.cpp [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/third_party/WebKit/Source/platform/mediastream/MediaStreamCenter.h [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/third_party/WebKit/Source/platform/mediastream/MediaStreamComponent.cpp [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/third_party/WebKit/Source/platform/mediastream/MediaStreamComponent.h [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/third_party/WebKit/public/platform/WebMediaStreamCenter.h [modify] https://crrev.com/a592cf9a183f3caa8f9df64c962277d57c9d5662/third_party/WebKit/public/platform/WebMediaStreamTrack.h
,
Dec 13 2016
#8: [1] defines a partial interface, so I suggest moving it to a file on its own, the same way as e.g. NFC does [2] (note the special .gni file section for these files [3]). [1] https://wicg.github.io/mst-content-hint/#extension-to-mediastreamtrack [2] https://cs.chromium.org/chromium/src/third_party/WebKit/Source/modules/nfc/NavigatorNFC.idl?q=third_party/webkit/Source+file:%5C.idl+%22partial+interface%22&sq=package:chromium&l=9&dr=C [3] https://cs.chromium.org/chromium/src/third_party/WebKit/Source/modules/modules_idl_files.gni?q=NavigatorNFC.idl&sq=package:chromium&dr=C&l=540
,
Dec 14 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1d43319164d27261a4e15df9e7b78b5700754f6a commit 1d43319164d27261a4e15df9e7b78b5700754f6a Author: pbos <pbos@chromium.org> Date: Wed Dec 14 20:06:28 2016 Move MediaStreamTrack contentHint to separate IDL. BUG= chromium:653531 R=esprehn@chromium.org, mcasas@chromium.org Review-Url: https://codereview.chromium.org/2578623002 Cr-Commit-Position: refs/heads/master@{#438603} [modify] https://crrev.com/1d43319164d27261a4e15df9e7b78b5700754f6a/third_party/WebKit/Source/modules/mediastream/BUILD.gn [modify] https://crrev.com/1d43319164d27261a4e15df9e7b78b5700754f6a/third_party/WebKit/Source/modules/mediastream/MediaStreamTrack.idl [add] https://crrev.com/1d43319164d27261a4e15df9e7b78b5700754f6a/third_party/WebKit/Source/modules/mediastream/MediaStreamTrackContentHint.h [add] https://crrev.com/1d43319164d27261a4e15df9e7b78b5700754f6a/third_party/WebKit/Source/modules/mediastream/MediaStreamTrackContentHint.idl [modify] https://crrev.com/1d43319164d27261a4e15df9e7b78b5700754f6a/third_party/WebKit/Source/modules/modules_idl_files.gni
,
Dec 16 2016
I'd like to understand the plan regarding degradation preference. As far as I understand, there are three parts to that: 1. It needs to be propagated down from js into webrtc. 2. Then somewhere a decision has to be made on precisely how to apply it. I imagine that would be close the the cpu adaptation (for constrained cpu) and quality scaler (constrained bandwidth). 3. And then the decision has to be applied to the stream of frames. I think the VideoAdapter is the natural place to do that in the current code, unless we want to rip that out, e.g., to move it down the pipeline closer to the encoder.
,
Dec 16 2016
FWIW this isn't degradation preference, but closely related. The property is set on a MediaStreamTrack and any sinks can listen to it, even if their corresponding standard do not expose degradation preferences. For RTCRtpSender I think it'd help inform what "balanced" should do, but not override either preferring framerate or resolution. For PeerConnection that does not expose degradation preferences, I believe it would be treated as if it had "balanced". The initial implementation for video tracks will use contentHints to toggle between treating the stream as "screencast", "not screencast" and its default (tab/desktop capture == "screencast", anything else is "not screencast"). I think going from "screencast" to "not screencast" to make WebRtcVideoCapturerAdapter adapt/not adapt incoming frames + setting VideoOptions::is_screencast. In the process these should probably/possibly be renamed. 1) The blink/js implementation is already done under a flag in comment #8, just that no sinks yet listen to it. I have an ongoing implementation on disk that works when setting contentHints before connecting the tracks but they need to propagate. 2) This decision is made in content/renderer initially. We could push it down further, but since VideoAdapter is in Chromium right now Chromium needs to be aware of it. 3) For now I think this can be kept where it is. Otherwise we need to propagate the decision down. Either works, and having "one place" where adaptation happens might make more sense. Something to consider is that we might want to make the chain: WebRtcVideoCapturerAdapter::OnFrameCaptured -> webrtc::Something::OnFrameCaptured -> (callback)::ScaleFrame(crop_x, crop_y, width, height, ...) -> webrtc:: continues taking care of the frame. In that case it would be sufficient for Chromium to implement ScaleFrame instead of all of OnFrameCaptured today. Chromium should the configure webrtc::Something in this model. This is diverging from the bug topic though. :)
,
Dec 16 2016
The following revision refers to this bug: https://chromium.googlesource.com/external/webrtc.git/+/5214a0ab8e2b622fa7aff888a740e46e80f7ed18 commit 5214a0ab8e2b622fa7aff888a740e46e80f7ed18 Author: pbos <pbos@webrtc.org> Date: Fri Dec 16 23:39:11 2016 Add support for content hints to VideoTrack. Permits overriding the source-default is_screencast option to be able to treat screencast sources as fluid video, preserving motion at the loss of individual frame quality (or vice versa). BUG= chromium:653531 R=deadbeef@webrtc.org Review-Url: https://codereview.webrtc.org/2579993003 Cr-Commit-Position: refs/heads/master@{#15659} [modify] https://crrev.com/5214a0ab8e2b622fa7aff888a740e46e80f7ed18/webrtc/api/mediastreaminterface.h [modify] https://crrev.com/5214a0ab8e2b622fa7aff888a740e46e80f7ed18/webrtc/api/mediastreamtrackproxy.h [modify] https://crrev.com/5214a0ab8e2b622fa7aff888a740e46e80f7ed18/webrtc/api/rtpsender.cc [modify] https://crrev.com/5214a0ab8e2b622fa7aff888a740e46e80f7ed18/webrtc/api/rtpsender.h [modify] https://crrev.com/5214a0ab8e2b622fa7aff888a740e46e80f7ed18/webrtc/api/rtpsenderreceiver_unittest.cc [modify] https://crrev.com/5214a0ab8e2b622fa7aff888a740e46e80f7ed18/webrtc/api/test/fakevideotracksource.h [modify] https://crrev.com/5214a0ab8e2b622fa7aff888a740e46e80f7ed18/webrtc/api/videotrack.cc [modify] https://crrev.com/5214a0ab8e2b622fa7aff888a740e46e80f7ed18/webrtc/api/videotrack.h
,
Dec 17 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/68b13d51ceed682741339133dd461bddfc092b8d commit 68b13d51ceed682741339133dd461bddfc092b8d Author: pbos <pbos@chromium.org> Date: Sat Dec 17 01:15:51 2016 Roll WebRTC 15652:15659 (7 commits) Changes: https://chromium.googlesource.com/external/webrtc/trunk/webrtc.git/+log/5ea8737..37d7b3f $ git log 5ea8737..37d7b3f --date=short --no-merges --format=%ad %ae %s 2016-12-16 pbos@webrtc.org Add support for content hints to VideoTrack. 2016-12-16 asapersson@webrtc.org Add full stack tests: - ForemanCif30kbpsWithoutPacketLoss - ForemanCif30kbpsWithoutPacketLossH264 2016-12-16 terelius@webrtc.org Remove redundant local variable from OveruseDetector::Detect. 2016-12-16 kthelgason@webrtc.org Remove device HW id -> marketing name mapping table for iOS devices. 2016-12-16 kthelgason@webrtc.org Delete unused code from systeminfo. 2016-12-16 ivoc@webrtc.org Fix for integer overflow in NetEq. 2016-12-16 danilchap@webrtc.org In RtpPacket do not keep pointer to RtpHeaderExtensionMap BUG= chromium:653531 R=mcasas@chromium.org CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_archive_rel_ng;master.tryserver.chromium.mac:mac_chromium_archive_rel_ng Review-Url: https://codereview.chromium.org/2580123002 Cr-Commit-Position: refs/heads/master@{#439276} [modify] https://crrev.com/68b13d51ceed682741339133dd461bddfc092b8d/DEPS
,
Dec 20 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/49b5535454c4a02113b41d7f7dc5a597461270be commit 49b5535454c4a02113b41d7f7dc5a597461270be Author: pbos <pbos@chromium.org> Date: Tue Dec 20 00:55:34 2016 Propagate MediaStreamTrack video content hints. Fluid/detailed overrides the screencast settings inside webrtc which disables downscaling and configures libvpx differently. Also disables resolution adaptation on the Chromium side corresponding to the IsScreencast setting. BUG= chromium:653531 R=emircan@chromium.org, mcasas@chromium.org Review-Url: https://codereview.chromium.org/2585313002 Cr-Commit-Position: refs/heads/master@{#439643} [modify] https://crrev.com/49b5535454c4a02113b41d7f7dc5a597461270be/content/renderer/media/webrtc/media_stream_video_webrtc_sink.cc [modify] https://crrev.com/49b5535454c4a02113b41d7f7dc5a597461270be/content/renderer/media/webrtc/media_stream_video_webrtc_sink.h [modify] https://crrev.com/49b5535454c4a02113b41d7f7dc5a597461270be/content/renderer/media/webrtc/webrtc_video_capturer_adapter.cc [modify] https://crrev.com/49b5535454c4a02113b41d7f7dc5a597461270be/content/renderer/media/webrtc/webrtc_video_capturer_adapter.h [modify] https://crrev.com/49b5535454c4a02113b41d7f7dc5a597461270be/content/renderer/media/webrtc/webrtc_video_capturer_adapter_unittest.cc
,
Dec 21 2016
A demo page for this feature is now available on https://webrtc.github.io/samples/src/content/capture/video-contenthint/. It requires experimental web platform features for both mst-content-hint and video tag captureStream.
,
Dec 28 2016
https://github.com/WICG/mst-content-hint/commit/f6b8b9b2be7392571c7e15fd7e7746650ed8826e renames "fluid" and "detailed" to "motion" and "detail".
,
Jan 10 2017
This is in under a flag but not likely hitting general availability in M57, punting milestone.
,
Jan 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c8e5f25d92df69e6d4ea39223d22eb7f1e4e0cd3 commit c8e5f25d92df69e6d4ea39223d22eb7f1e4e0cd3 Author: pbos <pbos@chromium.org> Date: Tue Jan 17 16:24:59 2017 Update MediaStreamTrack content-hint values. Spec draft has been updated to use 'motion' and 'detail' instead of 'fluid' and 'detailed'. This was updated as part of https://github.com/WICG/mst-content-hint/issues/20. BUG= 653531 R=foolip@chromium.org, mcasas@chromium.org Review-Url: https://codereview.chromium.org/2623263003 Cr-Commit-Position: refs/heads/master@{#444067} [modify] https://crrev.com/c8e5f25d92df69e6d4ea39223d22eb7f1e4e0cd3/content/renderer/media/webrtc/media_stream_video_webrtc_sink.cc [modify] https://crrev.com/c8e5f25d92df69e6d4ea39223d22eb7f1e4e0cd3/content/renderer/media/webrtc/webrtc_video_capturer_adapter.cc [modify] https://crrev.com/c8e5f25d92df69e6d4ea39223d22eb7f1e4e0cd3/content/renderer/media/webrtc/webrtc_video_capturer_adapter_unittest.cc [modify] https://crrev.com/c8e5f25d92df69e6d4ea39223d22eb7f1e4e0cd3/third_party/WebKit/LayoutTests/fast/mediastream/MediaStreamTrack-contentHint.html [modify] https://crrev.com/c8e5f25d92df69e6d4ea39223d22eb7f1e4e0cd3/third_party/WebKit/Source/modules/mediastream/MediaStreamTrack.cpp [modify] https://crrev.com/c8e5f25d92df69e6d4ea39223d22eb7f1e4e0cd3/third_party/WebKit/Source/platform/mediastream/MediaStreamComponent.cpp [modify] https://crrev.com/c8e5f25d92df69e6d4ea39223d22eb7f1e4e0cd3/third_party/WebKit/public/platform/WebMediaStreamTrack.h
,
Feb 28 2017
,
Sep 5 2017
Discussion moved to adopt into mediacapture-main. This needs standardization to be viable to remove from experiment. https://github.com/w3c/mediacapture-main/issues/478
,
Sep 12 2017
This issue has been automatically relabelled type=task because type=launch-owp issues are now officially deprecated. The deprecation is because they were creating confusion about how to get launch approvals, which should be instead done via type=launch issues. We recommend this issue be used for implementation tracking (for public visibility), but if you already have an issue for that, you may mark this as duplicate. For more details see here: https://docs.google.com/document/d/1JA6RohjtZQc26bTrGoIE_bSXGXUDQz8vc6G0n_sZJ2o/edit For any questions, please contact owencm, sshruthi, larforge
,
Feb 27 2018
,
Feb 27 2018
Reassigning to hta@ who's tracking progress in standards bodies.
,
May 30 2018
Standards body status: The spec has been adopted by the W3C WEBRTC WG, and is being progressed. The only change suggested so far is to adopt a third type "text" for video content hints - this would be "detailed", but stronger. https://github.com/w3c/mst-content-hint/issues/23 Approved, but not yet published as FPWD.
,
Aug 8
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/2bf254863ad6b3d8df9807349660722025e18037 commit 2bf254863ad6b3d8df9807349660722025e18037 Author: Harald Alvestrand <hta@chromium.org> Date: Wed Aug 08 17:15:40 2018 Count usage of MediaStreamTrack.contentHint Bug: chromium:653531 Change-Id: I3d9741962cd3845a5f351587509c5beb4e27c796 Reviewed-on: https://chromium-review.googlesource.com/1164949 Reviewed-by: Philip Jägenstedt <foolip@chromium.org> Reviewed-by: Guido Urdaneta <guidou@chromium.org> Commit-Queue: Philip Jägenstedt <foolip@chromium.org> Cr-Commit-Position: refs/heads/master@{#581608} [modify] https://crrev.com/2bf254863ad6b3d8df9807349660722025e18037/third_party/blink/public/platform/web_feature.mojom [modify] https://crrev.com/2bf254863ad6b3d8df9807349660722025e18037/third_party/blink/renderer/modules/mediastream/media_stream_track_content_hint.idl [modify] https://crrev.com/2bf254863ad6b3d8df9807349660722025e18037/tools/metrics/histograms/enums.xml
,
Aug 16
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/db1f983e0b873a034f75af1efc52f7d14eb884e1 commit db1f983e0b873a034f75af1efc52f7d14eb884e1 Author: Harald Alvestrand <hta@chromium.org> Date: Thu Aug 16 09:06:07 2018 Enable the "Content hint" attribute of MediaStreamTrack Also moves a test from internal to public repo. Bug: chromium:653531 Change-Id: I99224f86fb9abb44784c861b6c2ba0b3c9e020c4 Reviewed-on: https://chromium-review.googlesource.com/1175124 Reviewed-by: Philip Jägenstedt <foolip@chromium.org> Commit-Queue: Harald Alvestrand <hta@chromium.org> Cr-Commit-Position: refs/heads/master@{#583585} [rename] https://crrev.com/db1f983e0b873a034f75af1efc52f7d14eb884e1/third_party/WebKit/LayoutTests/external/wpt/mst-content-hint/MediaStreamTrack-contentHint.html [modify] https://crrev.com/db1f983e0b873a034f75af1efc52f7d14eb884e1/third_party/WebKit/LayoutTests/virtual/stable/webexposed/global-interface-listing-expected.txt [modify] https://crrev.com/db1f983e0b873a034f75af1efc52f7d14eb884e1/third_party/blink/renderer/platform/runtime_enabled_features.json5
,
Aug 16
,
Aug 27
,
Sep 24
will contentHint be get-able from tracks as defined in the standard in M70? currently it seems that it is not exposed via track API ``` > window.pc1.getSenders()[0].track.contentHint *undefined* ``` |
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by pbos@chromium.org
, Oct 6 2016