Firefox is favorable to support this spec. Firefox's project FoxEye [1] aims to bring the power of computer vision and image processing to the Web. They are interested in depth camera to support Gestural control API. 
[1] https://wiki.mozilla.org/Project_FoxEye

Link to entry on the feature dashboard
https://www.chromestatus.com/feature/5713419084759040

Dongseong, can you also add a link/send an
Intent-to-Implement to blink-dev ? (see
http://www.chromium.org/blink#launch-process)

Hi mcasas, posted at https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/ezyGy6JE7hs
I'm looking forward to your feedback at the thread.

strawman proposal: the project scope is as follows

**Implementation scope**
* Add librealsense to src/thirdparty.
 * (draft) https://codereview.chromium.org/2028143002
* Add depth format in media pipeline.
* Add depth capture delegate (like V4L2CaptureDelegate). It may live in browser process, because it needs to access USB device.
* Implement the compositor to draw depth VideoFrame
* Implement texture copy from depth VideoFrame to WebGL2 texture
* Implement Web API in Blink

**Technical constraint**
* Chromium linux port uses gcc 4.6 while librealsense uses c++11 std library. So librealsense compile fails. :( We will support ChromeOS first.
* We should change librealsense to make it compatible to gcc 4.6. Before it’s done, (involved developers) use GN definition use_sysroot=false to build it in linux.
* Chromium has its own libusb thirdparty library. However, Android doesn't support libusb.
* kinect and hololens don't have official library for posix system.

** platform Roadmap (can be changed) **
* OS: 1. ChromeOS 2. Windows 3. MacOSX and Linux 4. Android
* devices: 1. realsense 2. kinect 3. tango and hololens

mcasas@,
Based on  Issue 352542  #c7, Tango support should be a name mapping from depth to video stream based on device name.

Basically, it would be like having this logic inside Chromium to create a convenience API implementation:
Note that the code isn't tested.

function getConstraintsForDepthDevice() {
  return new Promise(function(resolve, reject) {
    navigator.mediaDevices.enumerateDevices()
    .then(function(devices) {
      for (var i = 0; i < devices.length; ++i) {
        if (devices[i].label.match(/depth/i)) {
          return resolve({video:{deviceId: {exact: devices[i].deviceId}}});
        }
      }
      return reject("No depth device found");
    })
  });
}

function getDepthStream() {
  return new Promise(function(resolve, reject) {
    getConstraintsForDepthDevice()
    .then(function(constraints) {
      if (!constraints)
        return reject("No depth device found");
      return navigator.mediaDevices.getUserMedia(constraints);
    }).then(function(stream) {
      return resolve(stream);
    });
  });
}

Instead of match(/depth/i) we can use (indexOf("Depth") != -1) to address RealSense and (indexOf("depth") != -1) for Tango camera.

Do we need to do implement API extension for this?

Are other browsers going to do the same? If they don't, using the code here seems portable across browsers and extensible to cover future camera models.

#9: don't worry about Tango device. This only supported the
first generation, which AFAIK was prototype(-ish) and wasn't
widely distributed. As a matter of fact, I think I could 
remove the code. For the time being, best treat it as an 
aberration (i.e. ignore its existence).

mcasas@,
#11,
>This only supported the first generation,
I have this device [1] but cannot see the depth device in chrome using [2].
[1] https://developers.google.com/tango/hardware/tablet
[2] https://webrtc.github.io/samples/src/content/devices/input-output/

Do you have more details available about depth stream in the recent Tango devices?
Is it YUV or Y16 stream?
Thanks.

#12: The recent Tango devices and the API centralizes all camera
info in a service, to which you can register as a listener using
e.g. onPointCloudAvailable() [1]. Connecting to this service needed
a specific capability in the Manifest, and that was deemed 
unacceptable since users tend to not update apps when asked for
extended permissions, IIRC.

Probably the Tango 1 device got updated to the new service model
and the old Chromium Java files won't work. (Try [2] as well,
though).


[2] https://rawgit.com/Miguelao/demos/master/gum_multi.html


[1] https://developers.google.com/tango/apis/java/reference/Tango.OnTangoUpdateListener

#12
> Connecting to this service needed a specific capability in the Manifest
This might have changed.
I don't see anything specific needed here [1]:
There is a need to have standard camera in manifest. Is it already there?
If it is, then there is no new permission needed on update.

Then, there is a need to have user request popup for "area learning, depth perception or the camera previews" using the API specified [2].

[1]
https://developers.google.com/tango/ux/ux-permissions

[2]
https://developers.google.com/tango/apis/c/c-user-permissions

#12, #13
mcasas@,
I'll start prototyping this on Tango to check if it works with no new permission required.

#15: Since that work might be sizable, would you
mind creating a specific issue for it and making
this one Blocked on it?

#16:
PrototypedTango v2 capture and the issues identified are in  Issue 674440 .

There is an interesting thing identified in Tango API:
Float values in pointcloud data are scaled and projected to 3D space. They come in packed XYZ(C) form [1].
In order to keep the current mediacapture-depth API approach (width x height buffer of float or unsigned short) we'd need to deproject them to 2D and store only Z value in buffer.
Tango Support library has a way to do this [1] but, performance wise, it seems better to provide the original format (XYZ): I plan to raise this issue in mediacapture-depth and check with Tango support team.

Another thing is that float values are scaled (in meters) and we could reconsider doing the same. Currently the values in WebGL texture are normalized [0-1.0]. Getting the depth capture scale (near and far values) [3] is still work in progress so we could also reconsider to do this later.
Both, normalized and denormalized float values would fit as the scale would just be 1 in case of providing deprojected values.

[1]
https://developers.google.com/tango/apis/c/reference/struct/tango-point-cloud
[2]
https://developers.google.com/tango/apis/c/support/reference/group/depth-interpolation-support
[3]
https://w3c.github.io/mediacapture-depth/#mediatracksettings-dictionary

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac

commit 9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac
Author: aleksandar.stojiljkovic <aleksandar.stojiljkovic@intel.com>
Date: Tue Feb 07 07:44:20 2017

Media Capture Depth Stream Extensions API: focal length and depth range.

Under experimental MediaCaptureDepth feature, implements [1]:

partial dictionary MediaTrackSettings {
    double depthNear;
    double depthFar;
    double focalLengthX;
    double focalLengthY;
};

This patch adds the support for it to fake capture device only.

[1]
https://w3c.github.io/mediacapture-depth/#mediatracksettings-dictionary

BUG=616098

Review-Url: https://codereview.chromium.org/2606983002
Cr-Commit-Position: refs/heads/master@{#448567}

[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/content/browser/renderer_host/media/media_stream_dispatcher_host_unittest.cc
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/content/browser/renderer_host/media/media_stream_manager.cc
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/content/browser/renderer_host/media/media_stream_manager.h
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/content/browser/renderer_host/media/video_capture_manager.cc
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/content/browser/renderer_host/media/video_capture_manager.h
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/content/browser/webrtc/webrtc_depth_capture_browsertest.cc
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/content/common/media/media_stream_messages.h
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/content/public/common/media_stream_request.h
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/content/renderer/media/media_stream_video_track.cc
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/content/test/data/media/getusermedia-depth-capture.html
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/media/capture/ipc/capture_param_traits_macros.h
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/media/capture/video/fake_video_capture_device_factory.cc
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/media/capture/video/fake_video_capture_device_unittest.cc
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/media/capture/video/video_capture_device_descriptor.h
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/third_party/WebKit/Source/modules/mediastream/MediaConstraintsImpl.cpp
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/third_party/WebKit/Source/modules/mediastream/MediaStreamTrack.cpp
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/third_party/WebKit/Source/modules/mediastream/MediaTrackConstraintSet.idl
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/third_party/WebKit/Source/modules/mediastream/MediaTrackSettings.idl
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/third_party/WebKit/Source/modules/mediastream/MediaTrackSupportedConstraints.idl
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/third_party/WebKit/Source/platform/RuntimeEnabledFeatures.in
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/third_party/WebKit/Source/platform/exported/WebMediaConstraints.cpp
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/third_party/WebKit/public/platform/WebMediaConstraints.h
[modify] https://crrev.com/9e1e3f73fc41ca1f8a5d42b3a418c286c2e39fac/third_party/WebKit/public/platform/WebMediaStreamTrack.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/28ea8a96d0788781f3ef06053c921d8e740a30ef

commit 28ea8a96d0788781f3ef06053c921d8e740a30ef
Author: aleksandar.stojiljkovic <aleksandar.stojiljkovic@intel.com>
Date: Tue Feb 14 13:31:15 2017

Media Capture Depth Stream Extensions API: videoKind settings and constraint.

This enables selection of depth stream on Linux; both color and
depth device have the same Label and it is not possible to apart
depth from color stream ( https://crbug.com/686756 ).

Under experimental MediaCaptureDepth feature, implements [1]:

partial dictionary MediaTrackConstraintSet {
    ConstrainDOMString videoKind;
};

and constraining logic applying it to MediaStreamTrack object.

partial dictionary MediaTrackSettings {
    DOMString videoKind;
};

This patch is verified to work with fake capture device and RealSense SR300.

[1]
https://w3c.github.io/mediacapture-depth/#mediatracksettings-dictionary

BUG= 686756 , 616098

Review-Url: https://codereview.chromium.org/2664673002
Cr-Commit-Position: refs/heads/master@{#450335}

[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/content/browser/webrtc/webrtc_depth_capture_browsertest.cc
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/content/renderer/media/media_stream_constraints_util_video_source.cc
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/content/renderer/media/media_stream_constraints_util_video_source.h
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/content/renderer/media/media_stream_constraints_util_video_source_unittest.cc
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/content/renderer/media/media_stream_video_source.cc
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/content/renderer/media/media_stream_video_track.cc
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/content/test/data/media/depth_stream_test_utilities.js
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/content/test/data/media/getusermedia-depth-capture.html
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/third_party/WebKit/Source/modules/mediastream/MediaConstraintsImpl.cpp
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/third_party/WebKit/Source/modules/mediastream/MediaStreamTrack.cpp
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/third_party/WebKit/Source/modules/mediastream/MediaTrackConstraintSet.idl
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/third_party/WebKit/Source/modules/mediastream/MediaTrackSettings.idl
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/third_party/WebKit/Source/modules/mediastream/MediaTrackSupportedConstraints.idl
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/third_party/WebKit/Source/platform/exported/WebMediaConstraints.cpp
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/third_party/WebKit/public/platform/WebMediaConstraints.h
[modify] https://crrev.com/28ea8a96d0788781f3ef06053c921d8e740a30ef/third_party/WebKit/public/platform/WebMediaStreamTrack.h

 Issue 746765  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ed7555f98074e69a08a0fe4cf03dccb421d7fa6c

commit ed7555f98074e69a08a0fe4cf03dccb421d7fa6c
Author: Aleksandar Stojiljkovic <aleksandar.stojiljkovic@intel.com>
Date: Fri Aug 11 13:27:16 2017

Media Capture Depth Stream Extensions: split videoKind feature.

Split videoKind from MediaCaptureDepth to MediaCaptureDepthVideoKind
feature in order to ship it sooner then the other Media Capture Depth
stream API extensions.

videoKind is important as it enables constraining query (getUserMedia)
to depth stream - see https://codereview.chromium.org/2664673002/

BUG= 686756 , 616098

Change-Id: I4cca58d763278bbe74d693a2a2ff4b6801a4d126
Reviewed-on: https://chromium-review.googlesource.com/608067
Reviewed-by: Guido Urdaneta <guidou@chromium.org>
Reviewed-by: Kentaro Hara <haraken@chromium.org>
Commit-Queue: Aleksandar Stojiljkovic <aleksandar.stojiljkovic@intel.com>
Cr-Commit-Position: refs/heads/master@{#493711}
[modify] https://crrev.com/ed7555f98074e69a08a0fe4cf03dccb421d7fa6c/content/browser/webrtc/webrtc_depth_capture_browsertest.cc
[modify] https://crrev.com/ed7555f98074e69a08a0fe4cf03dccb421d7fa6c/third_party/WebKit/Source/modules/mediastream/MediaStreamTrack.cpp
[modify] https://crrev.com/ed7555f98074e69a08a0fe4cf03dccb421d7fa6c/third_party/WebKit/Source/modules/mediastream/MediaTrackConstraintSet.idl
[modify] https://crrev.com/ed7555f98074e69a08a0fe4cf03dccb421d7fa6c/third_party/WebKit/Source/modules/mediastream/MediaTrackSettings.idl
[modify] https://crrev.com/ed7555f98074e69a08a0fe4cf03dccb421d7fa6c/third_party/WebKit/Source/modules/mediastream/MediaTrackSupportedConstraints.idl
[modify] https://crrev.com/ed7555f98074e69a08a0fe4cf03dccb421d7fa6c/third_party/WebKit/Source/platform/RuntimeEnabledFeatures.json5

This issue has been automatically relabelled type=task because type=launch-owp issues are now officially deprecated. The deprecation is because they were creating confusion about how to get launch approvals, which should be instead done via type=launch issues.

We recommend this issue be used for implementation tracking (for public visibility), but if you already have an issue for that, you may mark this as duplicate.

For more details see here: https://docs.google.com/document/d/1JA6RohjtZQc26bTrGoIE_bSXGXUDQz8vc6G0n_sZJ2o/edit

For any questions, please contact owencm, sshruthi, larforge