Support depth capture and Media Capture Depth Stream Extensions API on Android Tango |
|||||||
Issue descriptionThis is a sub-task of Issue 616098. Currently, only Realsense depth camera capture is implemented. We should add also Tango depth capture if the API supports the use case here and the required permission model is acceptable for Chrome. So, the task should start by verifying if the permission model is acceptable. Comment at https://crbug.com/616098#c13 explains the permissions issue with the recent Tango devices. Note: In Issue 664549 , Tango v1 capture support was removed. "The recent Tango devices and the API centralizes all camera info in a service, to which you can register as a listener using e.g. onPointCloudAvailable() [1]. Connecting to this service needed a specific capability in the Manifest, and that was deemed unacceptable since users tend to not update apps when asked for extended permissions, IIRC. " Documentation shows that this might be changed [2] There is a need to have standard camera in manifest and to have user request popup for "area learning, depth perception or the camera previews" using the API specified [3]. [1] https://developers.google.com/tango/apis/java/reference/Tango.OnTangoUpdateListener [2] https://developers.google.com/tango/ux/ux-permissions [3] https://developers.google.com/tango/apis/c/c-user-permissions
,
Dec 16 2016
,
Dec 17 2016
Did a quick prototype on this: https://crrev.com/2582193002/ Do not review. Tango V2 depth capture prototype Findings: Good: Getting intrinsics from depth and color cameras works e.g. mTango.getCameraIntrinsics(TangoCameraIntrinsics.TANGO_CAMERA_DEPTH). onPoseAvailable and onTangoEvent events are received. No additional permission is required to access depth data (just the android.permission.CAMERA). UNRESOLVED: Only onPointCloudAvailable event is not received. The same code run in standalone application receives pointcloud in standalone example application, e.g: https://github.com/googlesamples/tango-examples-java/tree/master/java_basic_examples/hello_depth_perception/src/main No visible error in logcat output compare to other Tango apps that receive depth(pointcloud) data. This is why in the prototype I tried also running the tango interaction in display thread - no change. This needs to be resolved before continuing. Closed source warning: TangoSDK client java library is closed source. jar file added to third_party/TangoSDK is copied from https://developers.google.com/tango/downloads. The rest of the code is revert of https://crbug.com/664549 Consider removing Tango Chrome video capture support. This code would need to be refactored but a portion of it seems useful. Build model and device id for new Tango device are {"Project Tango Tablet Development Kit", "yellowstone"}. We need to get help from someone from Tango team on debugging why chromium is not receiving onPointCloudAvailable events only.
,
Dec 19 2016
,
Jan 18 2017
,
Jun 13 2017
Managed to get the pointcloud data in the prototype. Continuing. In addition to "standard capture" using android.hardware.Camera (we get front and back camera), Tango service offers access RGB camera (1920 x 1080), fisheye and depth camera. Plan to expose those, too as redirecting the access from chromium process through Tango Service to CameraService should allow simultaneous capture from multiple cameras. About the depth: Unfortunately, depth raw data is now available (it's format HAL_PIXEL_FORMAT_Y16 fits well with existing depth capture). Raw data size is size width 224 and height 172 (the information is from logcat pasted bellow). Pointcloud data is calculated based on it and for the first version I plan to translate (back) pointcloud data to 2D depth texture. Let's see how expensive projection 3D depth space (point cloud) -> 2D buffer on CPU for such resolution would be. mcasas@ I have an impression (assumption) that battery life could get better (Lenovo Phab 2 Pro) if changing the Tango API design; to allow the developer doing the raw -> pointcloud conversion on GPU when needed. I'll try to report the bug / feature request and to identify who to talk to in Tango team. 06-13 12:24:56.030 9817 9833 D tango-hal: tango-context.cpp:Start():257 Found camera id 3 with sensor width 224 and height 172 06-13 12:24:56.030 9817 9833 D tango-hal: tango-context.cpp:Start():305 Found depth sensor with id 3 06-13 12:24:56.032 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():120 Found 21 vendor tags! 06-13 12:24:56.032 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():138 Found vendor tag capture_range with id 2147876864 06-13 12:24:56.032 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():138 Found vendor tag long_range_fps with id 2147876865 06-13 12:24:56.032 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():138 Found vendor tag short_range_fps with id 2147876866 06-13 12:24:56.032 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():138 Found vendor tag pmd_debug_string with id 2147876872 06-13 12:24:56.032 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():138 Found vendor tag frameid with id 2147942400 06-13 12:24:56.033 565 565 E : tango-property-server.cpp:getProperty():53 Property persist.tango.accelhz not set 06-13 12:24:56.033 9817 10251 D tango-hal: tango-utils.cpp:getConfig():122 Property persist.tango.accelhz not set 06-13 12:24:56.033 9817 10251 D tango-hal: tango-sensors.cpp:run():132 Accel min delay 5000, using 5000 06-13 12:24:56.033 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():166 Found 6 short range fps values 06-13 12:24:56.033 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():171 Supported short-range FPS: 0 06-13 12:24:56.033 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():171 Supported short-range FPS: 5 06-13 12:24:56.033 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():171 Supported short-range FPS: 10 06-13 12:24:56.033 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():171 Supported short-range FPS: 15 06-13 12:24:56.033 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():171 Supported short-range FPS: 30 06-13 12:24:56.033 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():171 Supported short-range FPS: 45 06-13 12:24:56.034 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():187 Found 7 long range fps values 06-13 12:24:56.034 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():190 Supported long-range FPS: 0 06-13 12:24:56.034 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():190 Supported long-range FPS: 1 06-13 12:24:56.034 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():190 Supported long-range FPS: 2 06-13 12:24:56.034 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():190 Supported long-range FPS: 3 06-13 12:24:56.034 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():190 Supported long-range FPS: 5 06-13 12:24:56.034 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():190 Supported long-range FPS: 10 06-13 12:24:56.034 9817 9833 D tango-hal: tango-camera.cpp:readVendorTagsIds():190 Supported long-range FPS: 15 06-13 12:24:56.036 9817 9833 D tango-hal: tango-camera.cpp:isResolutionSupported():356 Supported RAW resolution on camera 3: 224 x 1557 06-13 12:24:56.036 9817 9833 D tango-hal: tango-camera.cpp:isResolutionSupported():360 RAW resolution supports desired resolution 06-13 12:24:56.038 9817 9833 D tango-hal: tango-camera.cpp:isFormatSupported():411 Supported format on camera 3: id=0x21 name:HAL_PIXEL_FORMAT_BLOB 06-13 12:24:56.038 9817 9833 D tango-hal: tango-camera.cpp:isFormatSupported():416 Camera 3 supports desired format HAL_PIXEL_FORMAT_BLOB 06-13 12:24:56.038 9817 9833 D tango-hal: tango-camera.cpp:isFormatSupported():411 Supported format on camera 3: id=0x20363159 name:HAL_PIXEL_FORMAT_Y16 06-13 12:24:56.038 9817 9833 D tango-hal: tango-camera.cpp:isFormatSupported():411 Supported format on camera 3: id=0x20203859 name:HAL_PIXEL_FORMAT_Y8 06-13 12:24:56.038 9817 9833 D tango-hal: tango-utils.cpp:getProcessName():52 Process name: com.google.tango 06-13 12:24:56.041 9817 9833 D tango-hal: tango-camera.cpp:createStream():638 Connecting to camera 3 06-13 12:24:56.042 549 549 I CameraService: CameraService::connect call (PID 9817 "com.google.tango", camera ID 3) for HAL version default and Camera API version 2 06-13 12:24:56.042 549 549 E CameraService: chenyl6 CameraService::connect1, clientPackageName = com.google.tango 06-13 12:24:56.043 549 549 E CameraService: chenyl6 CameraService::connect, tango 06-13 12:24:56.045 549 549 I Camera2ClientBase: Camera 3: Opened. Client: com.google.tango (PID 9817, UID 10095) 06-13 12:24:56.112 9817 9833 D tango-hal: tango-camera.cpp:createStream():646 Connected to camera 3 06-13 12:24:56.112 9817 9833 D tango-hal: tango-camera.cpp:createStream():656 Creating 224 x 1557 stream for PointCloud camera with id 3
,
Jun 14 2017
#6: crack on! Re. > Unfortunately, depth raw data is now available (it's format HAL_PIXEL_FORMAT_Y16 fits well with existing depth capture). Did you mean, "fortunately"? :-) Also, re. the feature request, what would be exactly the changes you would like to see? (And if you file a stack-overflow question, please link it here for easy reference).
,
Jun 14 2017
> > Unfortunately, depth raw data is now available (it's format HAL_PIXEL_FORMAT_Y16 fits well with existing depth capture). > Did you mean, "fortunately"? :-) Yes, this typo is bad. Unfortunately, depth raw data is NOT available. Only Point Cloud data generated from it. >what would be exactly the changes you would like to see? Raw depth data provided through API. This is what I asked from tango-help@google.com. As they instructed, I'll contact Google+ community and StackOverflow (and link it here). To summarize: in the first patch, I will reconstruct depth buffer from point cloud data (224x172) (project 3D to 2D). In follow up patches provide tango fisheye (640x480) and tango RGB (1080p) camera capture.
,
Jul 14 2017
The patch https://crrev.com/2983473002/ implements depth 2D frame capture by projecting 3D point cloud to color-frame-aligned depth frame. It took some time to optimize the "conversion" (projecting 3D point cloud to 2D frame) takes about 0.9ms (900microseconds) on Lenovo Phab 2 Pro. As the point cloud is produced with rate < 5 frames per second, this is acceptable cost (pays of later when uploading to GPU anyway). Attached mp4 is a demo on Lenovo Phab 2 Pro. I'll start working on hand gesture recognition demo, similar to typing in the air demo: https://01org.github.io/depth-camera-web-demo/typing_in_the_air/doc/tutorial.html.
,
Jul 23 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/81c548a0745d866ec8fd0d30b686dc9d95ae30fc commit 81c548a0745d866ec8fd0d30b686dc9d95ae30fc Author: Aleksandar Stojiljkovic <aleksandar.stojiljkovic@intel.com> Date: Sun Jul 23 22:41:27 2017 VideoCaptureDeviceTest: skip the test for Tango devices Following the same logic about main thread occupied by the test, as for legacy/deporecated devices, disabling the tests for Tango devices. The behavior is changed so that the tests are run if there is a device that is not legacy or Tango device. This way on tango devices the tests are still run on default camera. This patch is split from https://codereview.chromium.org/2983473002/. BUG= 674440 Change-Id: Ibeb9ee5afce470ea7b801bcb84197fe32d2ae191 Reviewed-on: https://chromium-review.googlesource.com/573060 Commit-Queue: Aleksandar Stojiljkovic <aleksandar.stojiljkovic@intel.com> Reviewed-by: Miguel Casas <mcasas@chromium.org> Cr-Commit-Position: refs/heads/master@{#488897} [modify] https://crrev.com/81c548a0745d866ec8fd0d30b686dc9d95ae30fc/media/capture/video/video_capture_device_unittest.cc
,
Aug 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/34373da996559b824ee6abb84208f32f7095b460 commit 34373da996559b824ee6abb84208f32f7095b460 Author: Aleksandar Stojiljkovic <aleksandar.stojiljkovic@intel.com> Date: Mon Aug 14 19:52:58 2017 VideoCaptureDeviceAndroid: extract common code to methods. Skip Android420ToI420 conversion if the frame is to be throttled. This patch is a split of https://codereview.chromium.org/2983473002/. BUG= 674440 Change-Id: I3fe2c2dbbe5aa10b1a2bf113b7ff83987c4a6336 Reviewed-on: https://chromium-review.googlesource.com/573040 Reviewed-by: Emircan Uysaler <emircan@chromium.org> Commit-Queue: Aleksandar Stojiljkovic <aleksandar.stojiljkovic@intel.com> Cr-Commit-Position: refs/heads/master@{#494143} [modify] https://crrev.com/34373da996559b824ee6abb84208f32f7095b460/media/capture/video/android/video_capture_device_android.cc [modify] https://crrev.com/34373da996559b824ee6abb84208f32f7095b460/media/capture/video/android/video_capture_device_android.h
,
Dec 19 2017
As ARCore got introduced as Tango replacement soon after the commits and now Tango got disconnected [1], stopping work on this. Need to revert submitted CLs and check ARCore support later. [1] https://twitter.com/projecttango/status/941730801791549440
,
Mar 1 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/220e6afba86d8021b397b62a3987dc8c585331d0 commit 220e6afba86d8021b397b62a3987dc8c585331d0 Author: Aleksandar Stojiljkovic <aleksandar.stojiljkovic@intel.com> Date: Thu Mar 01 11:19:52 2018 Remove Android Tango related code. BUG= 674440 Change-Id: Ie33ae36877d56987bb3b1ea366558a6afb677a8f Reviewed-on: https://chromium-review.googlesource.com/940301 Reviewed-by: Miguel Casas <mcasas@chromium.org> Reviewed-by: Guido Urdaneta <guidou@chromium.org> Reviewed-by: Kinuko Yasuda <kinuko@chromium.org> Commit-Queue: Aleksandar Stojiljkovic <aleksandar.stojiljkovic@intel.com> Cr-Commit-Position: refs/heads/master@{#540126} [modify] https://crrev.com/220e6afba86d8021b397b62a3987dc8c585331d0/content/browser/media/media_internals_unittest.cc [modify] https://crrev.com/220e6afba86d8021b397b62a3987dc8c585331d0/media/capture/mojo/video_capture_types.mojom [modify] https://crrev.com/220e6afba86d8021b397b62a3987dc8c585331d0/media/capture/mojo/video_capture_types_struct_traits.cc [modify] https://crrev.com/220e6afba86d8021b397b62a3987dc8c585331d0/media/capture/video/video_capture_device_descriptor.cc [modify] https://crrev.com/220e6afba86d8021b397b62a3987dc8c585331d0/media/capture/video/video_capture_device_descriptor.h [modify] https://crrev.com/220e6afba86d8021b397b62a3987dc8c585331d0/media/capture/video/video_capture_device_unittest.cc
,
Mar 1 2018
Cleaned up. Let’s revisit when similar capability is available from ARCore phones. |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by aleksand...@intel.com
, Dec 15 2016