New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 601657 link

Starred by 4 users

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug-Regression

Blocking:
issue 587553
issue 597034



Sign in to add a comment

Video capture timestamps are not valid TimeTicks values, causing problems downstream in the media stack

Project Member Reported by m...@chromium.org, Apr 8 2016

Issue description

Bugs  529949  and  549744  were resolved by making implementation changes that broke the video frame "reference clock" timestamps.  These changes wrapped the timestamps from the capture device into fake base::TimeTicks values.  This is an abstraction violation, since TimeTicks values are always interpreted relative to the system clock.

 Bug 597034 , and likely  bug 587553 , track problems caused by the bad timestamps.  In  bug 597034 , a workaround was needed because the video frame reference timestamps were bogus.  That was a band-aid solution to the real underlying problem.

The correct solution to the original problem is that BOTH the relative timestamp (from the capture device driver) and the reference clock timestamp (TimeTicks::Now()) need to be passed through the video stack, as each is necessary for different purposes:

1. The capture device timestamps (which should be set in media::VideoFrame::set_timestamp()) are needed to communicate the media duration/rate of the video frames for smooth playout.

2. The reference timestamps (from TimeTicks::Now()) are needed for the purposes of resolving Audio/Video synchronization via a common reference clock.  The reference timestamp is allowed to be a little "noisy" because it is only being used to align the playout of the separate audio and video streams.  In other words, it is used as a target or an "ideal" playout time, but never as an exact presentation time.

 

Comment 1 by m...@chromium.org, Apr 8 2016

Blocking: 587553
Cc: srnarayanan@chromium.org mflodman@chromium.org
M51 is scheduled to branch later today (https://www.chromium.org/developers/calendar). Are you on track to land a fix today?
Labels: -M-51 M-52
This will not be done for 51.
Cc: sprang@chromium.org

Comment 5 by holmer@chromium.org, Apr 25 2016

Cc: holmer@chromium.org nisse@chromium.org
+nisse who has also been looking at this.
Labels: -Pri-1 -M-52 Pri-2
Lower the priority as currently nothing is broken.

Comment 7 by m...@chromium.org, May 16 2016

qiangchen: I disagree. It is definitely broken when you consider devs have checked-in hacks more than once now to mask the root problem. Furthermore, the hacks are masking problems, such as  bug 587553 . Tons of developer time has been wasted and will continue to be wasted because the problems cannot be easily diagnosed. This is technical debt that needs to be paid off.

mcasas/dalecurtis: This is probably something your teams should own. The longer we wait on this, the more the "interest on the tech debt" will compound, making this harder to fix down-the-road.
Defer to niklase@ for ownership; this seems like a capture path issue. Once 1,2 are done as miu@ indicates, the only part of our pipelines that overlap is VideoRendererAlgorithm, and it will automatically do the right thing.
Labels: -Pri-2 Pri-1
Then let me bring back to P1
Project Member

Comment 10 by bugdroid1@chromium.org, May 25 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9

commit ed732c9c81df6d8fd4cfb4105ad154f52d458cd9
Author: qiangchen <qiangchen@chromium.org>
Date: Wed May 25 22:51:45 2016

Decouple capture timestamp and reference time

In this CL, we changed the signature of
VideoCaptureDevice::OnIncomingCapturedData
to take one more parameter called reference_time.

Background:
Previously, we have only timestamp parameter, and it actually
plays the role as timestamp and reference_time.
Here we define timestamp as the ideal time span between
the current frame and the first frame in the stream.
Define reference time as the system clock time when the frame
gets generated.

Ideally, timestamp can be derived from reference_time.
However, in reality, especially in a multi process application
like Chrome, system clock read has noise on the order of 5ms.
To overcome that, we found the original timestamp from camera
driver is much smoother. But the defect is that the camera
clock would have a drift with respect to the system clock.

Thus a perfect solution would be that we take system time for
reference time, which is used for AV sync; we stream timestamp
to the remote side, which will make the rendering smooth.

This CL is preliminary for the complete fix.

BUG= 601657 

Review-Url: https://codereview.chromium.org/1983193002
Cr-Commit-Position: refs/heads/master@{#396021}

[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/content/browser/media/capture/desktop_capture_device.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/content/browser/media/capture/desktop_capture_device_aura_unittest.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/content/browser/media/capture/desktop_capture_device_unittest.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/content/browser/media/capture/web_contents_video_capture_device_unittest.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/content/browser/renderer_host/media/video_capture_device_client.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/content/browser/renderer_host/media/video_capture_device_client.h
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/content/browser/renderer_host/media/video_capture_device_client_unittest.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/android/video_capture_device_android.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/android/video_capture_device_android.h
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/fake_video_capture_device.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/fake_video_capture_device.h
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/fake_video_capture_device_unittest.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/file_video_capture_device.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/file_video_capture_device.h
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/linux/v4l2_capture_delegate.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/mac/video_capture_device_decklink_mac.h
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/mac/video_capture_device_decklink_mac.mm
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/mac/video_capture_device_mac.h
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/mac/video_capture_device_mac.mm
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/video_capture_device.h
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/video_capture_device_unittest.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/win/sink_filter_observer_win.h
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/win/sink_input_pin_win.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/win/video_capture_device_mf_win.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/win/video_capture_device_mf_win.h
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/win/video_capture_device_win.cc
[modify] https://crrev.com/ed732c9c81df6d8fd4cfb4105ad154f52d458cd9/media/capture/video/win/video_capture_device_win.h

Project Member

Comment 11 by bugdroid1@chromium.org, Jun 10 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4387613998060e4d1691434d0be687bf012596ce

commit 4387613998060e4d1691434d0be687bf012596ce
Author: qiangchen <qiangchen@chromium.org>
Date: Fri Jun 10 22:45:48 2016

Decouple capture timestamp and reference time

In this CL, we make the captured VideoFrame to take capture device
timestamp; and take system clock reference time.

Background:
Previously, we have only timestamp parameter, and it actually
plays the role as timestamp and reference_time.
Here we define timestamp as the ideal time span between
the current frame and the first frame in the stream.
Define reference time as the system clock time when the frame
gets generated.

Ideally, timestamp can be derived from reference_time.
However, in reality, especially in a multi process application
like Chrome, system clock read has noise on the order of 5ms.
To overcome that, we found the original timestamp from camera
driver is much smoother. But the defect is that the camera
clock would have a drift with respect to the system clock.

Thus a perfect solution would be that we take system time for
reference time, which is used for AV sync; we stream timestamp
to the remote side, which will make the rendering smooth.

This CL is complete for the complete fix.
BUG= 601657 
TBR=palmer@chromium.org

Review-Url: https://codereview.chromium.org/2045813003
Cr-Commit-Position: refs/heads/master@{#399304}

[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/media/capture/desktop_capture_device_aura_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/media/capture/desktop_capture_device_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/media/capture/web_contents_video_capture_device_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_controller.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_controller.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_controller_event_handler.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_controller_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_device_client.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_device_client.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_device_client_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_gpu_jpeg_decoder.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_gpu_jpeg_decoder.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_host.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_host.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_manager_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/common/media/video_capture_messages.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/renderer/media/video_capture_impl.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/renderer/media/video_capture_impl.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/renderer/media/video_capture_impl_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/renderer/media/video_capture_message_filter.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/renderer/media/video_capture_message_filter_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/media/capture/content/thread_safe_capture_oracle.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/media/capture/content/thread_safe_capture_oracle.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/media/capture/video/fake_video_capture_device_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/media/capture/video/video_capture_device.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/media/capture/video/video_capture_device_unittest.cc

Status: Fixed (was: Assigned)
Project Member

Comment 13 by bugdroid1@chromium.org, Jun 15 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4387613998060e4d1691434d0be687bf012596ce

commit 4387613998060e4d1691434d0be687bf012596ce
Author: qiangchen <qiangchen@chromium.org>
Date: Fri Jun 10 22:45:48 2016

Decouple capture timestamp and reference time

In this CL, we make the captured VideoFrame to take capture device
timestamp; and take system clock reference time.

Background:
Previously, we have only timestamp parameter, and it actually
plays the role as timestamp and reference_time.
Here we define timestamp as the ideal time span between
the current frame and the first frame in the stream.
Define reference time as the system clock time when the frame
gets generated.

Ideally, timestamp can be derived from reference_time.
However, in reality, especially in a multi process application
like Chrome, system clock read has noise on the order of 5ms.
To overcome that, we found the original timestamp from camera
driver is much smoother. But the defect is that the camera
clock would have a drift with respect to the system clock.

Thus a perfect solution would be that we take system time for
reference time, which is used for AV sync; we stream timestamp
to the remote side, which will make the rendering smooth.

This CL is complete for the complete fix.
BUG= 601657 
TBR=palmer@chromium.org

Review-Url: https://codereview.chromium.org/2045813003
Cr-Commit-Position: refs/heads/master@{#399304}

[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/media/capture/desktop_capture_device_aura_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/media/capture/desktop_capture_device_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/media/capture/web_contents_video_capture_device_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_controller.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_controller.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_controller_event_handler.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_controller_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_device_client.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_device_client.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_device_client_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_gpu_jpeg_decoder.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_gpu_jpeg_decoder.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_host.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_host.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/browser/renderer_host/media/video_capture_manager_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/common/media/video_capture_messages.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/renderer/media/video_capture_impl.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/renderer/media/video_capture_impl.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/renderer/media/video_capture_impl_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/renderer/media/video_capture_message_filter.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/content/renderer/media/video_capture_message_filter_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/media/capture/content/thread_safe_capture_oracle.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/media/capture/content/thread_safe_capture_oracle.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/media/capture/video/fake_video_capture_device_unittest.cc
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/media/capture/video/video_capture_device.h
[modify] https://crrev.com/4387613998060e4d1691434d0be687bf012596ce/media/capture/video/video_capture_device_unittest.cc

Sign in to add a comment