New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 327425 link

Starred by 15 users

Issue metadata

Status: Fixed
Owner:
Closed: Sep 2014
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: Bug



Sign in to add a comment

Keep <audio> currentTime advancing by dead reckoning based on last internal ipc itme

Project Member Reported by esprehn@chromium.org, Dec 10 2013

Issue description

<audio> doesn't report the current play time on Android making it impossible to build music player apps with an accurate display of the time and also impossible to build apps where the audio is synced against a running animation.

For example if you wanted to create a lyrics page like SoundHound has where the lyrics scroll by as the music plays.

Apparently it only happens once every 250ms to avoid sending too many IPCs to the render process, but we already deal with so many IPCs it should be fine to send these sync messages far more often, probably once per frame.
 

Comment 1 by darin@chromium.org, Dec 11 2013

Is this a value that is polled by the page? If so, then we could probably use the same mechanism we use to poll efficiently across processes for gamepad data.

Comment 2 by darin@chromium.org, Dec 11 2013

Cc: timvolod...@chromium.org
We also use that polling mechanism for accessing orientation and motion data. +timvolodine

Comment 3 by qin...@chromium.org, Dec 11 2013

the android <audio> implementation uses the platform mediaplayer.
The android mediaplayer doesn't have any callbacks that are invoked once a frame is decoded: http://developer.android.com/reference/android/media/MediaPlayer.html

Besides having no API support, power consumption is another concern on having too many IPCs
Why not schedule an update every time the currentTime is read? That way we're only sending the IPCs as frequently as you're reading them.

Comment 5 Deleted

Comment 6 by qin...@chromium.org, Dec 12 2013

The call to currentTime() is more frequent than 4 times a second.

And even if we send an IPC each time currentTime() is called, it does not solve the issue you mentioned. There is a round trip delay for the IPCs, so the mediaplayer's timestamp is actually ahead of the timestamp the renderer gets. So you won't get the accurate timestamp by simply using IPCs. 

Comment 7 Deleted

Comment 8 Deleted

Comment 9 Deleted

Cc: tomhud...@chromium.org
Dead reckoning strikes me as a reasonable fallback?

Possible problem: does the javascript API require that the value we report be strictly monotonic?

As far as I know this used to work on Chrome 31 for Android.

These demos rely on audio.currentTime updating at 60fps:

http://www.mrdoob.com/files/temp/xplsv_obsidian/
http://www.mrdoob.com/files/temp/xplsv_wlz/
http://kile.stravaganza.org/lab/thiswayjs/
http://xplsv.com/prods/demos/xplsv_orsotheysay/

They work fine on Firefox for Android.
Labels: Cr-Internals-Media
qinmin: I don't believe the request is for precision accuracy (there'll always be a bit of delay whether it's in inside of the media engine, audio stack, hardware buffers, etc...) but rather a more continuously updated value

tomhudson: yeah some form of interpolation would work, although it's best to try to keep things monotonic

relevant spec bits:
http://www.whatwg.org/specs/web-apps/current-work/multipage/the-video-element.html#dom-media-currenttime
I'm also a bit suspicious of the "too many IPCs" claim ... if you look at any of the links in #11 I suspect sending a more-periodic IPC to fetch the current playback position would be one of the less battery consuming operations

Comment 14 by phil...@opera.com, Feb 6 2014

The spec has a somewhat new concept called official playback position:

http://www.whatwg.org/specs/web-apps/current-work/multipage/the-video-element.html#official-playback-position

I was planning to implement that once support for "await a stable state" has landed. With that in place, the platform will only be queried once per script execution, so maybe we could then try just removing whatever throttling is in place?
Should we consider updating the information once per animation frame?
That'd be ideal.
#15: for video, there is a onFrameAvailable() callback for once each frame is decoded. However, audio doesn't have animation frames


@qinmin: We're refactoring the entire engine to run on animation frames.  I'm sure we can teach the audio system to report this information once per frame.
Labels: -Cr-Internals-Media Cr-Internals-Media-Audio
What's the status of this?
from the media side I can tell you there hasn't been progress

abarth: how close are we to refactoring the engine to run on animations frames?
> how close are we to refactoring the engine to run on animations frames?

Already done.
sweet

qinmin: any thoughts?
Seems there are lots of things:
1. For embedded video, we can report the frame through onFrameAvailable().
2. For audio, both MSE and non-MSE case, there is no API callback from the system to report that a frame is rendered. We need to actively pull the current time from AudioTrack or MediaPlayer. However, we won't be able to get the precision at a frame level.
Any updates on this one?

Comment 26 Deleted

Owner: qin...@chromium.org
Status: Assigned
Min and I chatted birefly and we're going to try to fix this by dead reackoning. From a platform point of view, currentTime will appear to advance. It may be occasionally slightly wrong, but will self-correct every 250ms, but it will always advance. Internally, this will be done by dead reckoning the time based on the assumption that the player is still playing.
Why can we not just send IPCs every 16ms? 250ms is a really long time.
we can do both the interpolation and the 16ms internal
sgtm, super excited to see this fixed!
Project Member

Comment 31 by bugdroid1@chromium.org, Sep 12 2014

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4360c7c14b1ca7e4509015166d44d1e1ad17b4bd

commit 4360c7c14b1ca7e4509015166d44d1e1ad17b4bd
Author: qinmin <qinmin@chromium.org>
Date: Fri Sep 12 17:56:18 2014

Provide fine grained media playback time thru interpolation

This CL includes the following changes:
1. Reduces the interval of media time update msg to 16 ms
2. Use interpolation to calculate the current playback time, and guarantee that time is always incrementing

BUG= 327425 

Review URL: https://codereview.chromium.org/545993002

Cr-Commit-Position: refs/heads/master@{#294612}

[modify] https://chromium.googlesource.com/chromium/src.git/+/4360c7c14b1ca7e4509015166d44d1e1ad17b4bd/content/browser/media/android/browser_media_player_manager.cc
[modify] https://chromium.googlesource.com/chromium/src.git/+/4360c7c14b1ca7e4509015166d44d1e1ad17b4bd/content/browser/media/android/browser_media_player_manager.h
[modify] https://chromium.googlesource.com/chromium/src.git/+/4360c7c14b1ca7e4509015166d44d1e1ad17b4bd/content/common/media/media_player_messages_android.h
[modify] https://chromium.googlesource.com/chromium/src.git/+/4360c7c14b1ca7e4509015166d44d1e1ad17b4bd/content/renderer/media/android/renderer_media_player_manager.cc
[modify] https://chromium.googlesource.com/chromium/src.git/+/4360c7c14b1ca7e4509015166d44d1e1ad17b4bd/content/renderer/media/android/renderer_media_player_manager.h
[modify] https://chromium.googlesource.com/chromium/src.git/+/4360c7c14b1ca7e4509015166d44d1e1ad17b4bd/content/renderer/media/android/webmediaplayer_android.cc
[modify] https://chromium.googlesource.com/chromium/src.git/+/4360c7c14b1ca7e4509015166d44d1e1ad17b4bd/content/renderer/media/android/webmediaplayer_android.h
[add] https://chromium.googlesource.com/chromium/src.git/+/4360c7c14b1ca7e4509015166d44d1e1ad17b4bd/media/base/android/media_common_android.h
[modify] https://chromium.googlesource.com/chromium/src.git/+/4360c7c14b1ca7e4509015166d44d1e1ad17b4bd/media/base/android/media_player_bridge.cc
[modify] https://chromium.googlesource.com/chromium/src.git/+/4360c7c14b1ca7e4509015166d44d1e1ad17b4bd/media/base/android/media_player_manager.h
[modify] https://chromium.googlesource.com/chromium/src.git/+/4360c7c14b1ca7e4509015166d44d1e1ad17b4bd/media/base/android/media_source_player.cc
[modify] https://chromium.googlesource.com/chromium/src.git/+/4360c7c14b1ca7e4509015166d44d1e1ad17b4bd/media/base/android/media_source_player_unittest.cc
[modify] https://chromium.googlesource.com/chromium/src.git/+/4360c7c14b1ca7e4509015166d44d1e1ad17b4bd/media/media.gyp

Status: Fixed
I tried to report a new bug for this, but appspot is broken and constantly giving me 400s.

On Chrome 46 and 47 (and really, since Chrome 40 - this is over a year old) for Android, this is not fixed. The currentTime is always 0. timeupdate events never fire (in fact, the 'stalled' event fires, even though the file is still playing perfectly fine), and the 'ended' event never fires. I can neither tell my users the position in the file they're at, nor can I correctly determine when the file is done in order to switch to the next file.

Chrome 38 still works (the browser bundled with Samsung 5.1.1 devices is based on 38). Desktop Chrome works fine, as do other browsers including Amazon's silk and the apk files they generate.
Components: -Blink>Audio Blink>Media>Audio
Renaming Blink>Audio to Blink>Media>Audio for better characterization

Sign in to add a comment