New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 791658 link

Starred by 31 users

Issue metadata

Status: WontFix
Owner: ----
Closed: Dec 13
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android , Windows , Chrome , Mac
Pri: 3
Type: Feature



Sign in to add a comment

Support <img src="*.mp4">

Project Member Reported by addyo@chromium.org, Dec 4 2017

Issue description

Chrome Version: 62.0.3202.94 

What steps will reproduce the problem? 

(1) Attempt to supply a video `src` to the ` <img>` tag 
(2) Chrome will fetch the resource but does not support video supplied as an `<img src>`
(3) What is the expected result? We're able to interpret and correctly render video sources supplied to the `<img>` tag. 

What happens instead? 

This is unsupported in Chrome at present. Safari Tech Preview has implemented support for `<img src="animation.mp4">` with some details in https://calendar.perfplanet.com/2017/animated-gif-without-the-gif/. 

Notes from the post: 

- GIFs are awesome but terrible for quality and performance 
- Replacing GIFs with `<video>` is better but has perf. drawbacks: not preloaded, uses range requests 
- Now you can `<img src=".mp4">`s in Safari Technology Preview 
- Early results show mp4s in `<img>` tags display 20x faster and decode 7x faster than the GIF equivalent – in addition to being 1/14th the file size!
- Background CSS video & Responsive Video can now be a “thing”. 
- Finally cinemagraphs without the downsides of GIFs!
- The above post is 46MB on Chrome but 2MB in Safari TP

Developers appear interested in this feature:

- AMP TL: https://twitter.com/cramforce/status/937746796951957504
- Author of ImageOptim: https://twitter.com/kornelski/status/937755768446771200
 
support.png
627 KB View Download
Components: Internals>Images>Codecs
Status: Available (was: Untriaged)
Where does mp4 live from a decoding standpoint? Could Blink just plumb it in as a multi-frame image source and let the decoders handle the rest?

Comment 2 by addyo@chromium.org, Dec 4 2017

Cc: y...@yoav.ws
Labels: -Type-Bug Type-Feature
Cc: mlamouri@chromium.org
... why? I thought we discussed this with Apple and agreed that <video muted> was sufficient for all use cases. Also, what are they doing about the async nature of <video> vs <img>?
Components: Internals>Media>Video
https://groups.google.com/a/chromium.org/d/msg/blink-dev/9rNd-vOH918/qMz2TDXTim8J is the earliest discussion of this I remember.

Comment 7 by y...@yoav.ws, Dec 4 2017

The attached post details the use-cases:
* Optimization services can transcode animated GIFs to more efficient video formats without modifying the HTML.
* `<img>` is discovered earlier and downloads in a more efficient manner than video.
* Animated background images are becoming a thing. Videos in image contexts solves that without CSS changes.

I previously presented & discussed this at BlinkOn 7. I failed to find someone that knows the systems involved end-to-end though.
#1 is reasonable; #2 and #3 are questionable. E.g.,
- Why couldn't #2 be true for <video>?
- Sites are already using <video> for background.

As far as who knows what, for the <video> side of things, that would be mlamouri@ and I. The <img> side isn't well known to us though.

I do know img has lots of idiosyncrasies around synchronous loading that video does not provide. I.e., what's the fallback path for decode errors or selection of video/mp4 with unsupported codecs? For video this is an asynchronous process. What's the guarantee for copying to canvas?

Mounir and I are happy to chat with someone from the <img> side of things to at least flesh out the technicalities here.
Cc: vmp...@chromium.org chrishtr@chromium.org
https://docs.google.com/document/d/1VvrEGAjcnto5egR_rnX7PR5ymqWayMNRPYSbCf1bC4I/edit# is probably relevant to the sync/async issue here. I.e., video in img would likely need to be async always.
Cc: -vmp...@chromium.org -y...@yoav.ws jimbankoski@chromium.org jaikk@chromium.org
Cc: vmp...@chromium.org y...@yoav.ws
I'm not fully convinced that #1 is a strong long term use case as we should expect people to drop GIFs for <video muted>. It's only useful for websites that want to keep animated images and have a optimisation service that knows better.

I find the CSS use cases more interesting (as in allowing `background-image: foo.webm`).


FWIW, most of the points in the blog post would apply to <video muted>.

Comment 13 by y...@yoav.ws, Dec 5 2017

#1 - Do we have usage stats for animated GIFs? Has usage significantly decreased since the introduction of `<video muted>` support? I suspect the long tail is long and hard to deal with due to legacy software.

#2 - preloading is not necessarily appropriate for videos, but is appropriate for cinemagraphs. For range requests, we'd also need to distinguish cinemagraphs from long-form videos. Can we assume that all muted videos are short cinemagraphs?

#3 - Sites are using `<video>` to emulate background, but it's still part of the content, which is clumsy and probably doesn't support all the use cases people want to target.

I'm happy to discuss the `<img>` side of things. It's been a while since I touched that code, so would probably be good to have more recent contributors in the discussion as well.

Cc: khushals...@chromium.org

Comment 15 by sdy@chromium.org, Dec 5 2017

Cc: sdy@chromium.org
Re. #1:

Lots of services support embedded images but not embedded videos. Big example: Gmail* (and other mail clients). It's not just for optimizing proxies: every hotlinked gif from Imgur, Giphy, Gfycat, etc. could immediately start being served as mp4/webm if Chrome sends the right Accept header.

I investigated this a while back and for random images on a few gif hosting sites, the videos they (already) serve on-site are 12-18x smaller than the hotlinkable gifs. The bandwidth saved here could be incredible on both sides.

* Gmail might require a server-side change because it proxies images, but you get the idea: a _lot_ more websites allow, and currently have, hotlinked gifs than allow hotlinked videos with the right set of attributes. (I think you need <video autoplay loop muted src=…> to get the same behavior, right? That's significantly weirder/less likely for every site to get right vs. <img src=…>.)
> Lots of services support embedded images but not embedded videos

As you pointed out, they will need to make changes to accept videos at which point, they could as well use <video>. Using <video> will offer benefits like being able to play/pause.

> Imgur, Giphy, Gfycat, etc

"GIF" websites have all been serving videos for a long time. They might even sometimes ask. Twitter is another example. A lot of these used to use GIF on mobile and most switched to <video muted> after it was autoplay'able on mobile.

> I think you need <video autoplay loop muted src=…> to get the same behavior, right?

If you want literally the same behaviour as a GIF, yes. If you just want to be allowed to play it, <video muted> is enough or <video> without an audio track will work on Safari (and soon in Chrome).

Using <video> over <img> offers more features and more control to the website and I'm not sure there is a compelling argument for a website to use <img> over <video> if they had to make the decision today. It's great to see that Apple is allowing to play videos in <img> and I suggest we see how that goes. If they see strong success maybe it means it's something worth doing but otherwise I'm not sure we should offer a footgun.

Comment 17 by roh0...@gmail.com, Dec 6 2017

I think Apple introduced this feature for their HEIF format which is basically MP4 video.

Why not forget WEBP and jump forward into the world where every image interchangeable with 1 (or more) frame H.264/WebM/H.265 video?

Also picture present timing is not critically important for those use cases (don't require async).

Comment 18 by co...@bendell.ca, Dec 6 2017

Additionally, <img src=mp4> offers a hardware decoding. Since most mobile devices are equiped with h264 hardware decode, this would allow the image decode and resize to move off the cpu and improve battery and cpu.

This would be applicable for single frame h264 (and hevc) and multi frame.

A quick example of single frame mp4 on Safari Tech Preview:
https://colinbendell.github.io/webperf/img-mp4/index.html
https://colinbendell.github.io/webperf/img-mp4/index_hevc.html

Comment 19 by co...@bendell.ca, Dec 6 2017

Minor correction for compatibility on ios, I need playsinline: 
<video playsinline autoplay loop muted src=…>

It's clear that the use case for Cinemagraphs is very different than video. The user engagement is different: For example, a videos shouldn't restart if you scroll the video out of the viewport and then back because it was unloaded for memory concerns, where as a cinemagraph could without worry. Same thing applies for lazy loading, responsive images (srcset and sizes). For the developer, using <video> is not as natural or convenient for cinemagraph use cases.
+1 to the comment that Apple did this to support HEIF; that was my conclusion as well.

@colin: We already use hardware accelerated JPEG decoding on supported platforms, so <img> already benefits from that in some cases.
re #20, dalecurtis@, I'm curious where the hardware path for JPEG decoding is. I wasn't aware that we had this path, and would like to look into it a bit.

Comment 22 by co...@bendell.ca, Dec 6 2017

is there a way we could compare the pervasiveness of h264 hardware decode v. jpeg hardware decode? I thought only a very few vendors actually did jpeg hardware decode...
@ericrk, https://cs.chromium.org/chromium/src/media/video/jpeg_decode_accelerator.h - I just know of it, not how it's used or where though.

@colin, I don't expect it's as pervasive as H264 decoding, so you're correct that large images would benefit from having hardware decoding more regularly with 1-frame h264/vp9 files.
colin@, I'm not sure hardware decoding is an argument for enabling video in <img> as one can simply use <video> for this.

Comment 25 by co...@bendell.ca, Dec 6 2017

HW decode is an argument for third party services that want to use h264 (single frame) for images where the content is clearly not <video>. This is a flywheel/webp or cdn like play but leverages hw decoding for images without mutating markup. 

To be clear, this is adjacent to the cinemagraph use case where the apng/agif would be served as a multi-frame h264 video.

Comment 26 by hubbe@chromium.org, May 24 2018

Cc: hubbe@chromium.org

Comment 27 by hubbe@chromium.org, May 24 2018

Does anybody have a link to the webkit implementation of this?

For kicks (and because Dale asked me to), here is a prototype implementation image decoder that can decode video:

https://chromium-review.googlesource.com/#/c/chromium/src/+/1081272 

Biggest issue is probably that that DeferredImageDecoder assumes that the number of frames is easy to know, while video formats don't always encode that number explicitly anywhere.

However, I'm thinking that a better implementation might be to build a subclass of blink::Image which implements video-based images.

Interesting, thanks for the prototype. Some notes:
- You can use any software media::VideoDecoder without modification if we did a refactoring where we change the responsibility of using media::BindToCurrentLoop to caller instead of callee - this is something xhwang@ has wanted for a while and would also simplify OffloadingVideoDecoder. But if we're only ever going to allow AomVideoDecoder here, there may be no need to go down that path. 
- Hardware media::VideoDecoders will need the ability to return frames asynchronously.
- We can refactor AudioFileReader into a general audio+video synchronous demuxer instead of crafting a new one like you have. This is already done kind of for a few tests in a hacky fashion.
- Since we're thinking about using this for AV1 images, we could require that whatever container that is always includes a total frame count.
^^^^ - jaik@ :)

This implementation will use quite a bit of memory for long videos, but I guess that's already true of other long animated images. If a subclass of blink::Image can avoid that, it's probably a better long term implementation.


A subclass of blink::Image will have many potential advantages:

o It wouldn't have to know how many frames there are right away.
o It could use video layers for output
o It should be possible to use hardware video decoders. (But we may have to block the main thread if we get to the end of the data without having decoded the first frame yet.)

Of course, it's a fair amount more work to do as well.

I'm not sure if Image subclasses can be used for things like background-image  though.  (Can you set an SVG as a background?)

> I'm not sure if Image subclasses can be used for things like background-image  though.  (Can you set an SVG as a background?)

They can. (ImageResource/ImageResourceContent provides an Image and the upper layers "don't care"[1].)

[1] Save for size negotiations, image orientation and other peculiarities... ;-)
I've implemented a new prototype based subclassing blink::Image:

  https://chromium-review.googlesource.com/c/chromium/src/+/1222199
  
By doing this, we can re-use the video image paint path, and the video decoders
are invoked asynchronously, which means that all the video decoders should work. This is particularly important on android, where AV1 and h264 are only available through MediaCodec.

The CL needs some tests and some memory management stuff, but generally speaking it works well enough that it could be cleaned up and checked in.
So I think it's time to decide if this is something we want or not.

Comment 34 by tito.bou...@gmail.com, Dec 13 (4 days ago)

We want it!!! good work hubbe. This will make site usable when they embed too many gifs on the same page for example in a chat where people is freely linking gifs just for fun, if theres anything else in the page it will be very slow because gifs make the browser "lag". So being able to change just the format of the file to videos without having to rewrite all the img tags would be just awesome. It will also save bandwidth! etc, please add this thanks

Comment 35 by hubbe@google.com, Dec 13 (3 days ago)

In offline discussion we decided that all existing use cases are already covered, either by using webp, or by using the <video> tag, so we decided to not move forward with implementing this at this time.

Comment 36 by dalecur...@chromium.org, Dec 13 (3 days ago)

Status: WontFix (was: Available)
Closing as WontFix per c#35, due to the following:
- The widespread adoption of WebP (addresses CDN use case)
- Forthcoming AV1 based image formats (ditto).
- Memory inefficiency with allowing arbitrary video in image.
- Most sites have already switched to <video muted> now that autoplay is allowed.

Comment 37 by co...@bendell.ca, Dec 14 (3 days ago)

This is an unfortunate decision. If I can appeal this decision with a few comments:
1. WebP animation is notably slower in decode and memory than mp4 decode in an <img> tag on safari. I have a rough test case setup here: https://colinbendell.github.io/webperf/animated-gif-decode/ . I don't have published longitudinal test results but will make this available in the next month.

2. There are many cases where CMS don't allow the use of different markup or legacy systems that limit the markup. Can you quantify how you arrived at "most sites have already switched"? I can show you many example sites that still use heavy amounts of animated gifs still. I'm concerned that this statement is a 1-percenter kind of statement. "Twitter and Facebook have switched, so therefore most sites have switched"

3. Forthcoming AV1 based image formats are not landed and AFAIK, there is still much debate on the efficiency of this approach since it heavily depends on hardware decode - and as there aren't yet any av1 consumer devices with hardware decode, this solution is 2-3 years away for wide spread appeal. H264 mp4s are here today.

4. Can you help me understand what the memory efficiency concern is specifically? Can safeties be put in place on the length of a given video? I'm sure these same problems exist with animated gifs or animated webps. How has safari solved this problem?


In short, what data would help change your mind on this problem?

Comment 38 by dalecur...@chromium.org, Today (81 minutes ago)

1. Yes, we agree that WebP isn't as good as a true video, but it should be good enough until more advanced image formats come along next year.

2. I'm including sites like gfycat, imgur, reddit, etc. A CMS that doesn't let you upload into a <video> tag seems unlikely to let you try to upload a video as an image. Do you have an example of where this works without modification today?

3. You generally do not want hardware decode at low resolutions. E.g., macOS won't even use the hardware decoder below 720p. On Android we found the limit to be ~360p, but it will vary heavily with the device and form factor.

4. <video> is pretty conservative with decoded video frames, only keeping ~4 around at any time, while the image decoder builds a cache of all available frames. Even the Safari implementation does this:
https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm#L588

A video would instead keep just the 4 frames and re-decode them as needed to prevent memory accumulation. It's certainly possible that more efficient mechanisms can be designed for image, but we'd rather encourage folks to just use <video muted> instead. WebP is a sufficient stop gap on the image side of things.

Sign in to add a comment