To clarify: we intend to implement this extension on all platforms, with Android, Mac and Windows being the first priorities. It will yield significant speedups for uses like 360 degree videos in WebVR.

This extension in particular will require a lot of platform-specific code, and likely some refactoring to share code between the media pipeline and WebGL implementation.

This is a tracking bug for the implementation.

I have created WIP patch for this issue < https://chromium-review.googlesource.com/#/c/chromium/src/+/749608>.

Byungseon (above) just nicely provided this pointer to their implementation of the earlier OES_texture_external extension to WebGL:
https://github.com/xingri/chr53/commits/master

It's nice and simple, but isn't easily portable.

@ken, thank you for the update.

James is going to take a crack at updating LG's old OES_texture_external extension and integrating it as WEBGL_video_texture on Android only at first, just to have some scaffolding in place.

Hi, James,
I'd like to updated the status a little bit.
I've got the initial performance data for this extension using sharing resource method. I've tested vp9/8k/60fps 360 video with polyfill mode on kabylake and coffeelake desktop with HD630 and UHD630 GPU. It seems it could achieve around 34% fps improvment on both platform.

I want to brief introduce my sharing resource strategy. The main change is to transfer a WebGL texture to skia as the render target, so the YUV->RGB color convertion draw could draw to this external texture. Skia provided API to wrap an external texture as render target but has color format limit, I think in WEBGL_video_texture it is not an issue. Do you or kbr have any feedback for this strategy ?

And if you want more data, I'm happy to collect them and offer your a more detailed table in google doc.

Shaobo: your resource sharing strategy sounds good; agreed that for now the color format limitations are not a problem.

Assigning this bug to you for the moment as you're actively working on this extension on Windows. When you're ready to share some CLs and test cases please tell us.

Thanks for the work so far. Please do send a CL implementing your strategy. We can commit it and do some more performance testing to see where the next bottlenecks are.

Hi Ken and James,
Thx for your guidance. I'll rebase my local patches and send them out ASAP.

For the Android side, I tried rebasing LG's OES_EGL_image_external implementation. https://chromium-review.googlesource.com/c/chromium/src/+/1109446

It can be tested here: https://yt-dash-mse-test.commondatastorage.googleapis.com/demo-player/dash-player-020416.html?manifest_url=assets/spherical_vp9.mpd&sphericaloestexture=true&disableScreenSizeCheck=true

It's just a starting point for an eventual Android implementation of WEBGL_video_texture.

I've uploaded skia WIP patch :https://skia-review.googlesource.com/c/skia/+/146695
and chromium WIP patch : https://chromium-review.googlesource.com/c/chromium/src/+/1170431

The test sample I used is uploaded to : https://drive.google.com/open?id=1tDuZXxE9AbQPwb37bmr33-bFOJ8cZXT2

For the sample, you can view XXX-360-video.html and can try WEBGL_video_texture extension by pass enableWebGLSharingTexture=1 in url

The following revision refers to this bug:
  https://skia.googlesource.com/skia/+/d2fcfb58131b05f83adb30c3e0283e6c52d187b3

commit d2fcfb58131b05f83adb30c3e0283e6c52d187b3
Author: Brian Salomon <bsalomon@google.com>
Date: Tue Sep 18 02:22:00 2018

WEBGL_video_texture : sharing texture prototype in skia

Bug: 776222

Change-Id: I84c81bdaa6dc0b5548f116a28c6319e601effb32
Reviewed-on: https://skia-review.googlesource.com/146695
Commit-Queue: Brian Salomon <bsalomon@google.com>
Reviewed-by: Brian Salomon <bsalomon@google.com>
Reviewed-by: Kenneth Russell <kbr@google.com>

[modify] https://crrev.com/d2fcfb58131b05f83adb30c3e0283e6c52d187b3/src/image/SkImage_Gpu.cpp
[modify] https://crrev.com/d2fcfb58131b05f83adb30c3e0283e6c52d187b3/src/image/SkImage.cpp
[modify] https://crrev.com/d2fcfb58131b05f83adb30c3e0283e6c52d187b3/gn/gm.gni
[modify] https://crrev.com/d2fcfb58131b05f83adb30c3e0283e6c52d187b3/gm/imagefromyuvtextures.cpp
[modify] https://crrev.com/d2fcfb58131b05f83adb30c3e0283e6c52d187b3/include/core/SkImage.h
[modify] https://crrev.com/d2fcfb58131b05f83adb30c3e0283e6c52d187b3/src/image/SkImage_Gpu.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/32f6f5d770371885940e154a61145bd875ea5174

commit 32f6f5d770371885940e154a61145bd875ea5174
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Tue Sep 18 15:37:00 2018

Roll src/third_party/skia 6bdfebeb1594..9ec77b5574b4 (16 commits)

https://skia.googlesource.com/skia.git/+log/6bdfebeb1594..9ec77b5574b4


git log 6bdfebeb1594..9ec77b5574b4 --date=short --no-merges --format='%ad %ae %s'
2018-09-18 mtklein@google.com Revert "implement SkPixmap::erase() with draws"
2018-09-18 caryclark@skia.org fix bookmaker
2018-09-18 skia-autoroll@skia-public.iam.gserviceaccount.com Roll third_party/externals/angle2 a49f60fee74d..95277a300f52 (3 commits)
2018-09-18 bsalomon@google.com WEBGL_video_texture : sharing texture prototype in skia
2018-09-17 mtklein@google.com Revert "implement SkPixmap::getPixel() with this->readPixels()."
2018-09-17 mtklein@google.com simplify src->dst matrix init
2018-09-17 herb@google.com Make atomic lists list for bitmaps and paths
2018-09-17 brianosman@google.com Revert "Add Short4 vertex attributes, and benchmark them as 4.12 colors"
2018-09-17 brianosman@google.com Add Short4 vertex attributes, and benchmark them as 4.12 colors
2018-09-17 bsalomon@google.com When merging into an already chained op update the head's op bounds.
2018-09-17 bungeman@google.com Use onOnceBeforeDraw instead of constructor for glyphs.
2018-09-17 kjlubick@google.com Make fuzz::next overloads more consistent
2018-09-17 skia-autoroll@skia-public.iam.gserviceaccount.com Roll third_party/externals/angle2 017bda429667..a49f60fee74d (3 commits)
2018-09-17 chinmaygarde@google.com Allow users of Skia to optionally link against X11 and GL on Linux.
2018-09-17 mtklein@google.com implement SkPixmap::getPixel() with this->readPixels().
2018-09-17 mtklein@google.com implement SkPixmap::erase() with draws


Created with:
  gclient setdep -r src/third_party/skia@9ec77b5574b4

The AutoRoll server is located here: https://autoroll.skia.org/r/skia-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_trusty_blink_rel;luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG=chromium:776222
TBR=reed@chromium.org

Change-Id: I36aaa492376dfc5ad8663c5d5f1b7cdc305148d2
Reviewed-on: https://chromium-review.googlesource.com/1230318
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#592049}
[modify] https://crrev.com/32f6f5d770371885940e154a61145bd875ea5174/DEPS

From Dan's input about "ProduceTextureDirectCHROMIUM" in https://chromium-review.googlesource.com/c/chromium/src/+/1235335/8/media/renderers/paint_canvas_video_renderer.cc#1120 : 
"I assume that all the relevant cases work (multi-texture, no fences), but I should warn that this approach has failed previously.

It was used here: https://chromium-review.googlesource.com/c/chromium/src/+/1244835/5/media/renderers/paint_canvas_video_renderer.cc#1004

And reverted because it wasn't always possible to execute the texture copy on Mac: https://chromium-review.googlesource.com/c/chromium/src/+/1251321.

The solution was to make a copy of the video texture in that case so that the copy could be done in |destination_gl|: https://chromium-review.googlesource.com/c/chromium/src/+/1258212/5/media/renderers/paint_canvas_video_renderer.cc#957"

We should pay attention when testing on multiple platforms.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a6e3f5c9a201be4be2a35f08f82670f404e041fa

commit a6e3f5c9a201be4be2a35f08f82670f404e041fa
Author: Yan,Shaobo <shaobo.yan@intel.com>
Date: Tue Nov 20 05:29:19 2018

Implement WEBGL_video_texture

This patch implement a WebGL draft extension: WEBGL_video_texture. This extension will bring a new type of texture for WebGL and can
share the video frame to WebGL texture directly. This should reduce on blit operation and should have performance improvement for
uploading video to texture.

Bug: 776222
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: Id052010bb726c328b0362c28d2486ce1986cc6e2
Reviewed-on: https://chromium-review.googlesource.com/c/1235335
Commit-Queue: Shaobo Yan <shaobo.yan@intel.com>
Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
Reviewed-by: Kentaro Hara <haraken@chromium.org>
Reviewed-by: Mounir Lamouri <mlamouri@chromium.org>
Reviewed-by: Dan Sanders <sandersd@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Kai Ninomiya <kainino@chromium.org>
Cr-Commit-Position: refs/heads/master@{#609604}
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/content/test/gpu/gpu_tests/webgl_conformance_integration_test.py
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/media/blink/webmediaplayer_impl.cc
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/media/blink/webmediaplayer_impl.h
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/media/renderers/paint_canvas_video_renderer.cc
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/media/renderers/paint_canvas_video_renderer.h
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/WebKit/LayoutTests/webexposed/global-interface-listing-expected.txt
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/public/platform/web_media_player.h
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/core/html/media/html_video_element.cc
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/core/html/media/html_video_element.h
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/modules_idl_files.gni
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/BUILD.gn
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl2_compute_rendering_context.cc
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl2_compute_rendering_context.h
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl2_rendering_context.cc
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl2_rendering_context.h
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl_extension_name.h
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl_rendering_context.cc
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl_rendering_context.h
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl_rendering_context_base.cc
[modify] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl_rendering_context_base.h
[add] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl_video_frame_info.cc
[add] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl_video_frame_info.h
[add] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl_video_frame_info.idl
[add] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl_video_texture.cc
[add] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl_video_texture.h
[add] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl_video_texture.idl
[add] https://crrev.com/a6e3f5c9a201be4be2a35f08f82670f404e041fa/third_party/blink/renderer/modules/webgl/webgl_video_texture_enum.h

This prototype remains a simple angle patch to introduce the special sampler type.
And there remains some reduce duplicated codes to do.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e

commit 22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e
Author: Robert Kroeger <rjkroege@chromium.org>
Date: Tue Nov 20 16:03:49 2018

Revert "Implement WEBGL_video_texture"

This reverts commit a6e3f5c9a201be4be2a35f08f82670f404e041fa.

Reason for revert: This change is on the blame list for a webgl conformance test failure in WebglConformance_conformance_uniforms_no_over_optimization_on_uniform_array_09 on Windows FYI. Reverting to see if this is so.

Original change's description:
> Implement WEBGL_video_texture
> 
> This patch implement a WebGL draft extension: WEBGL_video_texture. This extension will bring a new type of texture for WebGL and can
> share the video frame to WebGL texture directly. This should reduce on blit operation and should have performance improvement for
> uploading video to texture.
> 
> Bug: 776222
> Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
> Change-Id: Id052010bb726c328b0362c28d2486ce1986cc6e2
> Reviewed-on: https://chromium-review.googlesource.com/c/1235335
> Commit-Queue: Shaobo Yan <shaobo.yan@intel.com>
> Reviewed-by: Philip Jägenstedt <foolip@chromium.org>
> Reviewed-by: Kentaro Hara <haraken@chromium.org>
> Reviewed-by: Mounir Lamouri <mlamouri@chromium.org>
> Reviewed-by: Dan Sanders <sandersd@chromium.org>
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Reviewed-by: Kai Ninomiya <kainino@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#609604}

TBR=kbr@chromium.org,haraken@chromium.org,mlamouri@chromium.org,sandersd@chromium.org,shaobo.yan@intel.com,foolip@chromium.org,kainino@chromium.org

Change-Id: Ia843571180c67a343d6db2c89eac484faf306fb5
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 776222
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Reviewed-on: https://chromium-review.googlesource.com/c/1344251
Reviewed-by: Robert Kroeger <rjkroege@chromium.org>
Commit-Queue: Robert Kroeger <rjkroege@chromium.org>
Cr-Commit-Position: refs/heads/master@{#609720}
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/content/test/gpu/gpu_tests/webgl_conformance_integration_test.py
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/media/blink/webmediaplayer_impl.cc
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/media/blink/webmediaplayer_impl.h
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/media/renderers/paint_canvas_video_renderer.cc
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/media/renderers/paint_canvas_video_renderer.h
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/WebKit/LayoutTests/webexposed/global-interface-listing-expected.txt
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/blink/public/platform/web_media_player.h
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/blink/renderer/core/html/media/html_video_element.cc
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/blink/renderer/core/html/media/html_video_element.h
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/blink/renderer/modules/modules_idl_files.gni
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/blink/renderer/modules/webgl/BUILD.gn
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/blink/renderer/modules/webgl/webgl2_compute_rendering_context.cc
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/blink/renderer/modules/webgl/webgl2_compute_rendering_context.h
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/blink/renderer/modules/webgl/webgl2_rendering_context.cc
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/blink/renderer/modules/webgl/webgl2_rendering_context.h
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/blink/renderer/modules/webgl/webgl_extension_name.h
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/blink/renderer/modules/webgl/webgl_rendering_context.cc
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/blink/renderer/modules/webgl/webgl_rendering_context.h
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/blink/renderer/modules/webgl/webgl_rendering_context_base.cc
[modify] https://crrev.com/22bad1f9ca27aa1d38c2ff8d1fa08f63597f517e/third_party/blink/renderer/modules/webgl/webgl_rendering_context_base.h
[delete] https://crrev.com/0694f7c1f92d46137fe536f1fde23963fccac0b5/third_party/blink/renderer/modules/webgl/webgl_video_frame_info.cc
[delete] https://crrev.com/0694f7c1f92d46137fe536f1fde23963fccac0b5/third_party/blink/renderer/modules/webgl/webgl_video_frame_info.h
[delete] https://crrev.com/0694f7c1f92d46137fe536f1fde23963fccac0b5/third_party/blink/renderer/modules/webgl/webgl_video_frame_info.idl
[delete] https://crrev.com/0694f7c1f92d46137fe536f1fde23963fccac0b5/third_party/blink/renderer/modules/webgl/webgl_video_texture.cc
[delete] https://crrev.com/0694f7c1f92d46137fe536f1fde23963fccac0b5/third_party/blink/renderer/modules/webgl/webgl_video_texture.h
[delete] https://crrev.com/0694f7c1f92d46137fe536f1fde23963fccac0b5/third_party/blink/renderer/modules/webgl/webgl_video_texture.idl
[delete] https://crrev.com/0694f7c1f92d46137fe536f1fde23963fccac0b5/third_party/blink/renderer/modules/webgl/webgl_video_texture_enum.h