New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 727385 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Task



Sign in to add a comment

Make Canvas stream capture use async GPU readbacks

Project Member Reported by junov@chromium.org, May 29 2017

Issue description

The call to SkImage::readPixels in content::CanvasCaptureHandler::CreateNewFrame may cause a GPU readback if the SkImage is GPU accelerated.  There is an opportunity to avoid blocking the main thread of the render process by using async readbacks.

To achieve this, we need to:
1) determine that the Image is texture-backed,
2) If so, flush the GrContext that owns the image (because there is a GPU deferral/batching mechanism inside skia).  Do this by calling SkImage::getTextureHandle with the flushPendingGrContextIO arg set to true.  This will also return the texture ID require for the next step.
3) Do something like this: https://cs.chromium.org/chromium/src/third_party/WebKit/Source/modules/webgl/WebGLGetBufferSubDataAsync.cpp?sq=package:chromium&l=90


 
Status: Started (was: Assigned)

Comment 2 by kbr@chromium.org, Sep 26 2017

Components: Blink>MediaStream>CaptureFromElement
Note that the Chromecast team added a different asynchronous readback path a while ago and there's a possibility that it's more efficient. It would be worth looking into it. One relevant entry point is GLHelper::CopyTextureToImpl::ReadbackAsync in src/components/viz/common/gl_helper.cc :

https://cs.chromium.org/chromium/src/components/viz/common/gl_helper.cc?type=cs&sq=package:chromium&l=419

Labels: -Pri-3 M-63 Pri-1
Cc: sanb...@stageten.tv d...@stageten.tv tma...@stageten.tv steve.se...@stageten.tv

Comment 5 by m...@chromium.org, Oct 11 2017

Cc: m...@chromium.org
I got a working version in the below CL using viz::GLHelper::ReadbackTextureAsync, suggested on #2. There were some issues re colorspace and texture origin, but it seems to work as far as I checked.
https://chromium-review.googlesource.com/c/chromium/src/+/711160

When I was discussing the problems, miu@ suggested that I shouldn't do it using viz::GLHelper, but rather move it into a smaller component. Preferably, this functionality can read back YUV. This would be an improvement(32 bpp->12/20bpp), but requires some more piping about alpha channel and flipped origin from SkImage. I think we can review/land the above CL using existing GLHelper functionality first as it is a clear improvement. Then, we can refactor GLHelper and replace calls. miu@ WDYT?

I also added some trace for metrics. It is from mbp16 reading from a 720p canvas playing video. As expected there are blocking readPixels() calls, up to ~68ms. Async readbacks should eliminate them. 

Also, I was considering adding a "canvas_capture" thread to do all the background work: readPixels() and convertToI420() calls. I now think that would be unnecessary if we have async readbacks. convertToI420() takes ~0.3 ms on average and there wouldn't be enough work to spin a new thread. Additionally, this class already operates on 2 threads. 
trace_canvas_capture.json.gz
17.4 KB Download

Comment 7 by kbr@chromium.org, Oct 12 2017

Great work. It seems fine to me to use viz::GLHelper at least as a start. Do you have a trace or numbers demonstrating the improvement? Let's get your CL out for review ASAP; this should be a significant speedup to the canvas capture implementation. Thanks.

Comment 8 by m...@chromium.org, Oct 12 2017

emircan's comment (#6) sums things up well. Another reason we should proceed with YUV readback, other than additional performance benefits, is color space conversions: libyuv currently uses BT.601 conversion factors, but the media video pipeline assumes BT.709 (the difference between the old standard-definition TV and HDTV).
Re #7, I attached a post CL trace. It clearly shows the gains from async call, by lower Wall duration occupied in main render thread. 
Overall CPU consumption is similar. Traced CPU consumption for convertToI420() increases though, due to extra flip call. We can monitor bots below to see what happens in different platforms after CL lands.
https://chromeperf.appspot.com/report?sid=3c68b43fe6238a179864d82b945a8516cdbfd2f7545bb843433c6599649a9257

Re #8, sounds good. I added a new bug to track that: issue 774297.

trace_canvas_capture_async.json.gz
15.2 KB Download
Labels: -Type-Bug Type-Task
Project Member

Comment 11 by bugdroid1@chromium.org, Nov 16 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cff667d9388ca757d9211cc3e769e4bb4c72b4e8

commit cff667d9388ca757d9211cc3e769e4bb4c72b4e8
Author: Emircan Uysaler <emircan@chromium.org>
Date: Thu Nov 16 03:55:42 2017

Use async GPU readbacks in canvas.captureStream()

This CL replaces the existing blocking call to SkImage::readPixels() with
GLHelper::ReadbackTextureAsync() and GLHelper::CreateReadbackPipelineYUV()
callbacks for texture backed images.
In order to use for these calls, a GLHelper instance is lazily created and
owned by blink::WebGraphicsContext3DProvider.

Bug:  727385 
Cq-Include-Trybots: master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2
Change-Id: Iad8561183afddff0794038213ee22cec3e04ed76
Reviewed-on: https://chromium-review.googlesource.com/756474
Commit-Queue: Emircan Uysaler <emircan@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Kentaro Hara <haraken@chromium.org>
Reviewed-by: Yuri Wiitala <miu@chromium.org>
Reviewed-by: Justin Novosad <junov@chromium.org>
Reviewed-by: danakj <danakj@chromium.org>
Cr-Commit-Position: refs/heads/master@{#516979}
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/components/viz/common/gl_helper.cc
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/components/viz/common/gl_helper.h
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/content/browser/webrtc/webrtc_capture_from_element_browsertest.cc
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/content/renderer/media_capture_from_element/canvas_capture_handler.cc
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/content/renderer/media_capture_from_element/canvas_capture_handler.h
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/content/renderer/media_capture_from_element/canvas_capture_handler_unittest.cc
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/content/renderer/webgraphicscontext3d_provider_impl.cc
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/content/renderer/webgraphicscontext3d_provider_impl.h
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/content/test/data/media/canvas_capture_color.html
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/third_party/WebKit/Source/core/html/HTMLCanvasElement.cpp
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/third_party/WebKit/Source/core/html/canvas/CanvasDrawListener.cpp
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/third_party/WebKit/Source/core/html/canvas/CanvasDrawListener.h
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/third_party/WebKit/Source/modules/mediacapturefromelement/OnRequestCanvasDrawListener.cpp
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/third_party/WebKit/Source/modules/mediacapturefromelement/OnRequestCanvasDrawListener.h
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/third_party/WebKit/Source/modules/mediacapturefromelement/TimedCanvasDrawListener.cpp
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/third_party/WebKit/Source/modules/mediacapturefromelement/TimedCanvasDrawListener.h
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/third_party/WebKit/Source/platform/graphics/gpu/DrawingBufferTestHelpers.h
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/third_party/WebKit/Source/platform/graphics/test/FakeWebGraphicsContext3DProvider.h
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/third_party/WebKit/public/platform/WebCanvasCaptureHandler.h
[modify] https://crrev.com/cff667d9388ca757d9211cc3e769e4bb4c72b4e8/third_party/WebKit/public/platform/WebGraphicsContext3DProvider.h

Project Member

Comment 12 by bugdroid1@chromium.org, Nov 16 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/146471c6311303fc604f96ea65a2684be9ae833b

commit 146471c6311303fc604f96ea65a2684be9ae833b
Author: Guido Urdaneta <guidou@chromium.org>
Date: Thu Nov 16 16:21:08 2017

Revert "Use async GPU readbacks in canvas.captureStream()"

This reverts commit cff667d9388ca757d9211cc3e769e4bb4c72b4e8.

Reason for revert: Suspect of breaking the following tests on the linux-chromeos-rel bot
WebRtcCaptureFromElementBrowserTest.CaptureFromOpaqueCanvas2DHandlesContextLoss
WebRtcCaptureFromElementBrowserTest.CaptureFromCanvas2DHandlesContextLoss

First failure
https://build.chromium.org/p/chromium.chromiumos/builders/linux-chromeos-rel/builds/2653

The bot has been red since.

Original change's description:
> Use async GPU readbacks in canvas.captureStream()
> 
> This CL replaces the existing blocking call to SkImage::readPixels() with
> GLHelper::ReadbackTextureAsync() and GLHelper::CreateReadbackPipelineYUV()
> callbacks for texture backed images.
> In order to use for these calls, a GLHelper instance is lazily created and
> owned by blink::WebGraphicsContext3DProvider.
> 
> Bug:  727385 
> Cq-Include-Trybots: master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2
> Change-Id: Iad8561183afddff0794038213ee22cec3e04ed76
> Reviewed-on: https://chromium-review.googlesource.com/756474
> Commit-Queue: Emircan Uysaler <emircan@chromium.org>
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Reviewed-by: Kentaro Hara <haraken@chromium.org>
> Reviewed-by: Yuri Wiitala <miu@chromium.org>
> Reviewed-by: Justin Novosad <junov@chromium.org>
> Reviewed-by: danakj <danakj@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#516979}

TBR=danakj@chromium.org,miu@chromium.org,kbr@chromium.org,haraken@chromium.org,junov@chromium.org,emircan@chromium.org

Change-Id: I9dee0e7e28c2c5e1375578fe6192aeefd9e0cdff
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  727385 
Cq-Include-Trybots: master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2
Reviewed-on: https://chromium-review.googlesource.com/775098
Reviewed-by: Guido Urdaneta <guidou@chromium.org>
Commit-Queue: Guido Urdaneta <guidou@chromium.org>
Cr-Commit-Position: refs/heads/master@{#517100}
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/components/viz/common/gl_helper.cc
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/components/viz/common/gl_helper.h
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/content/browser/webrtc/webrtc_capture_from_element_browsertest.cc
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/content/renderer/media_capture_from_element/canvas_capture_handler.cc
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/content/renderer/media_capture_from_element/canvas_capture_handler.h
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/content/renderer/media_capture_from_element/canvas_capture_handler_unittest.cc
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/content/renderer/webgraphicscontext3d_provider_impl.cc
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/content/renderer/webgraphicscontext3d_provider_impl.h
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/content/test/data/media/canvas_capture_color.html
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/third_party/WebKit/Source/core/html/HTMLCanvasElement.cpp
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/third_party/WebKit/Source/core/html/canvas/CanvasDrawListener.cpp
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/third_party/WebKit/Source/core/html/canvas/CanvasDrawListener.h
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/third_party/WebKit/Source/modules/mediacapturefromelement/OnRequestCanvasDrawListener.cpp
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/third_party/WebKit/Source/modules/mediacapturefromelement/OnRequestCanvasDrawListener.h
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/third_party/WebKit/Source/modules/mediacapturefromelement/TimedCanvasDrawListener.cpp
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/third_party/WebKit/Source/modules/mediacapturefromelement/TimedCanvasDrawListener.h
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/third_party/WebKit/Source/platform/graphics/gpu/DrawingBufferTestHelpers.h
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/third_party/WebKit/Source/platform/graphics/test/FakeWebGraphicsContext3DProvider.h
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/third_party/WebKit/public/platform/WebCanvasCaptureHandler.h
[modify] https://crrev.com/146471c6311303fc604f96ea65a2684be9ae833b/third_party/WebKit/public/platform/WebGraphicsContext3DProvider.h

Project Member

Comment 13 by bugdroid1@chromium.org, Nov 16 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f75ae2adde4ed0428c63909bd044464a941979d4

commit f75ae2adde4ed0428c63909bd044464a941979d4
Author: Emircan Uysaler <emircan@chromium.org>
Date: Thu Nov 16 23:01:47 2017

Reland "Use async GPU readbacks in canvas.captureStream()"

This is a reland of cff667d9388ca757d9211cc3e769e4bb4c72b4e8
Original change's description:
> Use async GPU readbacks in canvas.captureStream()
>
> This CL replaces the existing blocking call to SkImage::readPixels() with
> GLHelper::ReadbackTextureAsync() and GLHelper::CreateReadbackPipelineYUV()
> callbacks for texture backed images.
> In order to use for these calls, a GLHelper instance is lazily created and
> owned by blink::WebGraphicsContext3DProvider.
>
> Bug:  727385 
> Cq-Include-Trybots: master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2
> Change-Id: Iad8561183afddff0794038213ee22cec3e04ed76
> Reviewed-on: https://chromium-review.googlesource.com/756474
> Commit-Queue: Emircan Uysaler <emircan@chromium.org>
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Reviewed-by: Kentaro Hara <haraken@chromium.org>
> Reviewed-by: Yuri Wiitala <miu@chromium.org>
> Reviewed-by: Justin Novosad <junov@chromium.org>
> Reviewed-by: danakj <danakj@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#516979}

TBR=danakj@chromium.org,miu@chromium.org,kbr@chromium.org,haraken@chromium.org,junov@chromium.org

Bug:  727385 
Change-Id: Ic939674cf678a239a9a68bfb6211198fa33441c9
Cq-Include-Trybots: master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2
Reviewed-on: https://chromium-review.googlesource.com/775593
Reviewed-by: Emircan Uysaler <emircan@chromium.org>
Commit-Queue: Emircan Uysaler <emircan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#517217}
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/components/viz/common/gl_helper.cc
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/components/viz/common/gl_helper.h
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/content/browser/webrtc/webrtc_capture_from_element_browsertest.cc
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/content/renderer/media_capture_from_element/canvas_capture_handler.cc
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/content/renderer/media_capture_from_element/canvas_capture_handler.h
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/content/renderer/media_capture_from_element/canvas_capture_handler_unittest.cc
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/content/renderer/webgraphicscontext3d_provider_impl.cc
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/content/renderer/webgraphicscontext3d_provider_impl.h
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/content/test/data/media/canvas_capture_color.html
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/third_party/WebKit/Source/core/html/HTMLCanvasElement.cpp
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/third_party/WebKit/Source/core/html/canvas/CanvasDrawListener.cpp
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/third_party/WebKit/Source/core/html/canvas/CanvasDrawListener.h
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/third_party/WebKit/Source/modules/mediacapturefromelement/OnRequestCanvasDrawListener.cpp
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/third_party/WebKit/Source/modules/mediacapturefromelement/OnRequestCanvasDrawListener.h
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/third_party/WebKit/Source/modules/mediacapturefromelement/TimedCanvasDrawListener.cpp
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/third_party/WebKit/Source/modules/mediacapturefromelement/TimedCanvasDrawListener.h
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/third_party/WebKit/Source/platform/graphics/gpu/DrawingBufferTestHelpers.h
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/third_party/WebKit/Source/platform/graphics/test/FakeWebGraphicsContext3DProvider.h
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/third_party/WebKit/public/platform/WebCanvasCaptureHandler.h
[modify] https://crrev.com/f75ae2adde4ed0428c63909bd044464a941979d4/third_party/WebKit/public/platform/WebGraphicsContext3DProvider.h

Status: Fixed (was: Started)

Sign in to add a comment