We did a bi-sect between chrome 65.0.3325.5 and 66.0.3326.0, one suspicious patch we found is: 'blink/canvas: Switch Canvas to use cc's ImageDecodeCache.' (https://chromium-review.googlesource.com/c/chromium/src/+/809758).

w/o patch (master@{#530585}): 35.2 fps
w/  patch (master@{#530586}): 3.6 fps

We've captured the traces for both cases, which also showed large regression in frame time:
w/o patch (master@{#530585}): 29.18  ms (per frame)
w/  patch (master@{#530586}): 478.97 ms (per frame)

Major regression came from Canvas2DLayerBridge::flushRecording after using GpuImageDecodeCache w/ the patch. 

Traces are also attached for reference.

Deleted: CanvasEarth-withoutPatch-33fps-frame.png 230 KB

	CanvasEarth-withoutPatch-33fps-frame.png 230 KB View Download

Deleted: CanvasEarth-withPatch-3fps-frame.png 286 KB

	CanvasEarth-withPatch-3fps-frame.png 286 KB View Download

Deleted: CanvasEarth-withoutPatch-33fps-trace.json.gz 3.0 MB

CanvasEarth-withoutPatch-33fps-trace.json.gz 3.0 MB Download

Deleted: CanvasEarth-withPatch-3fps-trace.json.gz 2.7 MB

CanvasEarth-withPatch-3fps-trace.json.gz 2.7 MB Download

@khushalsagar, could you please give any comments?

Do you still see this on the latest canary? I can see that its constantly locking/unlocking the image which was known to be a perf hit and was fixed in https://chromium-review.googlesource.com/c/chromium/src/+/884881.

Thanks for the comment, khushalsagar@
I tried the latest canary image 10440.0.0 (platform version)/66.0.3355.0 (browser version) and the result is 3.8 fps, regression still exists.

I'll take a look. Can you attach the traces from canary if possible?

Sure, do you have any specific trace categories required?

cc,gpu is good.

Trace from canary attached with cc and gpu categories.

Deleted: CanvasEarth-latestCanary-trace.gz 3.2 MB

CanvasEarth-latestCanary-trace.gz 3.2 MB Download

Could you also provide a link to the benchmark on which I can repro this?

I've attached a simplified version, which can also reproduce the issue at my side. You may put it in Apache directory and access it from browser.

Deleted: CanvasEarth-Simple.tar.gz 661 KB

CanvasEarth-Simple.tar.gz 661 KB Download

I tried opening index.html directly in the browser and that doesn't work for some reason. The canvas just renders black.

It's because that the img file was blocked by CORS policy. 
Would you please try to open browser with '--disable-web-security --user-data-dir' and open index.html directly or put files in Apache server directory and then access from browser localhost.

Great. That worked and I can repro it. Looking now.

Its using a record shader which when decoded by ScopedRasterFlags creates a new SkPictureShader each time and thrashes skia's cache. For every op, we rasterize the picture repeatedly.

ericrk@, we had discussed this in the context of OOP raster. And mostly narrowed down on caching the resulting SkPicture.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/058b6774e4b0191372be43b68bf5aaa2418b829e

commit 058b6774e4b0191372be43b68bf5aaa2418b829e
Author: Khushal <khushalsagar@chromium.org>
Date: Mon Mar 12 19:40:13 2018

cc: Disable decoding of PaintRecord shaders in cc.

For PaintRecord backed shaders, skia performs the rasterization of the
shader's record at the required scale before tiling. In order to decode
images in these records using cc's decode cache, we create a temporary
SkPicture with images replaced with decodes from cc's cache and use
that in a new SkPictureShader each time the PaintShader is rasterized.

Skia internally maintains a cache of rasterized pictures used in these
shaders, keyed on the SkPictureShader's uniqueID. Since a new
SkPictureShader is created for each raster call, we don't get a cache
hit in skia and end up re-rasterizing the shader's record.

The ideal solution for this would be to maintain a PaintShaderCache
which caches rasterized records. But given the rarity of these cases,
its not worth duplicating skia's picture cache. However, if any of
the images in these records are animated, they must go through cc's
decode stack to update them and use cached animation frames. The
change ensures that we only decode shaders in cc if they have animated
images. Note that the performance hit of re-rasterization of records
in these cases will still exist.

R=chrishtr@chromium.org, enne@chromium.org, ericrk@chromium.org

Bug:  817640 
Cq-Include-Trybots: master.tryserver.blink:linux_trusty_blink_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: Ibd1f3db9902f21fca20c99be1cf9e36b54cd7085
Reviewed-on: https://chromium-review.googlesource.com/952529
Commit-Queue: Khushal <khushalsagar@chromium.org>
Reviewed-by: Eric Karl <ericrk@chromium.org>
Reviewed-by: enne <enne@chromium.org>
Reviewed-by: Chris Harrelson <chrishtr@chromium.org>
Reviewed-by: Antoine Labour <piman@chromium.org>
Cr-Commit-Position: refs/heads/master@{#542570}
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/cc/paint/discardable_image_map.cc
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/cc/paint/discardable_image_map.h
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/cc/paint/discardable_image_map_unittest.cc
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/cc/paint/paint_op_buffer_unittest.cc
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/cc/paint/paint_shader.h
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/cc/paint/paint_shader_unittest.cc
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/cc/paint/scoped_raster_flags.cc
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/cc/paint/scoped_raster_flags_unittest.cc
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/cc/trees/layer_tree_host_unittest.cc
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/SlowTests
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/platform/linux/virtual/gpu-rasterization/images/color-profile-svg-fill-text-expected.png
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/platform/mac-mac10.10/virtual/gpu-rasterization/images/color-profile-svg-fill-text-expected.png
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/platform/mac/images/cross-fade-background-size-expected.png
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/platform/mac/virtual/exotic-color-space/images/cross-fade-background-size-expected.png
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/platform/mac/virtual/gpu-rasterization/images/color-profile-background-image-space-expected.png
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/platform/mac/virtual/gpu-rasterization/images/color-profile-svg-expected.png
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/platform/mac/virtual/gpu-rasterization/images/color-profile-svg-fill-text-expected.png
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/platform/mac/virtual/gpu-rasterization/images/cross-fade-background-size-expected.png
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/platform/win/images/cross-fade-background-size-expected.png
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/platform/win/virtual/exotic-color-space/images/cross-fade-background-size-expected.png
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/platform/win/virtual/gpu-rasterization/images/color-profile-background-image-space-expected.png
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/platform/win/virtual/gpu-rasterization/images/color-profile-svg-expected.png
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/platform/win/virtual/gpu-rasterization/images/color-profile-svg-fill-text-expected.png
[modify] https://crrev.com/058b6774e4b0191372be43b68bf5aaa2418b829e/third_party/WebKit/LayoutTests/platform/win/virtual/gpu-rasterization/images/cross-fade-background-size-expected.png

+govind@. I'll verify in canary tomorrow and request a merge to 66.

+ josafat@ (Chrome OS M66 TPM)

The NextAction date has arrived: 2018-03-13

 Issue 788075  has been merged into this issue.

Your change meets the bar and is auto-approved for M66. Please go ahead and merge the CL to branch 3359 manually. Please contact milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), josafat@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c

commit ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c
Author: Khushal <khushalsagar@chromium.org>
Date: Wed Mar 14 23:12:12 2018

cc: Disable decoding of PaintRecord shaders in cc.

For PaintRecord backed shaders, skia performs the rasterization of the
shader's record at the required scale before tiling. In order to decode
images in these records using cc's decode cache, we create a temporary
SkPicture with images replaced with decodes from cc's cache and use
that in a new SkPictureShader each time the PaintShader is rasterized.

Skia internally maintains a cache of rasterized pictures used in these
shaders, keyed on the SkPictureShader's uniqueID. Since a new
SkPictureShader is created for each raster call, we don't get a cache
hit in skia and end up re-rasterizing the shader's record.

The ideal solution for this would be to maintain a PaintShaderCache
which caches rasterized records. But given the rarity of these cases,
its not worth duplicating skia's picture cache. However, if any of
the images in these records are animated, they must go through cc's
decode stack to update them and use cached animation frames. The
change ensures that we only decode shaders in cc if they have animated
images. Note that the performance hit of re-rasterization of records
in these cases will still exist.

R=chrishtr@chromium.org, enne@chromium.org, ericrk@chromium.org
TBR=khushalsagar@chromium.org

(cherry picked from commit 058b6774e4b0191372be43b68bf5aaa2418b829e)

Bug:  817640 
Cq-Include-Trybots: master.tryserver.blink:linux_trusty_blink_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: Ibd1f3db9902f21fca20c99be1cf9e36b54cd7085
Reviewed-on: https://chromium-review.googlesource.com/952529
Commit-Queue: Khushal <khushalsagar@chromium.org>
Reviewed-by: Eric Karl <ericrk@chromium.org>
Reviewed-by: enne <enne@chromium.org>
Reviewed-by: Chris Harrelson <chrishtr@chromium.org>
Reviewed-by: Antoine Labour <piman@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#542570}
Reviewed-on: https://chromium-review.googlesource.com/963608
Reviewed-by: Khushal <khushalsagar@chromium.org>
Cr-Commit-Position: refs/branch-heads/3359@{#252}
Cr-Branched-From: 66afc5e5d10127546cc4b98b9117aff588b5e66b-refs/heads/master@{#540276}
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/cc/paint/discardable_image_map.cc
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/cc/paint/discardable_image_map.h
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/cc/paint/discardable_image_map_unittest.cc
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/cc/paint/paint_op_buffer_unittest.cc
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/cc/paint/paint_shader.h
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/cc/paint/paint_shader_unittest.cc
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/cc/paint/scoped_raster_flags.cc
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/cc/paint/scoped_raster_flags_unittest.cc
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/cc/trees/layer_tree_host_unittest.cc
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/third_party/WebKit/LayoutTests/SlowTests
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/third_party/WebKit/LayoutTests/platform/linux/virtual/gpu-rasterization/images/color-profile-svg-fill-text-expected.png
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/third_party/WebKit/LayoutTests/platform/mac/images/cross-fade-background-size-expected.png
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/third_party/WebKit/LayoutTests/platform/mac/virtual/exotic-color-space/images/cross-fade-background-size-expected.png
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/third_party/WebKit/LayoutTests/platform/mac/virtual/gpu-rasterization/images/color-profile-background-image-space-expected.png
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/third_party/WebKit/LayoutTests/platform/mac/virtual/gpu-rasterization/images/color-profile-svg-expected.png
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/third_party/WebKit/LayoutTests/platform/mac/virtual/gpu-rasterization/images/cross-fade-background-size-expected.png
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/third_party/WebKit/LayoutTests/platform/win/images/cross-fade-background-size-expected.png
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/third_party/WebKit/LayoutTests/platform/win/virtual/exotic-color-space/images/cross-fade-background-size-expected.png
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/third_party/WebKit/LayoutTests/platform/win/virtual/gpu-rasterization/images/color-profile-background-image-space-expected.png
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/third_party/WebKit/LayoutTests/platform/win/virtual/gpu-rasterization/images/color-profile-svg-expected.png
[modify] https://crrev.com/ec45f4d2599aee0c6d0f75ddd90ae1a475193a1c/third_party/WebKit/LayoutTests/platform/win/virtual/gpu-rasterization/images/cross-fade-background-size-expected.png