Hi ccameron@, do you have any idea what might cause this issue?

You can test if it is related to CoreAnimation by running with the flags
  --disable-mac-overlays

That will force us to use the OpenGL path.

Hmm when I add that flag, the text becomes clear. It looks like the mask tiling code path does not cope with CA.

I tried printing out projection_matrix and quad_to_target_transform in DrawRPDQ.

This is the blurry one (With CA overlay):
projection_matrix =
[ +0.0100 +0.0000 +0.0000 -1.0000  
  +0.0000 +0.0100 +0.0000 -1.0000  
  +0.0000 +0.0000 +0.0000 +0.0000  
  +0.0000 +0.0000 +0.0000 +1.0000 ]
,
quad_to_target_transform =
[ +1.0000 +0.0000 +0.0000 +0.0000  
  +0.0000 +1.0000 +0.0000 +0.0000  
  +0.0000 +0.0000 +1.0000 +0.0000  
  +0.0000 +0.0000 +0.0000 +1.0000 ]

And this is the clear one (with --disable-mac-overlays):
projection_matrix =
[ +0.0012 +0.0000 +0.0000 -1.0000  
  +0.0000 +0.0017 +0.0000 -1.0000  
  +0.0000 +0.0000 +0.0000 +0.0000  
  +0.0000 +0.0000 +0.0000 +1.0000 ]
,
quad_to_target_transform =
[ +2.0000 +0.0000 +0.0000 +36.0000  
  +0.0000 +2.0000 +0.0000 +36.0000  
  +0.0000 +0.0000 +1.0000 +0.0000  
  +0.0000 +0.0000 +0.0000 +1.0000 ]

The output might not be helpful since they are drawing to different "frames".

The bug is somewhere in GLRenderer::CopyRenderPassDrawQuadToOverlayResource.

In that particular fiddle, the input content resource is 400x400, but we copy it to a 200x200 IOSurface.

I'd guess that somewhere we don't realize that we need the device scale factor to be respected.

We determine the IOSurface size by the size of RenderPassDrawQuad::rect. With !enable_mask_tiling, this is 400x400, with enable_mask_tiling, it is 200x200.

Meanwhile, RenderPassDrawQuad::shared_quad_state::quad_to_target_transform changes from being identity to a 2x scale.

I don't know offhand what is the "correct" size for the texture to allocate here. GLRenderer::CopyRenderPassDrawQuadToOverlayResource is built around the assumption that RenderPassDrawQuad::rect is the correct pixel size, and that RenderPassDrawQuad::shared_quad_state::quad_to_target_transform represents a transformation on those pixels.

The code in RenderSurfaceImpl::TileMaskLayer adjusts the transform matrix of the shared quad state, so it's not surprising that this would change with a different mode.

PictureLayerImpl::AppendQuads does have a special case for RasterSource::IsSolidColor, that ignores scale factors and just calls PopulateSharedQuadState vs PopulateScaledSharedQuadState that other code uses.

I wonder if adjusting that conditional to create a scaled shared quad state would provide the right values later in the pipeline.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/48493dc45de06867eec97e7944f56d511e3c4578

commit 48493dc45de06867eec97e7944f56d511e3c4578
Author: sunxd <sunxd@chromium.org>
Date: Thu Oct 12 20:25:14 2017

cc: make solid color picture layer tiling respect device scale factor

Solid color picture layer did not consider device scale factor when it
appends quads. This would result in blurry contents on high dpi devices.

This CL makes the solid color code path respect device scale factor as
non solid ones.

Bug:  769319 
Cq-Include-Trybots: master.tryserver.blink:linux_trusty_blink_rel
Change-Id: I3a383721b33378065ff5d868afcc729b08064d82
Reviewed-on: https://chromium-review.googlesource.com/697886
Reviewed-by: enne <enne@chromium.org>
Reviewed-by: Chris Harrelson <chrishtr@chromium.org>
Reviewed-by: Robert Flack <flackr@chromium.org>
Commit-Queue: Xianda Sun <sunxd@chromium.org>
Cr-Commit-Position: refs/heads/master@{#508442}
[modify] https://crrev.com/48493dc45de06867eec97e7944f56d511e3c4578/cc/layers/picture_layer_impl.cc
[modify] https://crrev.com/48493dc45de06867eec97e7944f56d511e3c4578/cc/layers/render_surface_impl_unittest.cc
[modify] https://crrev.com/48493dc45de06867eec97e7944f56d511e3c4578/third_party/WebKit/LayoutTests/TestExpectations
[delete] https://crrev.com/3d541cac26d562d9a6f621362faef99d807056f6/third_party/WebKit/LayoutTests/compositing/3d-corners-expected.png
[modify] https://crrev.com/48493dc45de06867eec97e7944f56d511e3c4578/third_party/WebKit/LayoutTests/compositing/overflow/scaled-overflow-expected.png
[delete] https://crrev.com/3d541cac26d562d9a6f621362faef99d807056f6/third_party/WebKit/LayoutTests/compositing/scaling/tiled-layer-recursion-expected.png
[modify] https://crrev.com/48493dc45de06867eec97e7944f56d511e3c4578/third_party/WebKit/LayoutTests/platform/linux/transforms/3d/point-mapping/3d-point-mapping-3-expected.png
[add] https://crrev.com/48493dc45de06867eec97e7944f56d511e3c4578/third_party/WebKit/LayoutTests/platform/mac/compositing/3d-corners-expected.png
[add] https://crrev.com/48493dc45de06867eec97e7944f56d511e3c4578/third_party/WebKit/LayoutTests/platform/mac/compositing/scaling/tiled-layer-recursion-expected.png
[modify] https://crrev.com/48493dc45de06867eec97e7944f56d511e3c4578/third_party/WebKit/LayoutTests/platform/mac/transforms/3d/point-mapping/3d-point-mapping-3-expected.png
[modify] https://crrev.com/48493dc45de06867eec97e7944f56d511e3c4578/third_party/WebKit/LayoutTests/platform/win/compositing/3d-corners-expected.png
[delete] https://crrev.com/3d541cac26d562d9a6f621362faef99d807056f6/third_party/WebKit/LayoutTests/platform/win/compositing/overflow/scaled-overflow-expected.png
[modify] https://crrev.com/48493dc45de06867eec97e7944f56d511e3c4578/third_party/WebKit/LayoutTests/platform/win/compositing/scaling/tiled-layer-recursion-expected.png
[modify] https://crrev.com/48493dc45de06867eec97e7944f56d511e3c4578/third_party/WebKit/LayoutTests/platform/win/transforms/3d/point-mapping/3d-point-mapping-3-expected.png
[delete] https://crrev.com/3d541cac26d562d9a6f621362faef99d807056f6/third_party/WebKit/LayoutTests/platform/win/virtual/prefer_compositing_to_lcd_text/compositing/overflow/scaled-overflow-expected.png
[delete] https://crrev.com/3d541cac26d562d9a6f621362faef99d807056f6/third_party/WebKit/LayoutTests/virtual/prefer_compositing_to_lcd_text/compositing/overflow/scaled-overflow-expected.png