https://codereview.chromium.org/2795703002 will make it easier to detect, solve and prevent these type of issues in the future.

ShouldCreateRenderSurface() in property_tree_builder.cc is probably the place to determine why a render surface is being created.

For UI, the cases that function cares about are non-Axis-Aligned clips, mask layers, opacity and filter. For opacity and filter that include any potential animation. Also for Opacity we do not check if there are overlapping, just that more than one layer should draw at the same opacity.

I noticed that in Layer::SetTransform, it checks Are2dAxisAligned transform. Does it mean any rotation animation will rebuild the property tree?

https://codereview.chromium.org/2795703002 has landed so it should be easy to reproduce and work on fixing this in ToT now.

In about_flags.cc, the switch hint is to use "ui-show-composited-layer-borders", but in the cc::switches [1] and histogram.xml [2], it is "ui-show-layer-borders".

Do you mind I make a change in the about_flags.cc to make it "ui-show-layer-borders" as well?

[1] https://cs.chromium.org/chromium/src/cc/base/switches.cc?type=cs&q=%22ui-show-layer-borders%22+package:%5Echromium$&l=61
[2] https://codereview.chromium.org/2795703002/diff/80001/tools/metrics/histograms/histograms.xml

Maybe better to use ui-show-composited-layer-borders everywhere as that's consistent with the renderer compositor flag:

https://cs.chromium.org/chromium/src/cc/base/switches.cc?l=60

Uploaded a cl to rename the switch: https://codereview.chromium.org/2821273003

Try to get familiar with the code.

First working on the Stylus tool using render surface, it seems creating render surface by this check in ShouldCreateRenderSurface() [1]:
  if (MaskLayer(layer)) {
    return true;
  }

[1] https://cs.chromium.org/chromium/src/cc/trees/property_tree_builder.cc?type=cs&q=ShouldCreateRenderSurface+package:%5Echromium$&l=757

There are two situations:
1. Creating the tray bubble view.
2. Destroying the tray bubble view.

1. Creating the tray bubble view, it create a mask [1] and return true in ShouldCreateRenderSurface() when check if (MaskLayer(layer)) [2].
2. Destroying the tray bubble view, it return true in ShouldCreateRenderSurface() when check [3] if (may_have_transparency && ShouldFlattenTransform(layer) && at_least_two_layers_in_subtree_draw_content).

~~~~~~~~~
For quick testing, commented out [1] to add mask layer and do not return true at [3], prevent creating render surface when creating and destroying the bubble view.
Questions:
a) is this mask layer a must?
b) can we avoid rendering render surface for mask layer?  
c) how to avoid the check in [3].

~~~~~~~~~~~~
[1] https://cs.chromium.org/chromium/src/ui/views/bubble/tray_bubble_view.cc?gsn=SetMaskLayer&l=229
[2] https://cs.chromium.org/chromium/src/cc/trees/property_tree_builder.cc?type=cs&l=757
[3] https://cs.chromium.org/chromium/src/cc/trees/property_tree_builder.cc?type=cs&l=816

There is one more place to add mask layer for system tray is during ink_drop animation [4]:
[4] https://cs.chromium.org/chromium/src/ui/views/animation/ink_drop_host_view.cc?type=cs&l=136

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c63400e394a50c8b42ef798eaee47245c5febf04

commit c63400e394a50c8b42ef798eaee47245c5febf04
Author: wutao <wutao@chromium.org>
Date: Wed Apr 19 00:50:22 2017

Rename ui-show-layer-borders to be consistent with other flag.

Rename ui-show-layer-borders to ui-show-composited-layer-borders to be
consistent with the renderer compositor flag.

BUG=708518
TEST=AboutFlagsHistogramTest.CheckHistograms and Manual
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_trusty_blink_rel

Review-Url: https://codereview.chromium.org/2821273003
Cr-Commit-Position: refs/heads/master@{#465450}

[modify] https://crrev.com/c63400e394a50c8b42ef798eaee47245c5febf04/cc/base/switches.cc
[modify] https://crrev.com/c63400e394a50c8b42ef798eaee47245c5febf04/tools/metrics/histograms/histograms.xml

Thanks for tracking this down. We can't avoid render surfaces for mask layer. Mask layers are expensive and should be avoided. Looks like we create the mask layer here:

https://cs.chromium.org/chromium/src/ui/views/bubble/tray_bubble_view.cc?rcl=8ac2349b4fc1649f4fef0a69de517ad74acabc10&l=214

and it's created to add rounded corners as explained here:

https://cs.chromium.org/chromium/src/ui/views/bubble/tray_bubble_view.cc?rcl=8ac2349b4fc1649f4fef0a69de517ad74acabc10&l=80

I'd suggest that we remove this mask layer completely and mark these layers as opaque to minimize overdraw. If we need rounded corners then we should go through the trouble of having the contents paint rounded corners. That's still efficient.

It seems the mask layer does little to the TrayBubbleView.

Only the Stylus Tools bubble has rounded corners. Please see the attached images.

Deleted: BubbleView_WithMask_StylusTools.png 79.7 KB

	BubbleView_WithMask_StylusTools.png 79.7 KB View Download

Deleted: BubbleView_NoMask_StylusTools.png 79.4 KB

	BubbleView_NoMask_StylusTools.png 79.4 KB View Download

Deleted: BubbleView_WithMask_Notification.png 77.3 KB

	BubbleView_WithMask_Notification.png 77.3 KB View Download

Deleted: BubbleView_NoMask_Notification.png 76.9 KB

	BubbleView_NoMask_Notification.png 76.9 KB View Download

+msw@

msw@, stevenjb@, sadrul@:
I saw you reviewed or commented in the original cl [1] to add this mask layer.
Could you please advise for removing this mask layer from the TrayBubbleView.

Thanks.

[1] https://chromiumcodereview.appspot.com/11293124

It seems fine to remove the mask if it doesn't regress any current behavior.
You should probably check that none of the TrayBubbleView users are affected.
(ie. isn't TrayBubbleView used for the main Chrome OS status area bubble itself?)
(perhaps some state-specific bubble content would draw over corners without a mask?)
aside: I'm not sure why all these bubbles don't have the same rounded corners...

+bartfab@, who wrote this mask several years ago.
Do you still recall: 
1) Under what circumstances that "This mask layer clips the bubble's content so that it does not overwrite the rounded bubble corners."
2) Is the rounded corners being overwrote the only concern? If we change the round corners to right angles, can we remove the mask layer as well?

Thanks.

IIRC, yes, the rounded corners were the only concern. The bubble is used as a menu. Tapping some of the items brings up a submenu, which slides in from the right inside the bubble. This is implemented as another rectangular area (surface/widget, not sure) that needs to be clipped to the bubble outline as it slides in. The only way to get that clipping working was to have a layer mask. If you give up on the rounded corners, the bubble's shape will be a simple rectangle and there are probably simpler / more efficient ways to clip the submenu to that than a layer mask.