Maybe we're not setting correctly the DrawQuad needs_blending
attribute, to investigate.  dcastagna@ do you know where'd be
the code for this CompositorFrame/RenderPass/DrawQuad creation? 
Don't find any reference under components/arc 

Miguel, that'd be in exo, since it's an android app. In particular, you might be looking for exo::Surface::AppendContentsToFrame where we create the render pass.
You can also check what android sends using adb and "dumpsys SurfaceFlinger".

Tom, does the UI get promoted to underlay? Is it possible that the UI quad is bigger than the content quad and we pick that one?

Yes, the toolbar [1] is getting promoted to an overlay but it's pretty tiny compared to the size of the canvas.

https://screenshot.googleplex.com/BGRrg0XVXok (note that the menu on the right is usually not open)

Thank you Tom.
The issue is that we run overlay strategies in a specific order. We usually prioritize the on-top strategy, and then we try the underlay one.
In this case we end up picking the toolbar with the on-top strategy, and then we stop.

One option could be to avoid marking some layers coming from exo as overlay candidates, currently we mark every layer.

Another option would be to pick the biggest quad across all the strategies, I'm afraid we'd see other interesting results with this option.

Is there some way we could prioritize a layer using EGL_KHR_mutable_render_buffer?

Unfortunately we don't know about EGL_KHR_mutable_render_buffer in exo/Chrome.

Is that something that could be passed through to exo/Chrome? Or some other signal such that we could know the app is using low-latency?

Concepts (https://play.google.com/store/apps/details?id=com.tophatch.concepts) is seeing the same issue. They are able to workaround it by placing a 1px transparent pixel above their main canvas.

This pixel seems to get the GPU Compositing and allow the main canvas underneath to get the low-latency output.

Two thoughts:
1 - Is it possible for us to give HW to the layer that is currently being redrawn, regardless of where it is in the stack? The layer that is changing (or changing the most) is likely to need the acceleration as is also the most likely to change again.

For concepts, the user is drawing on the canvas, the tool layers are not being redrawn so don't need HW.  If the user is using the tools, that could be promoted to get the acceleration as the underlying canvas is not being redrawn at that time. This would be ideal for this particular app (fast drawing, and fast animation on tools menu when in use).

2 - Perhaps there could be some way to for devs to specifically signal which layer should be prioritized? Perhaps using an Android manifest meta-flag with the layer id string? Then devs specifically targeting Chrome OS can choose for their specific app without system-wide effects? Possibly as short-term fix until we figure out something better.

hadrosaur@ when you say "canvas" do you mean the Web canvas
or some Android concept? If the first, it'd be great if you 
could make Concepts folks aware of 
 tinyurl.com/lowlatency-canvas-on-chromeos
where there's a specific Section "Things to check for" and another
"Further debugging" which will probably help unblocking them.

By "canvas" in this case I was speaking colloquially and actually meant the "drawing part of the app" from a user point of view.

Technically, I meant the main GLSurfaceView they are using for taking in input events from the stylus or finger. On top of this they have some other UI tool/brush elements. The current underlay strategy is granting HW underlays to these smaller tool/brush elements and not to the larger canvas where the drawing is happening.  They are using front-buffer rendering on the GLSurface and only redrawing damaged areas.

The current hack they are using is a 1px transparent pixel (or coloured pixel) drawn on top of everything. For some reason this causes that main GLSurfaceView to get the HW underlay and greatly improves performance.

It works but feels fragile - will this hack continue to work in M74? M78? Long term, it'd be great for us to have an optimal strategy and a way for devs to choose the preferred layer that gets the underlay if it is available.  Imagine, for example, a game that has a large game display with controls overlaid on top - the game display should get the underlay.

Optimal strategy is likely something like: the currently changing/most changing layer. A changing layer needs the acceleration vs. one that is not changing. It is also the mostly likely to change again. 

Dev preferred choice of layer that gets the underlay would be excellent for any app where latency is an issue and would provided a clean way for devs to request behaviour.

I understand HW over/underlay is not always available depending on device and current use, but ideally we could offer the best predictable behaviour when it is. 

Obviously, easier to say than do.