Some more testing indicates that this is Dru-specific but started in M67:
Dru M66: Fast
Dru M67: Slow (I think Tom meant M66 in his last line above)
Dru M68: Slow

Kevin M67: Fast

Testing with 67.0.3396.41

Is it no longer using a HW overlay? Compare before/after with chrome://flags/#tint-gl-composited-content.

If it's still using HW overlay then next step is to check if we're compositing for some other reason. The cc-triangles category in chrome://tracing can be used for this. It should drop to 0 when we're not compositing.

If both of the above items are working as expected then it might be that we no  longer enter single-buffer mode.

On M67, the drawing surface is not tinted when drawing in portrait but is tinted when drawing in landscape (as expected), but is slow in both orientations.

Squid is fast in both orientations but is tinted in both orientations.

http://nacl-latency.firebaseapp.com is tinted in portrait only and is fast in landscape only.

For tracing, which type of trace should I be doing and where should I look for cc-triangles?

I assuming Squid is tinted because they use a transparent overlay as otherwise we're not expecting it to be low-latency.

Not sure what's going on with http://nacl-latency.firebaseapp.com. I guess that code was never updated to handle rotations correctly.

In tracing, disable all categories, then enable 'cc' on the left and 'cc.debug.triangles' on the right. cc.debug.triangles produces a trace counter that is expected to be at 0 for low-latency. 0 means that we're compositing 0 triangles and whatever you see on screen is presented using hw overlays.

0 triangles during drawing. Will add some logging to see if we think we're getting single buffer.

According to our logs, we are getting a single-buffered surface.

Ok, this is very surprising then. Maybe something changed on the input side. Is the laser pointer feature in the chrome os ui fast on the device or does that suffer too?

Laser pointer is fast.

Correction from above: http://nacl-latency.firebaseapp.com is fast & hw overlay in portrait but not landscape.

Any update on this? 

We are planning on promoting 68 to beta next week so we need a fix for this soon.

To summarize:
- Laser pointer is fast
- Squid is fast at any rotation
- nacl-latency.firebase-app.com is fast in landscape

But Keep Chrome app is slow.

Keep Chrome app seems to be using HW overlays in portrait mode, but not in landscape, since the buffer is rotated.
It seems to have high latency in both cases.

Has anyone investigated what's going on on the input side?

We haven't done much digging in that direction because http://nacl-latency.firebaseapp.com/ appeared to be fast (in landscape) in my previous testing. Can you confirm you're seeing the same thing? Would the chrome app be different from NaCl in the browser?

I do confirm http://nacl-latency.firebaseapp.com/ is much better than Keep Chrome app.

I grabbed a trace of the two. It seems that in the Keep Chrome app the GPU process main thread is really busy dealing with CommandBufferService:PutChanged coming from .PPAPIContext. This is particularly bad in portrait mode where we don't hit the HW overlay case and the main GPU thread being busy makes us drop a lot of frames.

Any chance with the NaCL app we receive more input events and we flush every time, while maybe we should batch a few input events together?

I have a bisect range. But it's a big one. Couldn't narrow it down due to lack of  scarlet images on xbuddy server.

10502.0.0 ~ 67.0.3375.0 ---> Bad
10489.0.0 ~ 67.0.3369.0 ---> Good

Correction: 

10502.0.0 ~ 67.0.3375.0 ---> Bad
10488.0.0 ~ 67.0.3369.0 ---> Good

Looks like it's a Chrome OS regression. Deploying Chrome 67.0.3375.0 from "Bad" onto Chrome OS 10488.0.0 from "Good" resulted in not seeing the issue.

We cannot hold the beta any longer for this, punting to stable. 

dcastagna@ here's the trace we recorded together.

Deleted: trace_test.json.gz 11.1 MB

trace_test.json.gz 11.1 MB Download

Filed crbug.com/854460 to investigate what we think is a Mali regression.

In the meanwhile mrcasey has a patch for ink that throttles the number of GL commands per frame that should mitigate the issue.

Since this is a stable release blocker, any updates on the above mentioned patch mrcasey@?

The Keep team is in the process of launching the aforementioned patch, which was checked in on Thursday. Should be out tomorrow. Reduces the impact of the slowdown but doesn't eliminate it.

Any update?

This bug is marked as a 68 stable release blocker, and we are coming up on stable in a couple weeks, so we need to get it fixed or reclassify it.

The Keep workaround is launched. The source of the issue is filed at crbug.com/854460 so I think this can be closed unless dcastagna@ has additional work for this bug.

@dcastagna do you have a separate bug for the mali regression or do you want to re-use this?

We have http://crbug.com/854460 for the mali regression. Closing this one.

[Auto-generated comment by a script] We noticed that this issue is targeted for M-68; it appears the fix may have landed after branch point, meaning a merge might be required. Please confirm if a merge is required here - if so add Merge-Request-68 label, otherwise remove Merge-TBD label. Thanks.