All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=860177

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=61632ee55dba801935b6708280bc5583047f9f1c1f3be9f20c1ac3413df7da91


Bot(s) for this bug's original alert(s):

android-nexus5X

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/11992788a40000

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/11992788a40000

Turn on OOP Raster on android bots by enne@chromium.org
https://chromium.googlesource.com/chromium/src/+/d82c4fabc5cc6cbe6b475186ed9fa0216980e533
193 → 505.4 (+312.4)

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

Khushal did some investigation on this, and this regression (as well as similar regressions on smoothness bots) are due to oop raster removing some preemption on the gpu main thread.

This issue is that the gpu process finishes the raster of all the required work for the renderer.  Then, while waiting for the display compositor to submit a frame, it starts on non-required work.  While it's blocked on creating shaders for this non-required work, the display compositor frame comes in and is ready to draw, but it has to wait for the non-required raster task to finish.

One of the more expensive parts in the gpu process of rastering a new page is creating shaders.  Previously, the gpu scheduler could interrupt between shader creation, which limited the time that the display compositor drawing had to wait.  However, with oop raster the gpu scheduler can only interrupt at the raster task level, which means that some larger number (usually like 3-6) of shaders need to be created, which increases the latency discrepancy for the first time these shaders are used.  Because bots restart the browser between tests and do a cold start, this is the worst case for this scenario.

In practice, Skia generally only uses a small number of shaders, on the order of 25-50, roughly.  However, it's hard to know ahead of time which of these shaders to create.  We chatted with the Skia team, and shader creation is deep in the pipeline, so it would be incredibly tricky to attempt to yield and continue Skia execution to replicate the previous interruptible behavior.

There are a few future mitigations for this issue:
-Khushal's https://chromium-review.googlesource.com/c/1114287/ to share GrContext among raster decoders, means that the shader cache is also shared.  This means that the latency is only for the first renderer.
-There are plans to persist shader to disk, to help with the cold start:  http://crbug.com/840559  (however, this will not help the bots, as they clear the caches).  This is blocked on some bugs with the Skia persistent cache API.
-future work to use SkDDL could allow a return the previous interruptible behavior as the SkDDL could potentially explicitly enumerate all the required shaders and allow a caller to create them one by one, see: https://docs.google.com/document/d/14HLjxZIpLIYG8e1gF0bzgQ3bqDx3swX89VCp7gWLXaM/edit

I think this regression is shippable in the short term.  This latency discrepancy really happens once on the first tab in the browser, and then once the shaders are created then everything is smooth again for all future tabs.  We also have ways in the future to mitigate this.

It isn't clear to me what to do about the bots.  It seems bad for noise to have one bad frame here, but I don't see any way to fix that startup cost.  GPU raster already created this problem that wasn't there before.  It makes me wonder if it would be worth differentiating cold start from warm start.

 Issue 861658  has been merged into this issue.