All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=860264

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=8863cd19609a2178d922aef6b1e2042e5d00dde6b1aa41274d51e5ddf7ad9f9e


Bot(s) for this bug's original alert(s):

android-nexus5

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/169c49d2a40000

📍 Found significant differences after each of 3 commits.
https://pinpoint-dot-chromeperf.appspot.com/job/169c49d2a40000

Revert "[PE] Enable subsequence caching for all SVG." by chrishtr@chromium.org
https://chromium.googlesource.com/chromium/src/+/f320fa260e5b63576cb633cc0fdb746bc581d4fa
54.74 → 55.08 (+0.3351)

Turn on OOP Raster on android bots by enne@chromium.org
https://chromium.googlesource.com/chromium/src/+/d82c4fabc5cc6cbe6b475186ed9fa0216980e533
55.16 → 91.85 (+36.7)

[RootLayerScrolls] Don't use LocalFrameView as a ScrollableArea by szager@chromium.org
https://chromium.googlesource.com/chromium/src/+/566c97e6ac7ce97acde6b4de3725270971049a9f
91.59 → 92.35 (+0.7656)

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

Based on the pinpoint job, looks like enne's change is the culprit.

Investigating.

 Issue 859960  has been merged into this issue.

I was able to reproduce the mean_frame_time/css_value_type_shadow from  issue 859960  on my Linux desktop easily.

With gpu raster: 24.8ms raster time, 14.5ms gpu time
With oop raster: 3.7ms raster time, 34.3ms gpu time

The problem is that the gpu main thread is overloaded.  OOP Raster (currently) reduces parallelization by moving work from raster worker threads to the gpu main thread.  The total work is mostly a wash, but it's bottlenecked on a single thread.  We plan to fix this in the future by adding parallelization via SkDDL, which will reenable paralellization in the gpu process.

These CSS shadows also require a lot of GL work (there's ~620 TexSubImage2D calls per swap), and there's a ton of them on the page.  I don't think this is a common use cas eat all.  CCing bsalomon in case he thinks there's any inefficiency here that could be addressed.

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/14e491b2a40000

😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/14e491b2a40000

All of the runs failed. The most common error (20/20 runs) was:
SwarmingTaskError: The swarming task failed with state "TIMED_OUT".

It looks like there are really small round rects with square corners on the left side and small radii on the right side. The blur width is large relative to the size of the rrect such that they can't be "nine-patched". Because the overall size of the masks to be computed is smaller than a threshold we wind up sw rasterizing and blurring the rrects and uploading those blur results.

With a little restructuring of this code we could probably factor out the upper-left corner of the rrect and cache the masks and reuse them for all rrects with the same radii and width/height.

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/12a132b8a40000

😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/12a132b8a40000

All of the runs failed. The most common error (17/20 runs) was:
SwarmingTaskError: The swarming task failed with state "TIMED_OUT".

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/13b69742a40000

I was able to investigate several of these failures today.  Pinpoint hasn't come back yet, so I don't have traces for all cases.  I think these break down into two cases:

androidpolice_mobile(_sync_scroll)
espn_pathological

^ these both have an input_event_latency_discrepancy (and a small mean frame time, in androidpolice_mobile).  I verified locally that this is due to shader caching and parallelization by running these tests on an N4 with a page set repeat of 3 and disabling restarting the browser between runs.  See https://bugs.chromium.org/p/chromium/issues/detail?id=860177#c4 for an explanation here, and future things that can mitigate this.

web_animation_value_type_shadow

^ this is the same failure as css_value_type_shadow listed above in comment #7, where there is less parallelization

😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/13b69742a40000

All of the runs failed. The most common error (20/20 runs) was:
SwarmingTaskError: The swarming task failed with state "TIMED_OUT".

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/149caf9ea40000

😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/149caf9ea40000

All of the runs failed. The most common error (17/20 runs) was:
SwarmingTaskError: The swarming task failed with state "TIMED_OUT".

FYI, this also seems to have regressed mask_transition_animation on N5:

https://chromeperf.appspot.com/report?sid=11e5f439816ca6b06afc791b0a2326611daa482fdbb9a2bfbeb70f310e2ff855&start_rev=570057&end_rev=572180

Looking at traces, mask_transition_animation appears to be the same category as css_value_type_shadow in comment #7 (i.e. lack of parallelism causing longer frame times).

usamobile_today is in the same category as androidpolice_mobile and espn_pathological, in that there's a small mean frame time discrepancy, which is caused by a larger shader compilation regression.

Sorting by absolute and relative regressions from this bug, I think this sorts all the regressions into two buckets.  Both of these are WontFix.

The parallelization issue will be solved in the future with SkDDL and changes in the gpu process (that couldn't really be made until OOP-R landed).  The input event latency discrepancy is also something that can be addressed with SkDDL in different ways.  See https://docs.google.com/document/d/1aAmFiBlrv2wV8QE1ySdHoNi6sPspWyrJwnolk8QnfNU/edit#heading=h.2cxu0ql0ca6q for more details.