I've started investigating this (I have a local script which wraps around oatdump).

I have an initial list of class verification errors available here [1]. Class verification errors depend on the device SDK_INT, so I have results from a few different OS levels on separate spreadsheet tabs.

TL;DR we have 71 classes which fail compile time verification on L, most of which we own and can fix (some are libraries, such as GMS and support lib). Big optimization potential :)

This does record does not yet include clank classes (I've only looked at SystemWebViewGoogle.apk so far). I'll get to clank eventually, and add those lists in more sheets tabs.

[1] https://docs.google.com/spreadsheets/d/1syRh1djhmMHJi62snGlJ7ixk8HljIwcZQxOuN4dqDCY/edit?usp=sharing

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/1487b4f7240000

📍 Job complete. See results below.
https://pinpoint-dot-chromeperf.appspot.com/job/1487b4f7240000

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/15915ac7240000

📍 Job complete. See results below.
https://pinpoint-dot-chromeperf.appspot.com/job/15915ac7240000

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/11c96d20a40000

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/143701cb240000

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/1037da77240000

📍 Job complete. See results below.
https://pinpoint-dot-chromeperf.appspot.com/job/143701cb240000

📍 Job complete. See results below.
https://pinpoint-dot-chromeperf.appspot.com/job/11c96d20a40000

📍 Job complete. See results below.
https://pinpoint-dot-chromeperf.appspot.com/job/1037da77240000

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f50fc4d425345286e65488344fd85981bef36c94

commit f50fc4d425345286e65488344fd85981bef36c94
Author: Nate Fischer <ntfschr@chromium.org>
Date: Thu Jul 12 19:10:16 2018

AW: trace events for WindowAndroid verification issues

No change to logic.

This adds trace events inside AwContents to track WindowAndroid-related
class verification errors. Specifically, this should illustrate the
following verification errors:

 * WindowAndroid verification errors should cause activityFromContext to
   take longer than necessary.
 * ActivityWindowAndroid fails verification (because its superclass
   WindowAndroid fails verification), causing its constructor to take
   longer than necessary.

R=torne@chromium.org

Bug: 855796
Test: tools/perf/run_benchmark system_health.webview_startup --browser android-webview-google
Change-Id: I6633496fd672300216dfa6425910d19a7ae9e3da
Reviewed-on: https://chromium-review.googlesource.com/1132479
Reviewed-by: Richard Coles <torne@chromium.org>
Commit-Queue: Nate Fischer <ntfschr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#574663}
[modify] https://crrev.com/f50fc4d425345286e65488344fd85981bef36c94/android_webview/java/src/org/chromium/android_webview/AwContents.java

Issue 867160 has been merged into this issue.

I sent out a writeup to the team last week, covering the current state of class verification in WebView and Clank, and discussing potential solutions. We got some great feedback from team members and other Android teams.

The doc is still Google-only (sorry), but I'm happy to disclose non-secret info if anyone is particularly interested.

Link to write-up: https://docs.google.com/document/d/1i5cPpP9tXk6o8FHRO_HyLyvUrVArNo6BCOSoOqk80l8/edit?usp=sharing

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/008b13831c202c11288a0723843db26ba6e3ba08

commit 008b13831c202c11288a0723843db26ba6e3ba08
Author: Nate Fischer <ntfschr@chromium.org>
Date: Tue Oct 09 21:29:29 2018

Android: script to list class verification errors

This is a wrapper script around oatdump to make it easier to list class
verification errors.

This handles API-level specific details and deobfuscates Java class
names.

Bug: 855796
Test: Manual
Change-Id: I9be7c60dc352da45e2ec067d1d61519c60caf8fc
Reviewed-on: https://chromium-review.googlesource.com/c/1232690
Commit-Queue: Nate Fischer <ntfschr@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Reviewed-by: Changwan Ryu <changwan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#598085}
[modify] https://crrev.com/008b13831c202c11288a0723843db26ba6e3ba08/build/android/PRESUBMIT.py
[add] https://crrev.com/008b13831c202c11288a0723843db26ba6e3ba08/build/android/list_class_verification_failures.py
[add] https://crrev.com/008b13831c202c11288a0723843db26ba6e3ba08/build/android/list_class_verification_failures_test.py

I have a couple thoughts for next steps:

 1. We could run this script after our startup benchmark. This would enable us to correlate spikes in verification errors with spikes in startup time, assuming we can reasonably save the script output somewhere.
 2. We could add this to an experimental trybot, similar to android-binary-size. The bot would fail whenever we introduce a new verification failure (assuming we can diff script results before/after).

perezju@ in terms of #1, is it feasible to extend benchmarks to also surface other data? Such as, if the script produces an integer count for the number of unverified classes, is it possible to include this in the dashboard?

Not sure if the best thing is to tie the benchmark with your script. Both things seem to be independent of each other?

It should be possible, however, to get your script running on some bot and uploading data to the dashboard, similar to how it's done for apk binary size.

+agrieve who has done a similar thing in the past.