We will want to test each WebView Support Library release against each of the Android versions it supports.
The most straight-forward way to do this would be to run existing CTS-tests against the support library (rebuiding those tests against the support library classes).

It is not obvious what WebView versions we should test each Support library version / Android version against. We probably want to test each WebView version that is shipped in an Android system image (to ensure the support library supports features from the Android version that WebView version shipped in, on lower Android versions).

Furthermore, we will probably release one new Support Library version per Android version.

So, we will test 
1. each support library version, against
2. each Android version, against
3. each WebView version shipped in an Android version.

This means that as we release more Android versions the number of test configurations we will want to run will grow as (given that A is the number of Android versions) A^3. So, in the future, even if we only support 4 Android versions at a time, we will have to test 4^3 = 64 different configurations. This seems like way to much (maybe I've overestimated the number of versions of Support Lib / Android / WebView we need to test?).


A comment on the testability of the support library: we can duplicate our current CTS tests to ensure features are working on e.g. Nexus phones, but we can't make these support library tests part of CTS tests (since they will explicitly function on older devices which we can no longer change to pass the test). So it's difficult to know how well the support library will work on devices which we are not explicitly testing on. Maybe we should have extra testing with devices we don't usually test with?