A few options here that I see:

1) Do nothing, wait for Debian buster. Although that isn't a particularly new version of mesa: https://packages.debian.org/source/buster/mesa

But, the maintenance burden here is zero.

2) Set apt preferences to pull in newer versions of mesa from Debian. So for now, buster. When buster goes stable, we pull in from bullseye.

Maintenance burden: low, but more likelihood of breakage leaking in from the next release.

3) Build our own mesa and distribute it via apt.

Maintenance burden: high, we will need to add tests to ensure that we can push newer versions of the apt packages. The current CI flow for apt is an all-or-nothing push, we'll need to invest in improving that. I'd like someone with graphics experience to own at least part of that (davidriley or zachr?)

We are more interested in getting the very latest patches from mesa ToT for performance/stability reasons so we can see what impact they have. So what's in buster isn't recent enough for example. I guess in the meantime maybe we can overwrite with our own mesa for testing? Eventually things will roll out on their own (maybe getting what's in sid is a good compromise?)

Maybe then it would make sense for us to test against the sid mesa packages and then mirror them ourselves? Then we'll be using Debian built and maintained packages, which is a big plus.

Right now sid is the same as buster, because of where we are in the debian cycle. I suppose things will unblock later...

One other thing of note is that we'll need 32-bit i386 versions of the packages if we want full compatability (ie if we want to run steam) along with updated versions of drm, wayland-protocols, and wayland (which were packages I needed to update to build ToT mesa on stretch).

I spent a bit of time unsuccessfully going down two paths today:
- tried to update to sid's mesa ran into issues once it got to the point of needing a new libc which apt started warning against upgrading
- tried to build from source ToT mesa for i386 ran into issues trying to get i386 build-dep going (specifically python-moko:i386)

Both of these paths were things I wasn't really sure how to do, so it's quite possible that I was just doing things incorrectly.

So using zachr's handy notes (https://docs.google.com/document/d/16-Qkk09xQVXnEGcRwWxafu2b7cZKPdzR0Qw1qklS1sk/edit#) I built my own debian mesa packages for stretch, but then ran into issues installing them due to not having the right dependencies.

I added stretch-backports to my container, ran into some other issues with how we've pinned some packages for virtwl (eg libwayland) to an older version.  After figuring those out, I realized that stretch-backports actually has mesa 18.2.8 (Dec 27, 2018).  So if we add stretch-backports, set our pins right, we should get that for free without compiling anything or maintaining any packages.

The downside is that 18.2.8 is a bug fix release and 18.2.0 is actually from September 7, 2018.  That's still much better than 13.0.6, but still trailing ToT potentially by quite a bit.

So we need to make a decision between getting 18.2.X for pretty much free, or back to more work to get ToT (which I think might still be pretty reasonable once we sort out these issues).  

I got 18.2.8 from stretch-backports working, albeit with some trickiness regarding upgrading.

I also built 18.3.2 mesa using a modified version of their build branches.  Instructions and links to prebuilt packages are available for both amd64 and i386 in the care and feeding document: go/crosvm-gpu-care

The process to do the build wasn't too bad, but I definitely cut some corners in how I put together the stretch-unstable branch I used.