From the graphics perspective, we're thinking of:

1) Putting transfers in a command buffer (https://gitlab.freedesktop.org/virgl/virglrenderer/issues/51).

2) Reducing copies (udmabuf + PCI bar memory)

It would nice to know the cost of a vm-exit (in microseconds) for this.

I had to rebuild my setup due to losing my container fs again and my glbench texture_upload numbers are much higher than before:
davidriley@stretch-x:~/autotest/client/deps/glbench$ ./glbench -notemp -tests texture_upload
# board_id: Red Hat - Gallium 0.4 on virgl
# MAX_VIEWPORT_DIMS=(16384, 16384)
# GL_MAX_TEXTURE_SIZE=16384
# Running: ./glbench -notemp -tests texture_upload
# DateTime: Tue Nov 20 18:54:58 2018
@RESULT: texture_upload_luminance_teximage2d_32        =      18.27 mtexel_sec      [none]
@RESULT: texture_upload_luminance_teximage2d_128       =     248.05 mtexel_sec      [none]
@RESULT: texture_upload_luminance_teximage2d_256       =     909.93 mtexel_sec      [none]
@RESULT: texture_upload_luminance_teximage2d_512       =    2233.60 mtexel_sec      [none]
@RESULT: texture_upload_luminance_teximage2d_768       =    2451.31 mtexel_sec      [none]
@RESULT: texture_upload_luminance_teximage2d_1024      =    1987.26 mtexel_sec      [none]
@RESULT: texture_upload_luminance_teximage2d_1536      =    1858.69 mtexel_sec      [none]
@RESULT: texture_upload_luminance_teximage2d_2048      =    1705.08 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_32     =      17.39 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_128    =     258.76 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_256    =     696.15 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_512    =    1307.85 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_768    =    1931.87 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_1024   =    1889.34 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_1536   =    1850.51 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_2048   =    1792.57 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_32             =      71.66 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_128            =     773.61 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_256            =    1471.47 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_512            =    2690.15 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_768            =    2565.93 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_1024           =    2697.45 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_1536           =    2931.45 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_2048           =    3397.03 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_32          =      58.86 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_128         =     622.08 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_256         =    2006.99 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_512         =    2574.90 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_768         =    2565.87 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_1024        =    2739.74 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_1536        =    3239.20 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_2048        =    3587.97 mtexel_sec      [none]
# DateTime: Tue Nov 20 18:56:32 2018
@TEST_END

Some of it appears to be debug vs release (which I had tested before and it didn't make much difference), but that only explains a little of the difference.

I also hacked together a version of crosvm which returned TransferToHost3d commands immediately without waiting for the fences to be written by the GPU and it made another 2-2.5x improvement for small transfers and should more closely reflect the overhead caused by crosvm command processing (instead of gpu/fence processing overhead):
davidriley@stretch-x:~/autotest/client/deps/glbench$ ./glbench -notemp -tests texture_upload
# board_id: Red Hat - Gallium 0.4 on virgl
# MAX_VIEWPORT_DIMS=(16384, 16384)
# GL_MAX_TEXTURE_SIZE=16384
# Running: ./glbench -notemp -tests texture_upload
# DateTime: Tue Nov 20 19:54:06 2018
@RESULT: texture_upload_luminance_teximage2d_32        =      45.40 mtexel_sec      [none]
@RESULT: texture_upload_luminance_teximage2d_128       =     665.57 mtexel_sec      [none]
@RESULT: texture_upload_luminance_teximage2d_256       =    2043.34 mtexel_sec      [none]
@RESULT: texture_upload_luminance_teximage2d_512       =    3105.42 mtexel_sec      [none]
@RESULT: texture_upload_luminance_teximage2d_768       =    2951.39 mtexel_sec      [none]
@RESULT: texture_upload_luminance_teximage2d_1024      =    2383.29 mtexel_sec      [none]
@RESULT: texture_upload_luminance_teximage2d_1536      =    1987.47 mtexel_sec      [none]
@RESULT: texture_upload_luminance_teximage2d_2048      =    1791.65 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_32     =      45.04 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_128    =     461.71 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_256    =    1326.15 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_512    =    2370.53 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_768    =    2398.60 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_1024   =    2353.80 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_1536   =    2028.28 mtexel_sec      [none]
@RESULT: texture_upload_luminance_texsubimage2d_2048   =    1838.56 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_32             =     149.91 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_128            =    1582.76 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_256            =    2903.33 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_512            =    3195.76 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_768            =    2903.60 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_1024           =    2951.51 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_1536           =    3167.94 mtexel_sec      [none]
@RESULT: texture_upload_rgba_teximage2d_2048           =    3354.82 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_32          =     149.34 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_128         =    1553.83 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_256         =    2971.28 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_512         =    3310.81 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_768         =    2927.56 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_1024        =    2906.78 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_1536        =    3008.60 mtexel_sec      [none]
@RESULT: texture_upload_rgba_texsubimage2d_2048        =    3351.15 mtexel_sec      [none]
# DateTime: Tue Nov 20 19:55:40 2018
@TEST_END