Could you try booting with this patch applied and see if it makes any performance difference? https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1250022

No noticeable difference.

localhost /mnt/stateful_partition/home/.shadow/f66ea3860115aa350cc20760139aea056
25a4169/mount/root/crosvm # time concierge_client --start_termina_vm --cryptohome_id=f66ea3860115aa350cc20760139aea05625a4169 --name=tatl
[INFO:client.cc(670)] Starting Termina VM 'tatl'
[INFO:client.cc(371)] Creating disk image
[INFO:client.cc(759)] Started Termina VM with
[INFO:client.cc(760)]     ip address: 100.115.92.6
[INFO:client.cc(761)]     vsock cid:  14
[INFO:client.cc(762)]     process id: 146
[INFO:client.cc(763)]     seneschal server handle: 12

real	1m4.857s
user	0m0.006s
sys	0m0.004s

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
loop0             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
loop1             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
loop2             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
loop3             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
loop4             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
loop5             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
loop6             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0           0.00   286.40    2.20  359.30    11.20  9528.40    52.78     0.96    2.63    0.18    2.64   0.84  30.40
mmcblk0rpmb       0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot1      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot0      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zram0             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.42    0.00    4.22    6.69    0.00   71.68

<triage>Assigning to owner</triage>

My VM is now non-booting which appears to be from a timeout.  If I backed out a whole string of qcow changes, things boot again:
80b27f3 (HEAD -> xwayland-gpu2) Revert "qcow: Cache address and refcount tables"
29c5990 Revert "qcow: Add accessors for more file state"
be81a27 Revert "qcow: buffer reads and writes in QcowRawFile"
3a082d1 Revert "qcow: optimize sync_caches to avoid extra writes"
ea5c624 Revert "qcow: track deallocated clusters as unreferenced"
b1570f2 qcow: track deallocated clusters as unreferenced
46f70e8 qcow: optimize sync_caches to avoid extra writes
eaaf3e1 qcow: buffer reads and writes in QcowRawFile
39401ff qcow: Add accessors for more file state
32e17bc qcow: Cache address and refcount tables

80b27f3 - boots okay
localhost /mnt/stateful_partition # concierge_client --start_termina_vm --cryptohome_id=f66ea3860115aa350cc20760139aea05625a4169 --name=tatl
[INFO:client.cc(670)] Starting Termina VM 'tatl'
[INFO:client.cc(371)] Creating disk image
[INFO:client.cc(759)] Started Termina VM with
[INFO:client.cc(760)]     ip address: 100.115.92.6
[INFO:client.cc(761)]     vsock cid:  7
[INFO:client.cc(762)]     process id: 68
[INFO:client.cc(763)]     seneschal server handle: 5

b1570f2 - fails to boot
localhost /mnt/stateful_partition # concierge_client --start_termina_vm --cryptohome_id=f66ea3860115aa350cc20760139aea05625a4169 --name=tatl
[INFO:client.cc(670)] Starting Termina VM 'tatl'
[INFO:client.cc(371)] Creating disk image
[ERROR:object_proxy.cc(580)] Failed to call method: org.chromium.VmConcierge.StartVm: object_path= /org/chromium/VmConcierge: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
[ERROR:client.cc(739)] Failed to send dbus message to concierge service

I'm copying my qcow2 image up for debugging, but I think regardless of what we turn out we might want to increase the timeout if we can.  

Just checking in on this - has it reproduced after switching to a raw block device?

I believe the "writing lots of data" part of the original description is probably caused by I/O to the VM disk image filling up the host kernel's page cache and causing writeback of dirty pages, so I don't think that is specific to qcow.

I investigated opening the disk images with O_DIRECT, which would cause I/O to bypass the host page cache, but it's not easily possible to do this in general because our disk emulation is exposing 512-byte sectors, whereas the host may require 4096-byte aligned I/O when opened with O_DIRECT.  Using O_DIRECT would also potentially regress performance, since we lose any benefit of the host page cache.  (See https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/1324772 for the WIP O_DIRECT implementation.)

I haven't run into these issues recently, but:
a) I haven't been restarting my VM as much recently
b) I don't think I've installed everything in my VM that I had before