Investigate using sparse raw file for VM disks rather than qcow2 |
|||
Issue descriptiondgreid@ mentioned that since we have hole punching/discard support in crosvm, we don't necessarily need the image files to be qcow2 format. We could just use sparse raw files to get the desired behavior of returning disk space to the host system when files inside the VM guest are deleted. The guest VM filesystem (BTRFS) can provide snapshot features, so implementing this in the qcow layer inside crosvm is not necessary. This would also allow a major simplification in the disk I/O path in crosvm and should hopefully avoid the current performance issues inherent in having to update qcow ref/l1/l2 tables on each fsync. I'm investigating this now, and as a first step, I'm working on a more generic import/export command; currently `vmc export` just copies the VM disk image byte for byte, but we can instead generate a qcow2 image on demand. This will also help save space when exporting to removable disks with filesystems that don't support sparse files (e.g. FAT), since we can write a minimal qcow2 image that only contains populated clusters and no host filesystem holes (even if the source file is a raw file or a qcow2 file that has had data written and then discarded, creating holes). Also, this lets us maintain the user-visible interface (`vmc export` can still export qcow2 files) while allowing us to change the underlying file format.
,
Oct 15
,
Oct 19
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform2/+/dca69966821da86741fc324e8187adf87cb58267 commit dca69966821da86741fc324e8187adf87cb58267 Author: Daniel Verkamp <dverkamp@chromium.org> Date: Fri Oct 19 02:01:18 2018 vm_tools: concierge: add raw disk image support Adjust GetDiskPathFromName() so that it can find both *.img and *.qcow2 files, and add an optional parameter to specify what type is preferred when creating a new image. BUG= chromium:893380 TEST=Test `vmc start` and `vmc destroy` on device Change-Id: Ifacd4eead43935d434526533bf8d52f21448ee65 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1275425 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Stephen Barber <smbarber@chromium.org> Reviewed-by: Dylan Reid <dgreid@chromium.org> [modify] https://crrev.com/dca69966821da86741fc324e8187adf87cb58267/vm_tools/concierge/service.cc
,
Oct 19
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/crosvm/+/f02fdd1f663760fc884073ea74fffb39a3f4ea4c commit f02fdd1f663760fc884073ea74fffb39a3f4ea4c Author: Daniel Verkamp <dverkamp@chromium.org> Date: Fri Oct 19 19:19:52 2018 main: autodetect disk image type Make --disk and --rwdisk automatically distinguish between qcow2 and raw disk images. --qcow and --rwqcow are kept as aliases for compatibility. BUG= chromium:893380 TEST=Boot crosvm with both raw and qcow2 disk images. Change-Id: I5b572626ca5ab894c78454f59355f27d552cbf7d Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1275185 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Stephen Barber <smbarber@chromium.org> Reviewed-by: Zach Reizner <zachr@chromium.org> [modify] https://crrev.com/f02fdd1f663760fc884073ea74fffb39a3f4ea4c/src/linux.rs [modify] https://crrev.com/f02fdd1f663760fc884073ea74fffb39a3f4ea4c/src/main.rs [modify] https://crrev.com/f02fdd1f663760fc884073ea74fffb39a3f4ea4c/qcow/src/qcow.rs
,
Oct 19
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform2/+/d7153da2e057f4270436871dbb71d811a00938ed commit d7153da2e057f4270436871dbb71d811a00938ed Author: Daniel Verkamp <dverkamp@chromium.org> Date: Fri Oct 19 19:19:53 2018 vm_tools: concierge: use qcow_utils to export disk Rather than blindly copying the disk image when exporting, use the qcow_utils convert_to_qcow2() function to write a new qcow2 image with only the necessary cluster data. This will also cause the output file to have all of the data packed at the beginning of the file, even if the input file is sparse. CQ-DEPEND=CL:1272060 BUG= chromium:893380 TEST=`vmc export` a qcow2 image on nami Change-Id: I16522f0c9c2768425a8891899fd996cc15ae6244 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1272116 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Stephen Barber <smbarber@chromium.org> [modify] https://crrev.com/d7153da2e057f4270436871dbb71d811a00938ed/vm_tools/concierge/service.cc
,
Oct 23
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform2/+/75746d2d66af031f797b15f97681ee3510bd9e1d commit 75746d2d66af031f797b15f97681ee3510bd9e1d Author: Daniel Verkamp <dverkamp@chromium.org> Date: Tue Oct 23 20:01:27 2018 vm_tools: concierge: remove image type in StartVm Let crosvm automatically detect the correct disk type now that --disk/--rwdisk work for both raw and qcow2 images. CQ-DEPEND=CL:1275185 BUG= chromium:893380 TEST=vmc start termina Change-Id: Id4c9ac40328c4e7e115558b127206ab580489528 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1290295 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Stephen Barber <smbarber@chromium.org> [modify] https://crrev.com/75746d2d66af031f797b15f97681ee3510bd9e1d/vm_tools/concierge/virtual_machine.cc [modify] https://crrev.com/75746d2d66af031f797b15f97681ee3510bd9e1d/vm_tools/concierge/virtual_machine.h [modify] https://crrev.com/75746d2d66af031f797b15f97681ee3510bd9e1d/vm_tools/concierge/service.cc [modify] https://crrev.com/75746d2d66af031f797b15f97681ee3510bd9e1d/vm_tools/concierge/client.cc
,
Oct 23
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform2/+/e00c068afc50a73a6a94ce0b661d3b786035d637 commit e00c068afc50a73a6a94ce0b661d3b786035d637 Author: Daniel Verkamp <dverkamp@chromium.org> Date: Tue Oct 23 20:01:27 2018 vm_tools: concierge: pass image type to StartTerminaVm Allow concierge_client users to specify the disk image type to create. Previously it was hard-coded to qcow2. BUG= chromium:893380 TEST=Create raw and qcow2 disks via concierge_client --start_termina_vm Change-Id: I8a419ef06a845572c9cd8bf26945fab23a3adcd7 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1295189 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Reviewed-by: Stephen Barber <smbarber@chromium.org> [modify] https://crrev.com/e00c068afc50a73a6a94ce0b661d3b786035d637/vm_tools/concierge/client.cc
,
Nov 1
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform2/+/4dc89671ffa7028d8414fbdf5df7452b4616a6b1 commit 4dc89671ffa7028d8414fbdf5df7452b4616a6b1 Author: Daniel Verkamp <dverkamp@chromium.org> Date: Thu Nov 01 04:59:16 2018 system_api: concierge: add DISK_TYPE_AUTO value This new DiskImageType value will be used to allow concierge to choose the best image type for a given filesystem and kernel. BUG= chromium:893380 TEST=emerge-nami system_api Change-Id: I46aaa8910b4a62dcaca71f9745a6cd455aa582ea Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1309080 Reviewed-by: Stephen Barber <smbarber@chromium.org> [modify] https://crrev.com/4dc89671ffa7028d8414fbdf5df7452b4616a6b1/system_api/dbus/vm_concierge/service.proto
,
Nov 1
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform2/+/a96620038cc97b7d03e8716238fd105641ec1b49 commit a96620038cc97b7d03e8716238fd105641ec1b49 Author: Daniel Verkamp <dverkamp@chromium.org> Date: Thu Nov 01 04:59:17 2018 vm_tools: concierge: add auto disk type option Add a new DISK_IMAGE_AUTO value to DiskImageType that lets concierge choose the image type. This lets concierge pick the best image type for a given filesystem and kernel based on whether fallocate() with FALLOC_FL_PUNCH_HOLE is supported, while still allowing the flexibility to choose raw or qcow2 explicitly if needed. This changes the default image type created via concierge_client to auto, which prefers raw over qcow2. VMs created through the Chromium UI will still use qcow2 for now since crostini_manager explicitly passes DISK_IMAGE_QCOW2. BUG= chromium:893380 TEST=concierge_client --start_termina_vm on kevin and cyan Change-Id: I2667e8f7b18a7b03ca8551ff7dc0e9362d6a91c0 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1309293 Reviewed-by: Stephen Barber <smbarber@chromium.org> [modify] https://crrev.com/a96620038cc97b7d03e8716238fd105641ec1b49/vm_tools/concierge/service.cc [modify] https://crrev.com/a96620038cc97b7d03e8716238fd105641ec1b49/vm_tools/concierge/client.cc
,
Nov 2
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b195593de7d2e8738cfd606c3f65f486d09bb22e commit b195593de7d2e8738cfd606c3f65f486d09bb22e Author: Daniel Verkamp <dverkamp@chromium.org> Date: Fri Nov 02 16:27:43 2018 Crostini: use automatic disk image type This lets concierge choose the appropriate disk image type for the kernel and filesystem. Bug: 893380 Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Change-Id: I2b8d9ae847e5d2a87949a2e7b9ef965d038e11f2 Reviewed-on: https://chromium-review.googlesource.com/c/1312568 Reviewed-by: Nicholas Verne <nverne@chromium.org> Cr-Commit-Position: refs/heads/master@{#604953} [modify] https://crrev.com/b195593de7d2e8738cfd606c3f65f486d09bb22e/chrome/browser/chromeos/crostini/crostini_manager.cc
,
Nov 2
As of the commit above, we will now create raw disk images by default on systems that support FALLOC_FL_PUNCH_HOLE. Other systems will still get qcow2 images.
,
Nov 16
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/1cc15a200b59a0e645ce8453a9f1ad039b1139be commit 1cc15a200b59a0e645ce8453a9f1ad039b1139be Author: Daniel Verkamp <dverkamp@chromium.org> Date: Fri Nov 16 13:01:58 2018 tast-tests: Use auto disk type in Crostini tests Switch the disk image type to DISK_IMAGE_AUTO, which will let concierge select the best format for the kernel and filesystem (raw on systems that support FALLOC_FL_PUNCH_HOLE, qcow2 on older kernels that don't support it). The Chrome browser code and concierge_client/vmc tools have already switched to using auto, so we should make the tests use it as well to test what we are shipping. BUG= chromium:893380 TEST=tast run vm.CrostiniStartEverything on nami and kevin Change-Id: I79ac332225ecd6b379fe2859afe9bdcd01784dad Signed-off-by: Daniel Verkamp <dverkamp@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/1334490 Reviewed-by: Jeffrey Kardatzke <jkardatzke@google.com> Reviewed-by: Stephen Barber <smbarber@chromium.org> [modify] https://crrev.com/1cc15a200b59a0e645ce8453a9f1ad039b1139be/src/chromiumos/tast/local/vm/concierge.go |
|||
►
Sign in to add a comment |
|||
Comment 1 by bugdroid1@chromium.org
, Oct 15