automatically extract logs from CrOS VMs that don't come back up |
||
Issue descriptionif something breaks early in the boot such that ssh doesn't come back up, it significantly impacts our visibility into the failures. currently it requires people to manually download VM images and boot/extract log files by hand. we should improve the VM test frameworks such that, if ssh doesn't come back up, we manually mount + extract all system logs from the disk image. for example, issue 872391 is left with VMTest timeouts where the only error messages the builders have are: Could not initiate first contact with remote host Connection timed out during banner exchange Connection timed out during banner exchange
,
Sep 18
Upping this to P1. The lack of results makes it super difficult to debug failures.
,
Sep 19
How would we mount it? Using partx or something to parse the boot image? Alternatively, is it possible to mount /var/log/ in the VM as a directory shared on the host? That way, boot logs would be accessible on the host even if ssh is down on the VM. Though that might require rebuilding qemu w/ virtfs enabled, and would probably make the log's dir a special-case mount in the guest OS. So prob not worth the trouble.
,
Sep 29
use `losetup -P`, although we should have chromite libs to help here
,
Oct 1
There's also the "mount_gpt_image.sh" script, which appears to do all that for you: https://cs.corp.google.com/chromeos_public/src/scripts/mount_gpt_image.sh Though... isn't the log directory located in the "encrypted-stateful" partition of the image? Wouldn't it require decrypting to view them? |
||
►
Sign in to add a comment |
||
Comment 1 by ayatane@chromium.org
, Sep 10