Archive stage fails in build_image while creating factory_install_shim
Project Member Reported by sheriff-...@appspot.gserviceaccount.com, Dec 19 2017
Filed by email@example.com on behalf of firstname.lastname@example.org The Archive stage fails in build_image with messages related to the filesystem inside factory_install_shim.bin: INFO : The following images will be built factory_install_shim.bin. INFO : Clearing shadow utils lockfiles under /build/winky INFO : Verifying that the base image does not contain a blacklisted package. INFO : Generating list of packages for virtual/target-os-factory-shim. INFO : No blacklisted packages found. INFO : Using image type factory_install INFO : Using disk layout /mnt/host/source/src/overlays/overlay-winky/scripts/disk_layout.json WARNING: Primary GPT header is invalid WARNING: Secondary GPT header is invalid INFO : Creating FS for partition 8 with format ext4. INFO : Creating FS for partition 12 with format vfat. INFO : Creating FS for partition 3 with format ext2. INFO : Creating FS for partition 1 with format ext4. mount: wrong fs type, bad option, bad superblock on /dev/loop3, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. ERROR : mount failed: device=/mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/factory_install_shim.bin target=/mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/esp format=vfat ro/rw=rw options=offset=127926272 ERROR : Tue Dec 19 05:24:10 PST 2017 ERROR : PGID PPID PID ELAPSED TIME %CPU COMMAND ERROR : Arguments of 911: '--from' '/mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/factory_install_shim.bin' '--rootfs_mountpt' '/mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/rootfs' '--stateful_mountpt' '/mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/stateful' '--esp_mountpt' '/mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/esp' ERROR : Backtrace: (most recent call is last) ERROR : mount_gpt_image.sh:478:main(), called: mount_image ERROR : mount_gpt_image.sh:393:mount_image(), called: die 'Failed to mount all partitions in /mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/factory_install_shim.bin' ERROR : ERROR : Error was: ERROR : Failed to mount all partitions in /mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/factory_install_shim.bin This is showing up sporadically across boards: - https://luci-milo.appspot.com/buildbot/chromeos/winky-release/1779 - https://luci-milo.appspot.com/buildbot/chromeos/daisy_spring-release/3104 - https://luci-milo.appspot.com/buildbot/chromeos/ultima-release/1774
Dec 20 2017,
We also see this on the release branch for 64: https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos_release%2Fveyron_mickey-release_release-R64-10176.B%2F22%2F%2B%2Frecipes%2Fsteps%2FArchive%2F0%2Fstdout And this happened to Soraka on 10225: https://luci-milo.appspot.com/buildbot/chromeos/soraka-release/707 Gwendal, given this seems to be dying around the filesystem stuff is this something in your jurisdiction? If not please feel free to reassign. We really need to fix this as if it happens on releases it means we will be skipping boards when it does.
See it again here: https://luci-milo.appspot.com/buildbot/chromeos/elm-release/2127
Repro on https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-release/builds/2144 INFO : Creating FS for partition 8 with format ext4. INFO : Creating FS for partition 12 with format vfat. INFO : Creating FS for partition 3 with format ext2. INFO : Creating FS for partition 1 with format ext4. mount: wrong fs type, bad option, bad superblock on /dev/loop3, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. ERROR : mount failed: device=/mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/factory_install_shim.bin target=/mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/esp format=vfat ro/rw=rw options=offset=127926272 ERROR : Tue Apr 24 06:25:46 PDT 2018 ERROR : PGID PPID PID ELAPSED TIME %CPU COMMAND ERROR : Arguments of 926: '--from' '/mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/factory_install_shim.bin' '--rootfs_mountpt' '/mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/rootfs' '--stateful_mountpt' '/mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/stateful' '--esp_mountpt' '/mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/esp' ERROR : Backtrace: (most recent call is last) ERROR : mount_gpt_image.sh:478:main(), called: mount_image ERROR : mount_gpt_image.sh:393:mount_image(), called: die 'Failed to mount all partitions in /mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/factory_install_shim.bin' ERROR : ERROR : Error was: ERROR : Failed to mount all partitions in /mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/factory_install_shim.bin INFO : Unmounting image from /mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/stateful and /mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/rootfs INFO : Unmounting temporary rootfs /mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/rootfs//build/rootfs. Looking at the serial console on the VM: Apr 24 06:25:46 cros-beefy405-c2 kernel: [27993.358796] EXT4-fs (loop2): mounted filesystem with ordered data mode. Opts: (null) [27993.469294] FAT-fs (loop3): bogus number of reserved sectors Apr 24 06:25:46 cros-beefy405-c2 kernel: [27993.469294] FAT-fs (loop3): bogus number of reserved sectors Apr 24 06:25:46 cros-beefy405-c2 kernel: [27993.475175] FAT-fs (loop3): Can't find a valid FAT filesystem The BIOS parameter block (BPB) is not valid, part of the partition super block. I can not find the factory image in the artifact, but the EFI partition may not be formatted properly or is not present at all.
The factory shim is following same process as build_image so I'm not sure what may go wrong...
does this only fail on devices with cros_embedded set? (b/72749926)
#6: no, elm, veyron_minnie do not have cros_embedded set.
INFO : Creating FS for partition 12 with format vfat. The build script did create EFI (12) before starting to mount. And the message indicates a "sudo mkfs.vfat" should be called after that, with a mount (based on loop) then umount. In kernel source, the bogus number means reserved sectors = 0, but mkfs.vfat should create at least 1 (for FAT16 - which is the default selected value for our EFI partitions). I wonder if there's some cache/sync/flush issue, or mount_gpt_image has some bug, or the calculation of offset in mk_fs has overflow.
well, $(()) seems supporting 64 bit integer so overflow should not be a problem. I think we need to find a way to reproduce, or get a such corrupted image to analyze what happened. Or, meanwhile, we can print additional info in "Create FS ..." for offset and size it has created, to see if the EFI was then zeroed by creation of other partitions.
Seeing this on falco release: https://luci-logdog.appspot.com/v/?s=chromeos/bb/chromeos/falco-release/3505/+/recipes/steps/Archive/0/stdout
Issue 850555 has been merged into this issue.
We are seeing this on the 67 branch also, has anyone had a chance to look at this recently?
Sign in to add a comment