New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 796254 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: ----



Sign in to add a comment

Archive stage fails in build_image while creating factory_install_shim

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, Dec 19 2017

Issue description

Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of bmgordon@google.com

The Archive stage fails in build_image with messages related to the filesystem inside factory_install_shim.bin:

INFO    : The following images will be built            factory_install_shim.bin.
INFO    : Clearing shadow utils lockfiles under /build/winky
INFO    : Verifying that the base image does not contain a blacklisted package.
INFO    : Generating list of packages for virtual/target-os-factory-shim.
INFO    : No blacklisted packages found.
INFO    : Using image type factory_install
INFO    : Using disk layout /mnt/host/source/src/overlays/overlay-winky/scripts/disk_layout.json
WARNING: Primary GPT header is invalid
WARNING: Secondary GPT header is invalid
INFO    : Creating FS for partition 8 with format ext4.
INFO    : Creating FS for partition 12 with format vfat.
INFO    : Creating FS for partition 3 with format ext2.
INFO    : Creating FS for partition 1 with format ext4.
mount: wrong fs type, bad option, bad superblock on /dev/loop3,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail or so.
ERROR   : mount failed: device=/mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/factory_install_shim.bin target=/mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/esp format=vfat ro/rw=rw options=offset=127926272
ERROR   : Tue Dec 19 05:24:10 PST 2017
ERROR   :  PGID  PPID   PID     ELAPSED     TIME %CPU COMMAND
ERROR   : Arguments of 911:  '--from' '/mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/factory_install_shim.bin' '--rootfs_mountpt' '/mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/rootfs' '--stateful_mountpt' '/mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/stateful' '--esp_mountpt' '/mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/esp'
ERROR   : Backtrace:  (most recent call is last)
ERROR   :  mount_gpt_image.sh:478:main(), called: mount_image  
ERROR   :  mount_gpt_image.sh:393:mount_image(), called: die 'Failed to mount all partitions in /mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/factory_install_shim.bin' 
ERROR   : 
ERROR   : Error was:
ERROR   :   Failed to mount all partitions in /mnt/host/source/src/build/images/winky/R65-10229.0.0-a3/factory_install_shim.bin

This is showing up sporadically across boards:
- https://luci-milo.appspot.com/buildbot/chromeos/winky-release/1779
- https://luci-milo.appspot.com/buildbot/chromeos/daisy_spring-release/3104
- https://luci-milo.appspot.com/buildbot/chromeos/ultima-release/1774


 
Cc: vapier@chromium.org kbleicher@chromium.org bhthompson@chromium.org
Labels: -Pri-2 M-64 Pri-1
Owner: gwendal@chromium.org
We also see this on the release branch for 64:
https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos_release%2Fveyron_mickey-release_release-R64-10176.B%2F22%2F%2B%2Frecipes%2Fsteps%2FArchive%2F0%2Fstdout

And this happened to Soraka on 10225:
https://luci-milo.appspot.com/buildbot/chromeos/soraka-release/707

Gwendal, given this seems to be dying around the filesystem stuff is this something in your jurisdiction? If not please feel free to reassign.

We really need to fix this as if it happens on releases it means we will be skipping boards when it does.

Comment 2 by lepton@chromium.org, Apr 19 2018

Cc: lepton@chromium.org
See it again here:
https://luci-milo.appspot.com/buildbot/chromeos/elm-release/2127
Cc: hungte@chromium.org
Repro on https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_minnie-release/builds/2144

INFO    : Creating FS for partition 8 with format ext4.
INFO    : Creating FS for partition 12 with format vfat.
INFO    : Creating FS for partition 3 with format ext2.
INFO    : Creating FS for partition 1 with format ext4.
mount: wrong fs type, bad option, bad superblock on /dev/loop3,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail or so.
ERROR   : mount failed: device=/mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/factory_install_shim.bin target=/mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/esp format=vfat ro/rw=rw options=offset=127926272
ERROR   : Tue Apr 24 06:25:46 PDT 2018
ERROR   :  PGID  PPID   PID     ELAPSED     TIME %CPU COMMAND
ERROR   : Arguments of 926:  '--from' '/mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/factory_install_shim.bin' '--rootfs_mountpt' '/mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/rootfs' '--stateful_mountpt' '/mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/stateful' '--esp_mountpt' '/mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/esp'
ERROR   : Backtrace:  (most recent call is last)
ERROR   :  mount_gpt_image.sh:478:main(), called: mount_image  
ERROR   :  mount_gpt_image.sh:393:mount_image(), called: die 'Failed to mount all partitions in /mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/factory_install_shim.bin' 
ERROR   : 
ERROR   : Error was:
ERROR   :   Failed to mount all partitions in /mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/factory_install_shim.bin
INFO    : Unmounting image from /mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/stateful and /mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/rootfs
INFO    : Unmounting temporary rootfs /mnt/host/source/src/build/images/veyron_minnie/R68-10613.0.0-a3/rootfs//build/rootfs.


Looking at the serial console on the VM:
Apr 24 06:25:46 cros-beefy405-c2 kernel: [27993.358796] EXT4-fs (loop2): mounted filesystem with ordered data mode. Opts: (null)
[27993.469294] FAT-fs (loop3): bogus number of reserved sectors
Apr 24 06:25:46 cros-beefy405-c2 kernel: [27993.469294] FAT-fs (loop3): bogus number of reserved sectors
Apr 24 06:25:46 cros-beefy405-c2 kernel: [27993.475175] FAT-fs (loop3): Can't find a valid FAT filesystem

The BIOS parameter block (BPB) is not valid, part of the partition super block.

I can not find the factory image in the artifact, but the EFI partition may not be formatted properly or is not present at all.

Comment 4 by hungte@chromium.org, Apr 24 2018

Cc: stimim@chromium.org
The factory shim is following same process as build_image so I'm not sure what may go wrong... 

Comment 5 by vapier@chromium.org, Apr 25 2018

Components: Infra>Client>ChromeOS>Build
Labels: OS-Chrome
Summary: Archive stage fails in build_image while creating factory_install_shim (was: Archive stage fails in build_image)

Comment 6 by hungte@chromium.org, Apr 25 2018

does this only fail on devices with cros_embedded set? (b/72749926)
#6: no, elm, veyron_minnie do not have cros_embedded set.

Comment 8 by hungte@chromium.org, Apr 26 2018

INFO    : Creating FS for partition 12 with format vfat.

The build script did create EFI (12) before starting to mount.
And the message indicates a "sudo mkfs.vfat" should be called after that, with a mount (based on loop) then umount.

In kernel source, the bogus number means reserved sectors = 0, but mkfs.vfat should create at least 1 (for FAT16 - which is the default selected value for our EFI partitions).

I wonder if there's some cache/sync/flush issue, or mount_gpt_image has some bug, or the calculation of offset in mk_fs has overflow.

Comment 9 by hungte@chromium.org, Apr 26 2018

well, $(()) seems supporting 64 bit integer so overflow should not be a problem.

I think we need to find a way to reproduce, or get a such corrupted image to analyze what happened.

Or, meanwhile, we can print additional info in "Create FS ..." for offset and size it has created, to see if the EFI was then zeroed by creation of other partitions.
Cc: nnita@chromium.org dchan@chromium.org abod...@chromium.org cindyb@chromium.org
 Issue 850555  has been merged into this issue.
We are seeing this on the 67 branch also, has anyone had a chance to look at this recently?
Status: Assigned (was: Available)

Sign in to add a comment