I can only take a look after Tuesday, so please find an owner / mark it experimental and wait for me to be free.

Assigning to deputy as this is P0 for CQ.

I took a look and haven't made what would be called "progress", but some findings anyway.

The error is that /boot/vmlinuz is missing (obviously).

https://cs.corp.google.com/chromeos_public/src/platform2/installer/chromeos_postinst.cc?type=cs&q=%22Legacy+PostInstall%22&l=509
https://cs.corp.google.com/chromeos_public/src/platform2/installer/chromeos_legacy.cc?type=cs&q=RunLegacyUBootPostInstall&l=44
https://cs.corp.google.com/chromeos_public/src/platform2/installer/chromeos_legacy.cc?type=cs&q=RunLegacyUBootPostInstall&l=17

I downloaded the workspace and found no /boot/vmlinuz on the DUT (or the moblab).

Just for fun, I downloaded the most recent green betty and moblabvm images to set up my own workspace.  It fails.

It fails because the moblab image.zip was missing boot_images/vmlinuz  After hard linking vmlinuz.bin to boot_image/vmlinuz I was able to create the workspace.

I don't know if this is related, but it seems suspicious.

Anyway, there's no /boot/vmlinuz on the DUT in my new workspace either, so much for that.  Perhaps we're not supposed to be taking the kBiosTypeLegacy in the postinstall program in the first place.

Notes, postint called like

/postinst / /dev/sda3

postinst call is

 /postinst /dev/sda3

That calls cros_installer

 chroot . /usr/bin/cros_installer postinst "/" "$@"

https://cs.corp.google.com/chromeos_public/src/platform2/installer/cros_installer_main.cc?q=cros_installer&sq=package:%5Echromeos_public$&dr=C&l=19

cros_installer postinst <install_dev> <mount_point> [ args ]\n

That usage string sucks

    string install_dir = argv[optind++];
    string install_dev = argv[optind++];

So install_dir is / and install_dev is /dev/sda3

I do see the slotted kernels /syslinux/vmlinuz.{A,B} that are supposed to be created by copying /boot/vmlinuz in both the failed builds and my workspace

Actually WTF, these are EFI images, not legacy BIOS, right?  They shouldn't be going through this branch then.

Actually I'm rambling, since EFI uses the same logic and I'm not sure what controls EFI vs legacy anyway.  Basically, /boot/vmlinuz is missing and I made no progress.

tl;dr I spoke with Don and this is 99% caused by https://crrev.com/c/657262

Revert? This seems like an egrigious failure that the CL author needs to deal with sadly.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/crosutils/+/4409319b9bae3a5f6ea7f68859a04c58adffad1c

commit 4409319b9bae3a5f6ea7f68859a04c58adffad1c
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Sat Mar 17 06:21:24 2018

Revert "Remove bulky vmlinu[zx] from rootfs."

This reverts commit 9e730a112ca18e0c1be1c79fc1bfd7bdca8314f6.

This is suspected to have broken moblab-generic-vm-paladin

BUG= chromium:461595 ,  chromium:822795 

Change-Id: I663611d830d894a62f2f77487a20d2da1ad10992
Reviewed-on: https://chromium-review.googlesource.com/966696
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Ilja H. Friedel <ihf@chromium.org>

[modify] https://crrev.com/4409319b9bae3a5f6ea7f68859a04c58adffad1c/update_bootloaders.sh
[modify] https://crrev.com/4409319b9bae3a5f6ea7f68859a04c58adffad1c/build_library/base_image_util.sh
[modify] https://crrev.com/4409319b9bae3a5f6ea7f68859a04c58adffad1c/bin/cros_make_image_bootable

Yep revert in #15 fixed it: https://luci-milo.appspot.com/buildbot/chromiumos/moblab-generic-vm-paladin/1081

Re-purposing bug for Craig to figure out the path ahead for re-landing their CL.

Pathhmesh, re #14, what do you mean by egregious failure? This went through the commit queue, right?

Oh, I didn't at all mean that the CL was egregiously bad. Just that the failure was so bad that I was unlikely to be able to quickly figure out what needs to be done to make moblab-generic-vm not hate this CL (I try to do that for changes not specifically targeted at this board that end up affecting it, because I'm the "maintainer" for the board). 

In this case, I was likely to need the CL authors help, so revert first.

Sorry that it sounded like I was blaming the CL / author.

About CL passing the CQ -- the affected board was marked experimental when the CL landed due to other issues. haddowk@ has fixed all those and marked it green again.

Let me know if you want my help to figure out how / if we can migrate this board off the old removed kernel bits.

Thanks for your help, pprabhu@.  Rolling back was definitely the right thing to do.  I believe the path forward is to wrap the "remove vmlinuz" logic in a USE flag and set it for all except for a small set of boards.  Sound reasonable?

Do you have any idea how to find all broken boards before submitting?

I haven't dug into this too much, but I think #6 and related comments are onto it. The problem is that the VM flow goes via legacy bios, which needs the kernel image to be in rootfs. I remember hitting this a few years ago when working on mobbuild.

It's not really a question of the installed packages (i.e. USE flag related), but of which bios we're using (UEFI vs legacy).

Most importantly, it seems to me that moblab-generic-vm and betty VMTest should behave similarly wrt this bug. And we should fix them similarly. Likely, the fix is not a USE flag change, but updated logic in the build scripts (that you modified) to only drop the kernel from rootfs if we know that it won't be needed for boot.

If you go down that path, you may have to additionally copy the kernel back into the rootfs in image_to_vm.sh, if the VM boots via legacy even though base image boots via EFI.

I'm rambling here, so take with a *big* grain of salt.

Thanks, that's useful context.  Do you have any thoughts on how to determine if a given board has a legacy bios or uses the boot partition?

Good point about image_to_vm, that's almost certainly necessary.

I've got an updated changelist that remove /boot/vmlinuz by default but allows boards to set a USE flag to disable this feature.  It's not at all clear to me which boards were affected and should have this feature disabled.

Is there anyone out there who can lend a hand in figuring out which boards to disable this feature on?