Actually, I can't find a bug in Gentoo's tracker explaining why it's been held out of stable for so long. I'll send 1.26.2 through the trybots and deploy to a physical machine and see how that goes.
It seems we don't have tests on recovery images. This looks like busybox symlinks are broken in recovery which even a simple boot test should have found. I'm manually building and deploying recovery image now. Will also need to figure out how to get a serial console.
From https://bugs.chromium.org/p/chromium/issues/detail?id=712564:
Commenting out "exec >"${LOG_FILE}" 2>&1" in initramfs/recovery/init keeps all recovery output directed at stdout/stderr.
This shows the actual errors are related to unexecutable shell functions:
+ initialize
+ init_check_clock
+ date +%Y
sh: can't execute 'date': No such file or directory
+ [ -lt 1970 ]
sh: 1970: unknown operand
+ init_mounts
+ mount -n -t proc -o nodev,noexec,nosuid proc /proc
sh: can't execute 'mount': No such file or directory
+ mount -n -t sysfs -o nodev,noexec,nosuid sysfs /sys
sh: can't execute 'mount': No such file or directory
+ mount -t devtmpfs -o mode=0755,nosuid devtmpfs /dev
sh: can't execute 'mount': No such file or directory
... yeah, they are all busybox functions: [date, mount, sed, tail, tee, ...]
After much spelunking--crawling through places keep in the earth forgot even by mythical creatures--I've figured out how to reproduce this relatively rapidly. Indeed, all of the busybox symlinks are missing. Going to focus on finding a root cause, now.
It took a lot of serial console debugging to figure this this out. There were three root causes:
1. BusyBox 1.23 in Portage is configured to force static compilation by default "+static". 1.26 switch this to optional+off-by-default. The pending upgrade to 1.26 will enable static again.
2. We compile BusyBox statically without installing symlinks in the initramfs. When doing so, in BusyBox >=1.26, it attempts to use an execve of busybox to satisfy shell script references to busybox applets. This only works if it can find its own binary path through examination of /proc/self/exe. The mount of the /proc filesystem would therefore fail if we invoked a naked "mount". To work around this, we can invoke the actual busybox binary in the $PATH and then, subsequently, all built-ins will work. See: https://git.busybox.net/busybox/tree/shell/ash.c?h=1_26_2#n7606 . Work-around for this in: https://chromium-review.googlesource.com/c/484709 .
3. There is no way to test our Recovery images, at all. So, this wasn't caught until someone went to install a Recovery image.
I will probably also write a doc on serial-console debugging Recovery because that--alone--took 3 days to get working.
the initramfs/ subdir has some logic for testing. check out README.md and test/README.md. there is also discussion in general about test coverage missing on recovery images in the CQ as you're not the only one to run into this.
test/test.sh is broken at head. I've opened https://bugs.chromium.org/p/chromium/issues/detail?id=714774 for that. On the plus side, it's a faster version of what I was doing: building and deploying to hardware so I can continue to do that, for now.
Comment 1 by jclinton@chromium.org
, Apr 14 2017