Seems to be on boards using kernel 3.8.  For versions before 3.10 we limit kernel size to 8 MB: https://cs.corp.google.com/chromeos_public/src/third_party/chromiumos-overlay/eclass/cros-kernel2.eclass?l=1238

I guess we've exceeded that limit recently.

Between passing and failing on butterfly we have no changes on 3.8, I don't see anything obvious that would put us over.

https://crosland.corp.google.com/log/9937.0.0..9938.0.0

We seem to build the same kernel in build packages just fine.
https://uberchromegw.corp.google.com/i/chromeos/builders/butterfly-release/builds/4058/steps/BuildPackages%20%5Bafdo_use%5D/logs/stdio

But we fail emerging it later on in the process.
https://uberchromegw.corp.google.com/i/chromeos/builders/butterfly-release/builds/4058/steps/Archive/logs/stdio

Last editor of that size cap was olofj, I am not sure of a great owner here, we may want someone with more kernel expertise to decide the best course of action here.

Sameer, do you have any recommendations? 

I think we should be ok bumping up the size.

Doug, dtor, groeck, Sonny, thoughts?

I believe the limit is there because of a firmware signing validation bug or.. something. I forget.

the kernel size limit is non-negotiable ;).  older firmwares (like lumpy/stumpy) had problems loading larger kernels.  i don't know when the bug was fixed, but Olof at the time just used <=linux-3.8 to catch things (which i whined about since we should be using disk layouts, but no one ever got around to addressing).

keep in mind that this is the recovery image failing which bundles an initramfs and thus is going to be fatter than the normal kernels, and more things can perturb the sizes there (like package upgrades).

random thought: what if, for recovery kernels, we turn off things like sound and bluetooth support ?  it's not like the recovery userland ships with any of those enabled.

On the same train of thought... Do we need networking support during recovery? If not, can the kernel built without network support? Same thing can be said on things like power management, ... etc?

Kernel max size is 8MB = 8 * 1024 * 1024 = 8388608 bytes
Actual kernel size when I build it says:   8452288 bytes

...so we're 63680 bytes over.

===

The initramfs in the recovery kernel is 4268648 bytes:

$ ls -al /build/peppy/var/lib/initramfs/recovery_ramfs.cpio.xz
-rw-r--r-- 1 root root 4268648 Sep 14 11:15 /build/peppy/var/lib/initramfs/recovery_ramfs.cpio.xz

As per above, maybe that's the lowest hanging fruit and maybe that's what changed?


Looking at the archive, one would at first think that /etc/screens would be big (they are big uncompressed) and maybe we could save space there.  ...but they compress well.  Rough top level directory sizes (compressed) can be generated by decompressing the .cpio.xz file, then running:

for f in *; do tar cvfJ ${f}.tar.xz ${f}; done

-rw-r--r-- 1 dianders eng 2.2M Sep 14 11:38 lib64.tar.xz
-rw-r--r-- 1 dianders eng 1.8M Sep 14 11:38 bin.tar.xz
-rw-r--r-- 1 dianders eng 489K Sep 14 11:38 etc.tar.xz

---

Biggest few lib files:

-rw-r--r-- 1 dianders eng 780K Sep 14 11:42 libcrypto.so.1.0.0.tar.xz
-rw-r--r-- 1 dianders eng 568K Sep 14 11:42 libc.so.6.tar.xz
-rw-r--r-- 1 dianders eng 336K Sep 14 11:42 libm.so.6.tar.xz
-rw-r--r-- 1 dianders eng  90K Sep 14 11:42 libblkid.so.1.tar.xz
-rw-r--r-- 1 dianders eng  89K Sep 14 11:42 libpng16.so.16.tar.xz
-rw-r--r-- 1 dianders eng  79K Sep 14 11:42 libdevmapper.so.1.02.tar.xz
-rw-r--r-- 1 dianders eng  68K Sep 14 11:42 ld-linux-x86-64.so.2.tar.xz
-rw-r--r-- 1 dianders eng  48K Sep 14 11:42 libudev.so.1.tar.xz
-rw-r--r-- 1 dianders eng  43K Sep 14 11:42 libz.so.1.tar.xz

Biggest bin files:

-rw-r--r-- 1 dianders eng 918K Sep 14 11:45 busybox.tar.xz
-rw-r--r-- 1 dianders eng 538K Sep 14 11:45 flashrom.tar.xz
-rw-r--r-- 1 dianders eng 398K Sep 14 11:45 mosys.tar.xz
-rw-r--r-- 1 dianders eng 276K Sep 14 11:45 crossystem.tar.xz
-rw-r--r-- 1 dianders eng 159K Sep 14 11:45 udevd.tar.xz
-rw-r--r-- 1 dianders eng 156K Sep 14 11:45 udevadm.tar.xz
-rw-r--r-- 1 dianders eng  72K Sep 14 11:45 futility.tar.xz
-rw-r--r-- 1 dianders eng  28K Sep 14 11:45 frecon-lite.tar.xz

Biggest etc/screens files:

-rw-r--r-- 1 dianders eng  83K Sep 14 11:48 ja.tar.xz
-rw-r--r-- 1 dianders eng  55K Sep 14 11:48 fr.tar.xz
-rw-r--r-- 1 dianders eng  49K Sep 14 11:48 de.tar.xz
-rw-r--r-- 1 dianders eng  43K Sep 14 11:48 nl.tar.xz
-rw-r--r-- 1 dianders eng  43K Sep 14 11:48 pt-BR.tar.xz
-rw-r--r-- 1 dianders eng  42K Sep 14 11:48 es-419.tar.xz
-rw-r--r-- 1 dianders eng  42K Sep 14 11:48 es.tar.xz
-rw-r--r-- 1 dianders eng  42K Sep 14 11:48 it.tar.xz
-rw-r--r-- 1 dianders eng  41K Sep 14 11:48 sv.tar.xz
-rw-r--r-- 1 dianders eng  40K Sep 14 11:48 ko.tar.xz
-rw-r--r-- 1 dianders eng  39K Sep 14 11:48 en-AU.tar.xz
-rw-r--r-- 1 dianders eng  39K Sep 14 11:48 en-CA.tar.xz
-rw-r--r-- 1 dianders eng  39K Sep 14 11:48 en-US.tar.xz
-rw-r--r-- 1 dianders eng  39K Sep 14 11:48 en-GB.tar.xz


Anyone have any idea how to trim some of the above?

===

We can also try to look at removing features (like networking?) from the recovery kernel if need be...

when it comes to trimming files from the initramfs, i think it's pretty unlikely you'll be able to do so.  the initramfs is constructed using lddtree which already copies the bare min in terms of files and deps.  it isn't like we're emerging binpkgs and then have to manually strip out docs/include headers/etc...

i will refer to  issue 679858  though where flashrom/mosys are currently statically linked but really shouldn't be.  that's causing them to be much larger than they otherwise would be (certainly more than 64KiB).

however, even if we shave those off, we don't have a lot of breathing room here.  i think we should look at stripping out unused stacks from the recovery kernel (like audio).

i lean towards keep networking to help with debugging of recovery, but maybe we can live with removing that (and forcing people to use the attached USB stick for offloading).  or maybe we can nuke the wifi stack while retaining ethernet support (we need to keep USB in order to access the recovery stick, so USB eth isn't a lot on top).

power management makes me hesitant as that would include cpu throttling, and i fear it'd let our cpus run at full speed all the time (and make thermals super unhappy).

bluetooth seems like an easy target to rip out.

after that, v4l2 ?  i don't think it'll have any impact on frecon/kms graphics support.

For reducing the root filesystem:

1. Static linking

1a) Right, there's no reason to statically link mosys / flashrom.  I commented on that other bug.  Just need to figure out how to make that happen for the build system, I guess?  That would explain why they are so huge.  

1b) I'd guess "busybox" is also statically linked.  Does it need to be?  My quick testing shows that it works fine without "static" but  bug #710074  implies that there might be trouble?  Quick tests show that this saves 60736 bytes in the compressed initramfs.

--

2. Is there anything we can do to our config options to shrink some of this down?  

2a) Can we disable some features of busybox, or we we need all of them?  

2b) Do we need the full functionality of mosys / crossystem / flashrom, or is there a way we can build a "lite" version of them (like we do for frecon) that just has bits needed for the recovery image?  

3c) Do we need every single cryptography function in "libcrypto" or can we build a version of the library with only the function we need?

--

3. Can we compile programs with "-Os" instead of "-O2" for the recovery image?

--

4. Is it worth it to figure out how to auto-generate the image assets?  Over 400K of image assets isn't wonderful and it seems like you ought to be able to blit some fonts in less memory, but maybe I'm being naive.

--

I guess many of the above get into the whole problem of needing two different versions of the same tool.  As per  bug #679858  that's not so supported.  :(

===

For reducing the kernel:

Without CONFIG_NET on peppy (doesn't boot):

 * Kernel image size is 7788992 bytes.

Without CONFIG_NET and CONFIG_MEDIA_SUPPORT:

 * Kernel image size is 7745664 bytes.


As per above turning off CONFIG_NET made things not boot for me (I tested on a kevin since that's what I had at my desk).  :(  ...so we'd need to dig more there.


I can try to poke with more of these but I'm really worried about the testing impact.  

I'm probably being overly paranoid, but I could sorta imagine that there could be some weird / subtle dependency here that nobody ever noticed.  Maybe it's important that the probe function of the media device get called because it sets some clock rate that's shared with some other peripheral?  I'm not saying that would be a good design but given that our build system doesn't really test recovery images very well I'm a little scared to touch something like this that would need to be manually tested on every board variant that we support...


Anyway, if we need to turn off big subsystems in the kernel we can, but it wouldn't be my first choice.

===

Maybe the place to start is to figure out why this got bigger in the first place?  Everything used to work.

Is it feasible to generate non-static versions of all binaries for the initfs, but use static versions on the rootfs? I'm mostly thinking of postinst.

I can't think of any reason why an initfs binary would need to be static.

#13: It makes sense to have statically linked binaries in initramfs if 1) the number of binaries is small and 2) shared libraries are not embedded in initramfs. For example, if the binaries using libcrypto were statically linked, libcrypto.so would not be needed, and (only) the required parts of libcrypto would be part of the statically linked image. My wild guess is that the statically linked binaries may be a leftover from times when _only_ statically linked binaries were present.

> 1b) I'd guess "busybox" is also statically linked.  Does it need to be?

oh hai old bug we didn't clean up.

 issue 712564  and  issue 710074  track recovery image breakage due to dynamic busybox.  but those were due to an underlying bug in our bb patch that should be fixed by:
  https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5c7ecf36f0bbbe18b513d7afb82b0f7bf342889c
and our current copy of busybox includes.  but we never got around to backing out the USE=static setting.  that will shave off like ~1MiB (uncompressed) which might be sufficient here to get green builds again.

i've posted a CL but need help testing (my USB sticks aren't big enough to fit samus images anymore and new ones haven't yet arrived):
  https://chromium-review.googlesource.com/668036

> 2. Is there anything we can do to our config options to shrink some of this down?  

perhaps, but i'm not sure shaving busybox config options will recoup as much as shaving kernel config options.  it's also not as easy to pull off currently as we don't build any userland packages with unique settings for the recovery initramfs.  we do however build a unique kernel which is why shaving options there is much easier.

> 2c) Do we need every single cryptography function in "libcrypto" or can we build a version of the library with only the function we need?

openssl doesn't provide a whole lot of knobs to customize its feature set.  there might be somethings we want to disable everywhere, but that'll require a decent amount of testing since it'll impact all targets (see above wrt userland).

> 3) Can we compile programs with "-Os" instead of "-O2" for the recovery image?

the kernel already uses -Os iirc.  wrt userland compile settings, see above.

> 4) Is it worth it to figure out how to auto-generate the image assets

i don't think so.  the auto-generation is not fast, and will likely require even more overhead to pull off.  it needs graphics libraries and fonts!

> Without CONFIG_NET on peppy (doesn't boot):

i suspect that also kills things that userland relies on (like netlink sockets?).  i would still lean towards not killing that entire subsystem.  i'd focus on other things like SOUND.

> I can try to poke with more of these but I'm really worried about the testing impact.  

sure, but we can't be paralyzed forever here ... we're going to have to accept the pain to make reasonable progress.

Looking at what sizes used to be to see why this broke:

1. Download ChromeOS-recovery-R63-9935.0.0-peppy.tar.xz

2. Untar:

tar xvf ~/Downloads/dev-channel%2Fpeppy%2F9935.0.0%2FChromeOS-recovery-R63-9935.0.0-peppy.tar.xz

3. Extract (would need extra steps for ARM since vmlinuz needs to run through dtc, but this is x86):

START_BLK=$(cgpt show recovery_image.bin | grep KERN-A | awk '{print $1};')
SIZE_BLK=$(cgpt show recovery_image.bin | grep KERN-A | awk '{print $2};')
dd if=recovery_image.bin of=kern.bin bs=512 skip=${START_BLK} count=${SIZE_BLK}
vbutil_kernel --get-vmlinuz kern.bin --vmlinuz-out vmlinuz
~/trunk/src/third_party/kernel/v3.8/scripts/extract-vmlinux vmlinuz > vmlinux
grep -abo 7zXZ vmlinux

# Look at the output of the grep.  I see:
# 9784226:7zXZ
# 14079921:7zXZ

# This doesn't work:
tail -c +9784226 vmlinux | xz -d --stdout > b
cpio -t < b

# This does (ignore the errors)
tail -c +14079921 vmlinux | xz -d --stdout > b
mkdir out
cd out
cpio -i < ../b


4. Using same tricks as above, it looks like "bin" increased by ~.2M.  Sizes from before (compare to above):

-rw-r--r-- 1 dianders eng 2.2M Sep 14 16:36 lib64.tar.xz
-rw-r--r-- 1 dianders eng 1.6M Sep 14 16:36 bin.tar.xz
-rw-r--r-- 1 dianders eng 489K Sep 14 16:36 etc.tar.xz

Top files inside bin:

-rw-r--r-- 1 dianders eng 918K Sep 14 16:38 busybox.tar.xz
-rw-r--r-- 1 dianders eng 538K Sep 14 16:38 flashrom.tar.xz
-rw-r--r-- 1 dianders eng 277K Sep 14 16:38 crossystem.tar.xz
-rw-r--r-- 1 dianders eng 159K Sep 14 16:38 udevd.tar.xz
-rw-r--r-- 1 dianders eng 156K Sep 14 16:38 udevadm.tar.xz
-rw-r--r-- 1 dianders eng  72K Sep 14 16:38 futility.tar.xz
-rw-r--r-- 1 dianders eng  28K Sep 14 16:38 frecon-lite.tar.xz

===

NOTICE: mosys isn't there!  That ~400K is what killed us, I think...

> and our current copy of busybox includes.  but we never got around to 
> backing out the USE=static setting.  that will shave off like ~1MiB 
> (uncompressed) which might be sufficient here to get green builds again.

It's almost enough, but not quite in my testing.  As per above this saved 60736 bytes in the final image size in my testing.  We needed to save 63680 bytes.

---

> i've posted a CL but need help testing (my USB sticks aren't big enough 
> to fit samus images anymore and new ones haven't yet arrived):

I'll try to do a little more testing tomorrow.

---

> > I can try to poke with more of these but I'm really worried about the testing impact.  
> 
> sure, but we can't be paralyzed forever here ... we're going to have to accept the pain 
> to make reasonable progress.

In theory we're not paralyzed forever.  We only need to hold out until 3.8 is deprecated and then we have a much more sane limit.

If we can save several hundred kilobytes in userspace we might be fine for another few years and by then maybe 3.8 will finally be gone?  ...and it's easy to save a few hundred kilobytes by solving the static linking problem.

---

> > Without CONFIG_NET on peppy (doesn't boot):
> 
> i suspect that also kills things that userland relies on (like netlink sockets?).  
> i would still lean towards not killing that entire subsystem.  i'd focus on other 
> things like SOUND.

I'll see if I can find anything super low hanging that ought to be uber safe.  I'd rather avoid anything that prevents one of our device drivers from loading if possible just because I'm paranoid, but it would probably be OK.

---

> > 4) Is it worth it to figure out how to auto-generate the image assets
> 
> i don't think so.  the auto-generation is not fast, and will likely require even 
> more overhead to pull off.  it needs graphics libraries and fonts!

It's not fast because it's generating every possible asset.  In this case we would generate a single asset on demand.  We should be able to handle that.

We already have some ability to blit fonts, right?  Doesn't "frecon-lite" do that?  It's tiny so it can't be too hard.  ...but you're right that the fonts count possibly just be too big...

> It makes sense to have statically linked binaries in initramfs if 1) the number of 
> binaries is small and 2) shared libraries are not embedded in initramfs. For example, 
> if the binaries using libcrypto were statically linked, libcrypto.so would not be 
> needed, and (only) the required parts of libcrypto would be part of the statically 
> linked image. My wild guess is that the statically linked binaries may be a leftover 
> from times when _only_ statically linked binaries were present.

Hard to say.  Certainly I found that _not_ statically linking busybox ended up in a total initramfs that was 60K smaller (after compression).  I'd guess you'd want to think of this on a case-by-case basis.

BTW: given the mosys discovery above, we probably broke because of:

https://chromium-review.googlesource.com/c/chromiumos/platform/initramfs/+/657518


I'm at the end of my day.  Hung Te: any chance you can continue?  If not, there's always tomorrow.

#18: Sure, because the shared libraries are already there. In that case it is all but pointless to keep static binaries around (unless, for example, libcrypto or libm is only needed by one binary and that binary is compiled statically). In a simper system, busybox might be the only binary.

> We already have some ability to blit fonts, right?  Doesn't "frecon-lite" do that?  It's tiny so it can't be too hard.  ...but you're right that the fonts count possibly just be too big...

unfortunately, it's not that easy.  if you want to only produce messages in ASCII, then OK.  but we're international now and proper text rendering of many scripts is anything but easy.  granted, the hardest ones we have *currently* are CJK, but those still aren't trivial.

> mosys https://chromium-review.googlesource.com/657518

that probably is what put us over the limit, but considering the direction we're going in CrOS (model management is standardizing on mosys), i don't think we have a choice here.

> ...shared libs...

with lddtree, creating an initramfs using dynamic libs is a non-issue.  we've got enough programs now that trying to just do static linking doesn't make sense, and even then it's more of a pita than it's worth.  let's not waste time going down that route.

> -rw-r--r-- 1 dianders eng 277K Sep 14 16:38 crossystem.tar.xz

why is that static ?  that would go from ~800KB uncompressed to ~40KB.  moved to  issue 765499 .  that + busybox might get us back down below the critical level and allow us to keep mosys.

A quick trick is to make mosys producing both static and dynamic version - we've done this for futility and vpd. So recovery image can get dynamic version with copytree while firmware updater (or other programs need static ones) can find the static binaries.

I made a quick patch by changing mosys to have both dynamic and static version, and that gives me 

 164K dyn.tbz2
 488K static.tbz2

+apronin. mosys was added for solving issue 763264. Another solution is to only include mosys if the crossytem will use mosys for NVRAM?

Ok, here're mosys changes:

 https://chromium-review.googlesource.com/#/c/chromiumos/overlays/chromiumos-overlay/+/668263/
 https://chromium-review.googlesource.com/#/c/chromiumos/platform/mosys/+/668266/

I've kicked trybots to verify:

Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:butterfly-release] [buildbucket_id:8968413511074732304].
Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:butterfly-full] [buildbucket_id:8968413510676444688].
Tryjob submitted!
Go to https://uberchromegw.corp.google.com/i/chromiumos.tryserver/waterfall?committer=hungte@chromium.org&builder=full&builder=release to view the status of your job.

plus one change to crossystem, which make save more kbytes: Change to crossystsem:
 https://chromium-review.googlesource.com/#/c/chromiumos/platform/vboot_reference/+/668274/

crossytem size changes:

16K  dyn.tbz2
336K static.tbz2

After changing crossystem and mosys into dynamic program, on butterfly:

 * Kernel image size is 8394176 bytes.

Seems like the not very helpful - since mosys is pretty large even in dynamic (du = 128k).

I think the extra messages and inclusion of mosys made the kernel too big...

@mnissler/@apronin, do we really need to call "crossystem clear_tpm_owner_request=1" in recovery shim?

Or can we reduce the messages you've recently added for recovery shim?

By also making flashrom dynamic, I can build butterfly recovery images. So that's probably the best step we can do now...

... the CL for flashrom https://chromium-review.googlesource.com/#/c/chromiumos/overlays/chromiumos-overlay/+/668352/

One alternative is to move programs that is needed "after" rootfs is verified to be loaded from rootfs, or put images on stateful partition (with checksum pre-recorded) and load from there, so we don't need to squeeze everything into kernel partition.

Re comment 29: We do not call crossystem clear_tpm_owner_request=1 from within the recovery scripts, this is running from the rootfs. I haven't followed issue 763264 closely - I'd be surprised if we needed mosys after all though.

I guess the extra messages you refer to is https://chromium-review.googlesource.com/c/chromiumos/platform/initramfs/+/647426 ?

These add 4 new messages, with 2 being a variation of the same "insufficient battery charge" message for the cases where a charger is connected or not. The UX folks wanted that distinction, but we can probably negotiate.

Regarding relying on verification and loading stuff from the rootfs, you need to be very careful. This possible in theory, but there is no point in execution after which you can assume you have a verified rootfs, as it's also possible to call the installer on an unverified rootfs for recovery+dev boots.

Here's an idea of how we could potentially address the initramfs size issue at a more fundamental level: Assuming we can't change kernel image size limits, our best option is to keep the initramfs as small as possible. One avenue would be to make the initramfs bundled with the kernel to hold only a minimal script that mounts a dm-verity image stored elsewhere on the recovery media and then call into that. That way, the dm-verity image can grow as needed without any hard limits (not sure that's enitrely a good thing ;-)). For storing the dm-verity image, one of the currently unused partition entries might be workable? Rootfs is probably not a good idea since the recovery scripts + utilities aren't needed in the installed system so they'd just waste space.

that's an interesting idea, but i think that's a very long term solution.  that'll require a lot of moving parts to change.  we need something shorter here.

Re #23: mosys was added to the recovery image to address issue 763264, which was branched from issue 762934. After addressing the parent issue 762934 in CL:654297, we no longer call "crossystem clear_tpm_owner_request=1" when the recovery image is run in recovery mode. 

However, if the same recovery image boots in normal/dev mode through Ctrl-U, it will call it (assuming other conditions for the update are met: tpm not owned, consent given). So, this path is still broken without mosys. Not sure if that's an important use case. Short-term we may decide to sacrifice it and revert adding mosys to the recovery image.

* the previous comment was response to #33, not #23

It seems like we have a good short term solution with:

* crosreview.com/668274 - Makefile: Build utils for both dynamic and static version
* crosreview.com/668263 - sys-app/mosys: Install dynamic and static binaries at same time
* crosreview.com/668352 - sys-apps/flashrom: Build both dynamic and static binaries
* crosreview.com/668036 - busybox: turn off static linking

All of those are marked to go in.  I built locally and my initramfs is now:

-rw-r--r-- 1 root root 4053512 Sep 15 09:31 /build/peppy/var/lib/initramfs/recovery_ramfs.cpio.xz

...so we saved about 200K (215136 bytes).  

Confirming with the kernel build, my kernel now says:

 * Kernel image size is 8237760 bytes.

That's compared with 8452288 before.  From there we get that we saved 214528 bytes.  Pretty consistent.

From the above math, we have 150848 bytes of head room after all the above changes

===

NOTE: If we want a low hanging fruit to save a metric buttload of more space, we can also look to Guenter's comments for inspiration.  Specifically note that there is only one user of "libcrypto" and libcrypto is ginormous.

...so it turns out that if we just statically link futility we get a big savings.  Taking all the above and statically linking futility:

-rw-r--r-- 1 root root 3321248 Sep 15 09:43 /build/peppy/var/lib/initramfs/recovery_ramfs.cpio.xz

...so we save 732264 bytes (!).  In that case kernel ends up:

 * Kernel image size is 7505344 bytes.

Yeah we pay a little bit because we've statically linked other things to futility, but wow the savings.  We already build both futility and futility_s.  Does anyone know the small magic that would choose futility_s for the initramfs?  My solution was just hacky...  I can poke a bit, too...

NOTE: if we could build yet a 3rd version of futility that statically linked libcrypto but dynamically linked everything else that would be even better...

===

Potentially we can open a bug about saving a bit of extra space in the kernel if need be.  I'd tend not to by default due to reasons I argued about above, though and it seems like we can very easily get to 883K of margin w/ no kernel hackery.  That seems like enough for now?

===

I tried all pending changes (to move away from static version) plus my hacky futility change (to use the static version) on kevin and was able to boot a recovery image and recover.

i think we should pursue dynamic linkage everywhere all the time and then we don't have to mess around with multiple static copies of programs for diff envs

i've posted a series of CLs already to convert firmware updater to use dynamic linkage

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/d27a1a52bb7b7c063f40532b419887237719df49

commit d27a1a52bb7b7c063f40532b419887237719df49
Author: Mike Frysinger <vapier@chromium.org>
Date: Fri Sep 15 17:50:32 2017

busybox: turn off static linking

We turned this on because it was historically enabled, and because
there was a latent bug in the hush wrapper patch.  Those should be
fixed by the upstream Gentoo 5c7ecf36f0bbbe18b513d7afb82b0f7bf3428
(which we included in CL:489122), so we can drop this again.

BUG= chromium:712564 , chromium:710074 , chromium:764753 
TEST=precq passes
TEST=recovery image still works

Change-Id: Ia8de426f7da212ef2970340adfca691d9ebfc553
Reviewed-on: https://chromium-review.googlesource.com/668036
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Reviewed-by: Douglas Anderson <dianders@chromium.org>

[modify] https://crrev.com/d27a1a52bb7b7c063f40532b419887237719df49/profiles/targets/chromeos/package.use

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/913e2d3af7709280730183c7732daefa1fa1adc0

commit 913e2d3af7709280730183c7732daefa1fa1adc0
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Sep 15 17:50:29 2017

sys-app/mosys: Install dynamic and static binaries at same time.

The auto update process (especially firmware updater) needs static mosys
but normal OS images, including recovery images, don't need that. We
should build both dynamic and static binaries at the same time for
images to choose what they need.

BUG= chromium:764753 
TEST=emerge-reef mosys

Change-Id: I8a0ad746a47f819fab47ddc241fa409d63492240
Reviewed-on: https://chromium-review.googlesource.com/668263
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[add] https://crrev.com/913e2d3af7709280730183c7732daefa1fa1adc0/chromeos/config/env/sys-apps/mosys
[modify] https://crrev.com/913e2d3af7709280730183c7732daefa1fa1adc0/sys-apps/mosys/mosys-9999.ebuild
[modify] https://crrev.com/913e2d3af7709280730183c7732daefa1fa1adc0/profiles/targets/chromeos/package.use

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/vboot_reference/+/c8e3d27c814634c629b46dab0b708d3cf375c641

commit c8e3d27c814634c629b46dab0b708d3cf375c641
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Sep 15 17:50:29 2017

Makefile: Build utils for both dynamic and static version.

The auto update process (especially firmware updater) needs static vboot
utilitys but normal OS images, including recovery images, don't need
that. We should build both dynamic and static binaries at the same time
for images to choose what they need.

Currently only `crossystem` will build static version. And after this
change is merged:

(cd /build/reef/usr/bin; file crossystem*)
crossystem:   ELF 64-bit LSB shared object
crossystem_s: ELF 64-bit LSB executable

(cd /build/reef/usr/bin; du -sh crossystem*)
40K  crossystem
808K crossystem_s

BUG= chromium:764753 , chromium:765499 
TEST=emerge-reef vboot_reference
BRANCH=None

Change-Id: Ibd66c87bb44c5593767aeb710240e0165103f016
Reviewed-on: https://chromium-review.googlesource.com/668274
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/c8e3d27c814634c629b46dab0b708d3cf375c641/Makefile

If we want the free ~700K of extra headroom in the recovery image, this works:

* https://chromium-review.googlesource.com/#/c/chromiumos/platform/initramfs/+/668689

I know Mike said he wants all dynamic all the time, but 700K is a lot...

i'm not against landing the futulity_s change if it's needed to get us out of the current mess, but i think we're out of the woods now ?  building a lumpy recovery image works for me.  guess we'll check the release builds tonight.

OK, this is technically "Fixed".  Builders seem to be passing now given the fixes above.

---

How do we want to track this?  It seems like we've already got ample bugs for future work:

*  bug #765753  - chromeos-postinst: run inside of the target root instead of outside
*  bug #679858  - flashrom/mosys should not be static
*  bug #405595  - Convert chromeos-postinst to a dynamically linked ELF (~1.2MiB reduction)
*  bug #765844  - futility on the recovery image sucks in libcrypto, which is too big


Some of those don't really have owners.  I'm not sure who the usual owner for this stuff is, but I've got a bunch of other stuff I've got to get back to.  ;-)  Hopefully someone can step up to own most of those?

---

Q: Do we want a bug about trying to shrink down the kernel, or are others OK with not doing this?

Q: Do we want a separate bug to capture the idea in comment 34 (aka move some stuff to a separate image)?


IMHO if we can solve all the already-filed bugs we should be fine.  I don't think we can get an order of magnitude of extra space from the kernel.  Maybe we can get another 1MB in low hanging fruit by hacking things out of the kernel if we are willing to eat the testing / development effort.  ...but also if we can just last a few more years with our 700KB then 3.8 will be obsolete and we'll have quite a bit more margin...

---

I think Hung Te and Mike did most of the actual work here, so assigning ownership and marking Fixed.

#46: Given that there is an easy available means to reduce image size by another 700k, I don't think we should even consider reducing the kernel size. This would add a lot of risk for no practical purpose other than to avoid static linking of one binary.

Re apronin's comment #36: Even if we run a recovery image in developer mode (i.e. Ctrl-U boot), we still execute tpm-firmware-updater in an environment that's chroot()ed to the rootfs. In other words, crossystem clear_tpm_owner_request=1 never runs within the recovery environment AFAICS.

So the whole include-mosys-in-recovery thing shouldn't be needed. I'm indifferent on whether we should roll this back though: On one hand, crossystem failing in obscure ways is a trap for future engineers making recovery changes. On the other hand, the initramfs size savings might be more important.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/mosys/+/5e062b2fe2232a23b6b808e0beaf83c9ab8ecac9

commit 5e062b2fe2232a23b6b808e0beaf83c9ab8ecac9
Author: Hung-Te Lin <hungte@chromium.org>
Date: Mon Sep 18 12:32:07 2017

Build dynamic and static binaries at same time.

The auto update process (especially firmware updater) needs static mosys
but normal OS images, including recovery images, don't need that. We
should build both dynamic and static binaries at the same time for
images to choose what they need.

BUG= chromium:764753 
TEST=emerge-reef mosys

Change-Id: I78e8add8dcbfcb74a069c8f87d3f4b86d5a1e189
Reviewed-on: https://chromium-review.googlesource.com/668266
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/5e062b2fe2232a23b6b808e0beaf83c9ab8ecac9/Makefile

@48: It's probably fine to include mosys.  It should be fairly small (131K compressed) now that we made it dynamically linked.  We can think about yanking it out again if we get super tight.  Personally I'd rather land the futility change ( bug #765844 ).

---

As a side note: I checked and it seems like builders are passing...

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/416790d57a6d19f460cd83ee15089beee81d5960

commit 416790d57a6d19f460cd83ee15089beee81d5960
Author: Hung-Te Lin <hungte@chromium.org>
Date: Tue Sep 26 20:36:05 2017

sys-apps/flashrom: Build both dynamic and static binaries.

The auto update process (especially firmware updater) needs static
flashrom but normal OS images, including recovery images, don't need
that. We should build both dynamic and static binaries at the same time
for images to choose what they need.

BUG= 764753 
TEST=cros_workon --board butterfly start flashrom;
     cros_workon --host start flashrom;
     emerge-butterfly flashrom
     sudo emerge flashrom

Change-Id: I851357160c9a7a0c0a375bbe2eb249c27820d443
Reviewed-on: https://chromium-review.googlesource.com/668352
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[add] https://crrev.com/416790d57a6d19f460cd83ee15089beee81d5960/chromeos/config/env/sys-apps/flashrom
[modify] https://crrev.com/416790d57a6d19f460cd83ee15089beee81d5960/sys-apps/flashrom/flashrom-9999.ebuild
[modify] https://crrev.com/416790d57a6d19f460cd83ee15089beee81d5960/profiles/targets/chromeos/package.use

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46

commit 78d2b7a83f71309a2b4eb3da3d8aacd660a46e46
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat Oct 14 02:43:43 2017

flashrom: disable raiden_debug_spi on linux-3.8/3.10 systems

Our recovery image is too fat, so drop this driver from flashrom
as it pulls in libusb (and is the only thing that does).

BUG= chromium:764753 
TEST=building flashrom no longer links in libusb

Change-Id: Ib10bf8d76749c79720c9dc7ad4cd8e425b67a0a6
Reviewed-on: https://chromium-review.googlesource.com/719896
Commit-Ready: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Douglas Anderson <dianders@chromium.org>

[modify] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/overlay-nyan/profiles/base/package.use
[modify] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/overlay-link/profiles/base/package.use
[add] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/overlay-puppy/profiles/base/package.use
[add] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/overlay-parrot/profiles/base/package.use
[add] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/overlay-stumpy/profiles/base/package.use
[add] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/overlay-butterfly/profiles/base/package.use
[add] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/overlay-panther/profiles/base/package.use
[add] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/overlay-x86-alex/profiles/base/package.use
[modify] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/overlay-daisy/profiles/base/package.use
[modify] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/chipset-hsw/profiles/base/package.use
[modify] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/overlay-peach/profiles/base/package.use
[add] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/overlay-lumpy/profiles/base/package.use
[modify] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/overlay-x86-mario/profiles/base/package.use
[add] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/overlay-stout/profiles/base/package.use
[add] https://crrev.com/78d2b7a83f71309a2b4eb3da3d8aacd660a46e46/overlay-x86-zgb/profiles/base/package.use