once a device ships, we can't change the partition layout

I think we may not want to do this for boards in production, if this works like the rootfs partition, we don't have a way of changing it in the field.

So while this is functionally ok 'as it is today' to change the layout for devices in the field, the problem is if we were to actually start using the additional space on a device in the field, we would not see the failure on our lab machines or any that have been recovered after the change, but users in the field would likely start failing to auto update (or worse). So changing the field partition layout is kind of like a time bomb that goes off when we grow too large. 

* Right that I'm pretty sure we can't resize partitions in the field, so definitely anything that shipped it seems like a no-go here.

* For anything that hasn't shipped but has people trying to get auto updates (or use cros flash?) it's probably good to change this.  I guess you'd have to figure out if it's a hard failure when you AU or if it's a soft failure (AKA maybe it will work as long as the kernel is small enough).  I think I'd prefer the soft failure in this case since we're not likely to go over the old 16 MB soon and AUs will keep working.

===

In any case it seems best to post up patches for one board at a time and folks in charge of that board can make a decision.

I would say anything that has not FSIed yet should get this, it could potentially impact dogfood machines (and the device owner should be consulted), but it is probably better to have to recover a few dogfooders now than have to live with a cramped kernel partition.

Cheza at:

https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1299575

...if that looks good I'll go through the list above and create CLs for anything else that seems like it should move over.

UE won't be able to resize the partition.  i don't know if UE will choke if the partition isn't big enough even if we don't fill it ... i don't see why it should, but who knows.  i.e. if the partition has ~10MB of data in a 32MB partition (during payload gen), will UE die if the partition (at runtime) is only 16MB ?

recovery flows will redo the partition table.  if you use cros flash to image a DUT directly, that won't work afaik.

if UE is fine, then yeah, lets update the layout even if people are dogfooding (assuming the device hasn't hit FSI yet).

Maybe these folks can tell us what UE will do?

Looking at the list above, my best guesses:

baseboard-cheza/scripts/disk_layout.json:        yes
baseboard-dragonegg/scripts/disk_layout.json:    yes
baseboard-fizz/scripts/disk_layout.json:         no
baseboard-kalista/scripts/disk_layout.json:      no?
baseboard-krabbylake/scripts/disk_layout.json:   no?
baseboard-meowth/scripts/disk_layout.json:       no
baseboard-nami/scripts/disk_layout.json:         no
baseboard-octopus/scripts/disk_layout.json:      yes
baseboard-poppy/scripts/disk_layout.json:        no
baseboard-rammus/scripts/disk_layout.json:       no?
overlay-eve/scripts/disk_layout.json:            no
overlay-fizz-moblab/scripts/disk_layout.json:    no
overlay-kidd/scripts/disk_layout.json:           no?
overlay-nautilus/scripts/disk_layout.json:       no

This seems like an awfully manual / error-prone process.  Maybe someone in partner engineering knows of a way we can make sure we don't actually miss a board?

Update engine might not fail if the data is still under 10MB. (This can easily be tested by cros flashing a device which has 16 MB kernel to an image which has 32MB kernel). But that doesn't solve the problem, because the partitions size will not change and to fix it needs recovery. So yeah, for production devices out there, there is nothing to be done. But for devices which hasn't hit FSI, I think we should do it even if it means dogfooders need to do recovery?!

I think we should start having all kernel partitions to 32 MB regardless the size of the rootfs.

OK, it sounds as if everyone is in agreement that we move to 32 GB.  Regardless of whether AU is broken or not it seems like we want it.  If AU is broken then any existing dogfooders will just have to recover and we'll have to send out a PSA.

...so let's move forward.  I'm happy to submit CLs for boards, but I'm not sure I'm going to be that good at identifying boards.  I also am not sure how we make sure that no new boards forget to do this.  I'd tend to defer to partner engineering for that.

I have done experiment downloading a small full payload (40MB) to a big partition (a 100M image file) and the resulted file can be successfully mounted and the content side are valid.

As dianders@ and vapier@ mentioned, UE does not have capacity to resize the partition or change layout. It does the update job by following a pre-populated list of operations.

to clarify, we move to 32MB kernel partitions when the main storage is larger than 16GB.  we don't use 4GB rootfs when the disk is still only 16GB.

do we have a policy for newer devices that they ship with 32GB+ ?

FSI status is here: 
https://cros-goldeneye.corp.google.com/chromeos/console/listFsi?status=APPROVED

pre-fsi:
baseboard-cheza/scripts/disk_layout.json:        yes
baseboard-dragonegg/scripts/disk_layout.json:    yes
baseboard-fizz/scripts/disk_layout.json:         no
baseboard-kalista/scripts/disk_layout.json:      yes
baseboard-krabbylake/scripts/disk_layout.json:   no (this is nocturne)
baseboard-meowth/scripts/disk_layout.json:       obsolete
baseboard-nami/scripts/disk_layout.json:         no
baseboard-octopus/scripts/disk_layout.json:      yes
baseboard-poppy/scripts/disk_layout.json:        no
baseboard-rammus/scripts/disk_layout.json:       yes
overlay-eve/scripts/disk_layout.json:            no
overlay-fizz-moblab/scripts/disk_layout.json:    no
overlay-kidd/scripts/disk_layout.json:           ??
overlay-nautilus/scripts/disk_layout.json:       no

re #11: I think the problem is updating a smaller partition with an update payload that was generated for a larger partition (but with smaller data in the partition) if that is what you're describing ;)

Does anybody know a convenient way to check the total amount of space being used on the kernel partition for devices today?

Re comment 12, yes the hardware requirements specifies a minimum of 32GB of storage. 

> Does anybody know a convenient way to check the total amount of space being used on the kernel partition for devices today?

You can run 'futility show /dev/sda2' on a kernel partition and it will show something like "Body size: 0x72c000", which should be the total amount used (minus a few more KB for the VBLOCK and preamble).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/52ea20693d882cfb0ec68ef95994368d549fe02a

commit 52ea20693d882cfb0ec68ef95994368d549fe02a
Author: Douglas Anderson <dianders@chromium.org>
Date: Fri Oct 26 07:43:34 2018

baseboard-kukui: 4 GB root fs; 32 MB kernel partitions

All the cool kids are doing it.

BUG=chromium:898971
TEST=build_image

Change-Id: I57096e3829f48713c882de72cce0f10031f9cef1
Reviewed-on: https://chromium-review.googlesource.com/1299579
Commit-Ready: Hsin-Yi Wang <hsinyi@chromium.org>
Tested-by: Hsin-Yi Wang <hsinyi@chromium.org>
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>
Reviewed-by: Hsin-Yi Wang <hsinyi@chromium.org>
Reviewed-by: Nicolas Boichat <drinkcat@chromium.org>

[modify] https://crrev.com/52ea20693d882cfb0ec68ef95994368d549fe02a/baseboard-kukui/scripts/disk_layout.json

Using Nick's list from @13:


dragonegg:

remote:   https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1301542 baseboard-dragonegg: Update URL to disk layout docs        
remote:   https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1301543 baseboard-dragonegg: 32 MB kernel partitions        

kalista:

remote:   https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1301545 baseboard-kalista: Update URL to disk layout docs        
remote:   https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1301546 baseboard-kalista: 32 MB kernel partitions        

octopus:

remote:   https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1301548 baseboard-octopus: Update URL to disk layout docs        
remote:   https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1301549 baseboard-octopus: 32 MB kernel partitions        
remote:   https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1301550 baseboard-octopus: Specify rootfs size the same way as everyone else        
remote:   https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1301551 baseboard-octopus: Make the USB disk layout match everyone else        
remote:   https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1301552 baseboard-octopus: Add the VM root fs        

rammus:

remote:   https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1302033 baseboard-rammus: Update URL to disk layout docs        
remote:   https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1302034 baseboard-rammus: 32 MB kernel partitions        

===

Those are all I plan to do unless someone speaks up.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/1dffbd21c3b2c13a74c952bc930fec3f81ca0563

commit 1dffbd21c3b2c13a74c952bc930fec3f81ca0563
Author: Douglas Anderson <dianders@chromium.org>
Date: Fri Oct 26 19:14:54 2018

baseboard-cheza: Switch kernel paritions to 32 MB

We want to give ourselves breathing room, so move to 32 MB kernel
partitions.  This should likely be done for any new boards.

BUG=chromium:898971, b:118342513
TEST=build_image now creates 32 MB kernel partitions

Change-Id: I70c4785f8b30bc550cd685f8789a983c0b01e0b5
Reviewed-on: https://chromium-review.googlesource.com/1299575
Commit-Ready: Douglas Anderson <dianders@chromium.org>
Tested-by: Douglas Anderson <dianders@chromium.org>
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>

[modify] https://crrev.com/1dffbd21c3b2c13a74c952bc930fec3f81ca0563/baseboard-cheza/scripts/disk_layout.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/45c57520d8152f31016b1863c79932004a9ccd82

commit 45c57520d8152f31016b1863c79932004a9ccd82
Author: Douglas Anderson <dianders@chromium.org>
Date: Tue Oct 30 23:54:19 2018

baseboard-dragonegg: 32 MB kernel partitions

Gives us room to grow.  All the cool kids are doing it.

BUG=chromium:898971
TEST=build_image

Change-Id: Ic49d96a653f44abbc64c2bae79bc2be7fec4b058
Reviewed-on: https://chromium-review.googlesource.com/1301543
Commit-Ready: Rajat Jain <rajatja@chromium.org>
Tested-by: Rajat Jain <rajatja@chromium.org>
Reviewed-by: Rajat Jain <rajatja@chromium.org>
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>

[modify] https://crrev.com/45c57520d8152f31016b1863c79932004a9ccd82/baseboard-dragonegg/scripts/disk_layout.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/0f3302d40d662f8f3e0c630de4139c2f47162a26

commit 0f3302d40d662f8f3e0c630de4139c2f47162a26
Author: Douglas Anderson <dianders@chromium.org>
Date: Mon Nov 05 20:38:24 2018

baseboard-kalista: 32 MB kernel partitions

Gives us room to grow.  All the cool kids are doing it.

BUG=chromium:898971
TEST=build_image

Change-Id: I0fd5168bce89b3d25702d14afb3c33fa1457ee8f
Reviewed-on: https://chromium-review.googlesource.com/1301546
Commit-Ready: Douglas Anderson <dianders@chromium.org>
Tested-by: Zhuohao Lee <zhuohao@chromium.org>
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>

[modify] https://crrev.com/0f3302d40d662f8f3e0c630de4139c2f47162a26/baseboard-kalista/scripts/disk_layout.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/10320d814418852a66c5174cf561f666dc0c1442

commit 10320d814418852a66c5174cf561f666dc0c1442
Author: Douglas Anderson <dianders@chromium.org>
Date: Mon Nov 05 20:38:24 2018

baseboard-rammus: 32 MB kernel partitions

Gives us room to grow.  All the cool kids are doing it.

BUG=chromium:898971
TEST=build_image

Change-Id: If04095245f59f9aa17fd48cf40b3bebb489a3a31
Reviewed-on: https://chromium-review.googlesource.com/1302034
Commit-Ready: Douglas Anderson <dianders@chromium.org>
Tested-by: Zhuohao Lee <zhuohao@chromium.org>
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>

[modify] https://crrev.com/10320d814418852a66c5174cf561f666dc0c1442/baseboard-rammus/scripts/disk_layout.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/fa4ef45c1fe9558a740e64df9564d2911f8674dd

commit fa4ef45c1fe9558a740e64df9564d2911f8674dd
Author: Douglas Anderson <dianders@chromium.org>
Date: Tue Nov 06 06:10:34 2018

baseboard-octopus: 32 MB kernel partitions

Gives us room to grow.  All the cool kids are doing it.

BUG=chromium:898971
TEST=build_image

Change-Id: I2423b2a192542a08911a77ab8b061cf6c9aa9501
Reviewed-on: https://chromium-review.googlesource.com/1301549
Commit-Ready: Douglas Anderson <dianders@chromium.org>
Tested-by: Douglas Anderson <dianders@chromium.org>
Tested-by: Karthikeyan Ramasubramanian <kramasub@chromium.org>
Reviewed-by: Furquan Shaikh <furquan@chromium.org>
Reviewed-by: Justin TerAvest <teravest@chromium.org>

[modify] https://crrev.com/fa4ef45c1fe9558a740e64df9564d2911f8674dd/baseboard-octopus/scripts/disk_layout.json

So I _think_ we're OK now.  In crrev.com/c/1291650 Stephen found that in some cases we also need to bump up syslinux to hold bigger kernels.  I don't know much about the syslinux partition in general since ARM boards don't use it.  Is it important that we bump that up too, or can we get by with 32M for the kernel partitions?

In any case, calling this Fixed and someone can reopen if we also need to bump up syslinux.

if by "syslinux partition" you're referring to the EFI partition, then yes, bumping it up sounds fine

how it's used on devices depends heavily on the board/execution environment. we can just say that it needs to be as big as the kernel partition to simplify.

> if by "syslinux partition" you're referring to the EFI partition, then yes, bumping it up sounds fine

How about if I call it "partition 12"?

---

> how it's used on devices depends heavily on the board/execution environment. 
> we can just say that it needs to be as big as the kernel partition to simplify.

It used to be twice as big as the kernel partitions (kernel partitions were 16 MB and it was 32 MB).  The above CLs bump kernel partitions to 32 M but don't touch partition 12.

Should I go through and submit another round of CLs to bump partition 12 up to 64 M?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/23b6878278cb47d4222c6b90fe20916c644fc3f6

commit 23b6878278cb47d4222c6b90fe20916c644fc3f6
Author: Douglas Anderson <dianders@chromium.org>
Date: Thu Jan 10 04:12:41 2019

baseboard-kalista: 32 MB kernel partitions

Gives us room to grow.  All the cool kids are doing it.

BUG=chromium:898971
TEST=build_image

Change-Id: I0fd5168bce89b3d25702d14afb3c33fa1457ee8f
Reviewed-on: https://chromium-review.googlesource.com/1301546
Commit-Ready: Douglas Anderson <dianders@chromium.org>
Tested-by: Zhuohao Lee <zhuohao@chromium.org>
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>
(cherry picked from commit 0f3302d40d662f8f3e0c630de4139c2f47162a26)
Reviewed-on: https://chromium-review.googlesource.com/c/1401913
Reviewed-by: Zhuohao Lee <zhuohao@chromium.org>
Commit-Queue: Zhuohao Lee <zhuohao@chromium.org>

[modify] https://crrev.com/23b6878278cb47d4222c6b90fe20916c644fc3f6/baseboard-kalista/scripts/disk_layout.json