There is no single CL in common in all three runs: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildCompare?lastN=3&masterBuilder=master-paladin&botConfig=coral-paladin&buildBotMaster=chromeos#highlightKeyValues=status%E2%86%92fail&operator=and

I think this is just a matter of the astronaut image being too close to limit and getting pushed over the edge by any CLs that affect it.

This is probably an indirect consequence of the recent move of modem firmware back to rootfs https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-coral-private/+/583895. That increased rootfs usage, and I'm guessing something else (ARC++ or Chrome) grew a bit recently too and took the board over the edge.

To unblock the CQ we could at least temporarily revert the modem firmware changes, but there's more than 1 I think, don't know how simple it is. Eric, what do you think?

Reverting https://chromium-review.googlesource.com/c/chromiumos/platform2/+/952624 will get modemfwd to load firmware via the component updater again. Reverting https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-coral-private/+/583895 will fix the space issue. The cros-cellular.eclass change isn't a necessary revert to fix this.

+ARC Constable: Can you help us determine when the last ARC roll was and if it is correlated with this failure?

taking a look at it.

This is a product issue.

Android tag 4643389 was first included in build 2736, it has larger size than the previous one (tag 4641729).

Anyone know the reason why we use 15MB as a "failure" and not a warning here? For instance, it used to be 20MB but was reduced once before (https://chromium-review.googlesource.com/386276).

FYI: coral-paladin has been marked experimental for this bug, so this isn't blocking the CQ. Still P0 though. We want coral back in + other builds / developer builds are still broken.

warning vs failure: I think warnings go unnoticed in cases like this. When we hit this roof, we need immediate action, so failure seems correct to me.

I've posted https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/957450 to only require 10MiB free (rather than 15MiB), in case we are ok with reducing the minimum amount of free space.

Posted revert CLs 
https://chromium-review.googlesource.com/c/chromiumos/platform2/+/957482
https://chromium-review.googlesource.com/c/chromiumos/platform2/+/957482
for modemfwd to load firmware from the component updater instead of adding to rootfs

Let modemfwd load the firmware from the component
https://chromium-review.googlesource.com/c/chromiumos/platform2/+/957482

Remove firmware from rootfs
https://chrome-internal-review.googlesource.com/c/chromeos/overlays/overlay-coral-private/+/586098

The latter should reduce rootfs usage by ~30MB and make the paladin work again. We shouldn't chump I think, it's good if they go through the CQ. coral-paladin is marked experimental anyway


Once that's in we can go back to a combination of:
- understand why ARC++ grew and by how much
- increase the size of the rootfs to get more overall space back
- reduce the size of the firmware files stored on disk. afaict they're uncompressed, it would be a win both for the component case and the rootfs case
- other improvements

Thank you for including both CLs norvez, I realize now I copied https://chromium-review.googlesource.com/c/chromiumos/platform2/+/957482 twice.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/overlay-coral-private/+/96a66d4b77643266c3f398181facb488cfb5c59d

commit 96a66d4b77643266c3f398181facb488cfb5c59d
Author: Mitchell Gildenberg <mgild@google.com>
Date: Sun Mar 11 01:44:43 2018

coral-paladin was green last night, lowering the priority. Recent runs are red again but due to a different issue (Android not starting).

Filed  Issue 821016  for the new type of failures (Android did not boot)

coral-paladin is green again, closing.

Follow-up bugs FYI:
Compress modem firmware: b/74552049
Increase size of the rootfs for coral:   Issue 821473 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/29617f6e6019b333af2f0ad9b3354dcbf200d228

commit 29617f6e6019b333af2f0ad9b3354dcbf200d228
Author: Nicolas Norvez <norvez@chromium.org>
Date: Thu Mar 15 00:23:14 2018

coral: increase rootfs size by 30MB

coral rootfs is running out of space, give it a bit more margin. Add
30MB to compensate for the size of the modem firmware.

BUG= chromium:821473 
BUG= chromium:820505 
TEST=build_image, see rootfs partition is 1800MB (460800 4K blocks)

Change-Id: I03c31a786652c20d96e9f0ca752aed442f40d51c
Reviewed-on: https://chromium-review.googlesource.com/961661
Commit-Ready: Nicolas Norvez <norvez@chromium.org>
Tested-by: Nicolas Norvez <norvez@chromium.org>
Reviewed-by: Amin Hassani <ahassani@chromium.org>

[modify] https://crrev.com/29617f6e6019b333af2f0ad9b3354dcbf200d228/baseboard-coral/scripts/disk_layout.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/c34bda4100a54fabe0d9955c14534dde8c742b57

commit c34bda4100a54fabe0d9955c14534dde8c742b57
Author: Nicolas Norvez <norvez@chromium.org>
Date: Fri May 04 16:23:30 2018

coral: increase rootfs size by 30MB

coral rootfs is running out of space, give it a bit more margin. Add
30MB to compensate for the size of the modem firmware.

BUG= chromium:821473 
BUG= chromium:820505 
TEST=build_image, see rootfs partition is 1800MB (460800 4K blocks)

Change-Id: I03c31a786652c20d96e9f0ca752aed442f40d51c
Reviewed-on: https://chromium-review.googlesource.com/961661
Commit-Ready: Nicolas Norvez <norvez@chromium.org>
Tested-by: Nicolas Norvez <norvez@chromium.org>
Reviewed-by: Amin Hassani <ahassani@chromium.org>
(cherry picked from commit 29617f6e6019b333af2f0ad9b3354dcbf200d228)
Reviewed-on: https://chromium-review.googlesource.com/1044275
Reviewed-by: Nicolas Norvez <norvez@chromium.org>
Commit-Queue: YH Lin <yueherngl@chromium.org>
Tested-by: YH Lin <yueherngl@chromium.org>

[modify] https://crrev.com/c34bda4100a54fabe0d9955c14534dde8c742b57/baseboard-coral/scripts/disk_layout.json