https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1226282 would probably fix this, however I am still somewhat hesitant to just burn more of the space we have reserved.

I wonder if we have stats somewhere of rootfs size over image versions so we can find what made this not fit.

+ this weeks gardeners

Unfortunately the output from successful builds doesn't include a breakdown of disk usage, but the largest files on the failing image are:

38383616 opt/google/chrome/resources.pak
170364928 opt/google/chrome/chrome
465408000 opt/google/containers/android/system.raw.img

So the Android system.raw.img is by far the largest file here, even though it may not be the straw that broke the proverbial camel's back.

On my week old Kevin image, system.raw.img is actually larger than that (480......), so it probably was something else getting larger. My /opt/google/chrome/chrome image on Kevin is only 96......, so that could be the difference. I'm trying to build 'cyan' locally.

My guess would be that this might be related to putting libassistant on all builds?

In such a case this is probably expected, +xiaohuic to confirm.

Maybe, but the cl has been reverted for other reasons, so if it is the cause, it should be fixed now.

https://bugs.chromium.org/p/chromium/issues/detail?id=884566

just FYI building libassistant adds about 10MB 

It looks like this was just a small increase bringing us over the threshold. 

Here's a link to the rootfs size perf dashboard (thanks vapier@!):
https://chromeperf.appspot.com/report?sid=648c989a1733f30c47f9b3b40791cefbd16a0c990cc14657bfac1d65b75ad0b9&start_rev=34880001087800000&end_rev=35520061107400000

(You'll need to log in to see it).

We saw a large increase during 70, this last one was small, just a few MB.

I'm not sure whether we should try to identify the increase in 70, or audit current system usage and maybe break down the tests by package (at least for the larger ones).

We are now failing on the CQ also, caroline, cave, chell, and sentry all started failing on the last run.

https://chromeperf.appspot.com/report?sid=61869f77e0af23493219a398ae2f96656a90d11ab1a3664aec73a1c5f18133ac&start_rev=34880001087800000&end_rev=35520061107400000

We seem to have had growth across the board with R70 in a 20MB or so jump that seems to have put us close to the edge, and now slightly more growth has knocked us over, it was something in https://crosland.corp.google.com/log/10926.0.0..10927.0.0 I highly suspect what is causing this is rooted in https://chrome-internal-review.googlesource.com/c/chromeos/overlays/chromeos-overlay/+/652593

I am not sure if we can revert the drivefs stuff, but it is not just coral that it does not fit into really.

We can buy more time by increasing all of these to 1.8GB but that is really a bandaid, we need to find a way to stop growth on these older systems, as we can never go over 2GB.

I think the short of it is, if we keep drivefs, and we are going to enable the assistant on everything, we probably need to grow all of our rootfs sizes globally to at least 1.8G.

Not sure whom would need to sign off on it or if we can just go for it, we will probably never get the space back.

https://chromium-review.googlesource.com/c/chromiumos/overlays/board-overlays/+/1232500 should paper over this problem for a while longer at least for the boards that just stared failing the CQ.

Thanks Bernie for setting this up.  If no one objects, I will try to reland building libassistant tomorrow. 
https://chromium-review.googlesource.com/c/chromium/src/+/1232194

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/04e8e12b3a0440603ff25c77e5ee29db2a9c8440

commit 04e8e12b3a0440603ff25c77e5ee29db2a9c8440
Author: Bernie Thompson <bhthompson@google.com>
Date: Wed Sep 19 15:59:28 2018

Expand glados and kunimitsu derivatives rootfs to 1.8G

BUG= chromium:884709 
TEST=None

Change-Id: I1781ec6c55010e3c332cd9e6bac411b07bab08ad
Reviewed-on: https://chromium-review.googlesource.com/1232500
Commit-Ready: Bernie Thompson <bhthompson@chromium.org>
Tested-by: Bernie Thompson <bhthompson@chromium.org>
Reviewed-by: Joel Kitching <kitching@chromium.org>

[modify] https://crrev.com/04e8e12b3a0440603ff25c77e5ee29db2a9c8440/baseboard-kunimitsu/scripts/disk_layout.json
[modify] https://crrev.com/04e8e12b3a0440603ff25c77e5ee29db2a9c8440/baseboard-glados/scripts/disk_layout.json

 Issue 887764  has been merged into this issue.