On R69-10773.0.0:
localhost ~ # df -B1 --print-type / | tail -1
/dev/root      ext2 1806172160 1762852864  43319296  98% /

On R69-10774.0.0:
df -B1 --print-type / | tail -1
/dev/root      ext2 1806172160 1790795776  15376384 100% /

The root cause is that Chrome has got bigger.

On R69-10773.0.0:
localhost ~ # du -sh /opt/google/chrome/
332M    /opt/google/chrome/

On R69-10774.0.0:
localhost ~ # du -sh /opt/google/chrome/
359M    /opt/google/chrome/

crosreview.com/1096617 has been uploaded to adjust the test
so that it will pass; I'd recommend we chump that (or do
something similar) while we sort out the permanent fix.

On the oncall chat, it was identified that we enabled building
Chrome with CFI around the time that Chrome got bigger.  That
was expected to increase the size by ~10MB, which seems consistent
with our problem.

Assigning to a randomly selected sheriff for action.

P0 because the CQ is blocked.

Can we add reef to PFQ so that we can find this type of error earlier?

Clarifying action required here:
  * This bug is about "make sure that we believe reef has enough
    root FS space."
  * There's no plan to file a separate bug for the mitigation
    steps to get the CQ running again; we'll track that here.
  * Bug 851987 has been filed as a suggestion for how to improve
    detecting this sort of failure in future.

> Can we add reef to PFQ so that we can find this type of error earlier?

Reef was already in the PFQ, but it didn't run the test to find this
problem.  Bug 851987 describes one easy way that we could detect this
problem in the PFQ.

Is it feasible to increase the rootFS size for reef? 

Doesn't that involve re-partitioning the disk? 

> Doesn't that involve re-partitioning the disk? 

No.  The disk_layout.conf has two sizes:
 A) The total partition size.
 B) The maximum FS size within the partition.

B isn't allowed to be bigger than A, but usually, it's
assigned to be somewhat smaller.  I checked the file for
Reef family devices.  The FS size is 1750 MiB, and the
partition gets 2000 MiB.  So, we have room to grow.

Note that we're still running a big tight for space on reef;
someone should maybe undertake to see if there are big space
hogs that need to be dealt with.

For the record...

$ cd src/overlays/
$ cat baseboard-reef/scripts/disk_layout.json 
{
  "_comment": "See http://www.chromium.org/chromium-os/building-chromium-os/disk-layout-format",
  "metadata":{
    "rootdev_base": "/sys/devices/pci0000:00/0000:00:1c.0/mmc_host/mmc1/mmc1:0001/block/mmcblk1"
  },
  "parent": "legacy_disk_layout.json",
  "layouts": {
    # common is the standard layout template.
    "common": [
      {
        # Slot A rootfs. Adjust to allow for more space.
        "num": 3,
        "fs_size_min": "1750 MiB",
        "size_min": "2000 MiB"
      }
    ],
    # Used for factory install images. No need to adjust fs size or partition
    # size since these do not need more space.
    "factory_install": [
      {
        "num": 3,
        "fs_size_min": "0 MiB",
        "size_min": "0 MiB"
      }
    ]
  }
}

Ah, thanks for the insight. So can we increase it just for reef? Although, perhaps down the line we may have to do the same for other boards...

... Also, one final follow-up:  Increasing the designated "fs_size_min"
from "1750 MiB" to "1800 MiB" would nominally be a permanent fix.  The
one drawback is that it moves us from 250 free MiB down to only 200.
So, we should consider some sort of look-see to confirm that that size
doesn't pose some future risk that we should try to mitigate right now.

> Ah, thanks for the insight. So can we increase it just for reef?

We can trivially increase it for "baseboard-reef", which is probably
good enough.  I believe it's not hard to create a reef-only config
that overrides the baseboard config, but that feels like more trouble
than it would be worth.

> Although, perhaps down the line we may have to do the same for other boards...

Matthew 6:34

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/cbc145ab5410d4cd16c018437e12213ab368c3aa

commit cbc145ab5410d4cd16c018437e12213ab368c3aa
Author: Yunlian Jiang <yunlian@google.com>
Date: Tue Jun 12 19:56:21 2018

build_RootFilesystemSize: change the free space threshold.

This changes the size of free space from 15 MB to 11 MB to make
reef-paladin pass.

BUG=chromium:851897
TEST=the threshold is changed from 15 MB to 11 MB.

Change-Id: Id81ac2d926282f2c61954cd8f614f8130084cf4b
Reviewed-on: https://chromium-review.googlesource.com/1096617
Reviewed-by: Lann Martin <lannm@chromium.org>
Tested-by: Aseda Aboagye <aaboagye@chromium.org>

[modify] https://crrev.com/cbc145ab5410d4cd16c018437e12213ab368c3aa/client/site_tests/build_RootFilesystemSize/build_RootFilesystemSize.py

I've uploaded https://crrev.com/c/1097803 to bump up the rootFS size for baseboard-reef.

I checked 10773.0.0 vs 10774.0.0 and found the following size differences within /opt/google/chrome:

chrome: 147 M -> 156 M
nacl_helper: 9.1 M -> 26 M

Furthermore, when I built chrome with simple chrome at the appropriate chrome revision (including updating the cros SDK version), both binaries were 132 M, so this appears to be a difference in how chrome is built and not a change to the chrome repo that is at fault.

Simple chrome does not use AFDO, ThinLTO and CFI, that's the reason of difference. The size increase from nacl_helper is suspicious, I will take a look at it.

Is it possible to support any/all of those in simple chrome? Bisecting Chrome changes within a chroot is... challenging.

Enabling ThinLTO and CFI for simple chrome is relatively easy, it only needs to change the gn files. For AFDO, it needs to modify the CFLAGS and download the AFDO profile data.
I will update the instructions somewhere (maybe https://chromium.googlesource.com/chromiumos/docs/+/master/simple_chrome_workflow.md ?)

What is the current plan?
Is the consensus that the cause is that Chrome has grown? That would make sense since betty and novato lacked disk space too, presumably all boards (or maybe x86?) are impacted.

Either way we should consider reverting https://chromium-review.googlesource.com/1096617 and put the threshold back to 15MB.

Yes, the size of Chrome has grown because of CFI.
http://crrev.com/c/1097803 will increase the size of rootfs for reef.

If no one objects, I'll CQ+1 that.

I just finished building chrome from source at ToT and @ 10773.0.0 (i.e. from R69-10773.0.0), both with ToT chromeos. Both results were larger (~359 M), confirming that this was not a change in the chrome source.

Enabling CFI makes sense. Does anyone know what CrOS change enabled CFI to cause the jump, for posterity?

Also, do we have numbers to justify the size increase?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/92c527d2a056cd17438250b3a8ebea22daa2cce5

commit 92c527d2a056cd17438250b3a8ebea22daa2cce5
Author: Aseda Aboagye <aaboagye@google.com>
Date: Tue Jun 19 00:19:21 2018

baseboard-reef: Increase rootFS size to 1800 MiB.

reef-paladin was running out of space in the rootFS due to a recent
change to enable CFI when building Chrome which increased the Chrome
binary a bit.  This commit simply bumps up the rootFS size so that
build_RootFilesystemSize may pass again.

BUG=chromium:851897
BRANCH=None
TEST=None

Change-Id: I10b76e8906153902d525f6dd4737a5fd5c0dd9aa
Signed-off-by: Aseda Aboagye <aaboagye@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1097803
Commit-Ready: Aseda Aboagye <aaboagye@chromium.org>
Tested-by: Aseda Aboagye <aaboagye@chromium.org>
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>

[modify] https://crrev.com/92c527d2a056cd17438250b3a8ebea22daa2cce5/baseboard-reef/scripts/disk_layout.json

Does it related to continuous failure of reed-paladin on SkylabHWTest ?

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?builderName=reef-paladin&buildNumber=6361

This is marked as P0, so I think it should be worked on or triaged.

s/reed-paladin/reef-paladin/

> Does it related to continuous failure of reed-paladin on SkylabHWTest ?

I don't see any reason to think so. That appears to be a failure in the skylab code.

This is not a P0.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/dd1da5f0cafa2297660279610530a9a6481fd5e1

commit dd1da5f0cafa2297660279610530a9a6481fd5e1
Author: Alexis Savery <asavery@chromium.org>
Date: Mon Sep 17 15:54:29 2018

baseboard-strago: Increase rootFS size to 1800 MiB

wizpig-paladin is running out of space. This bumps up the rootFS
size so that build_RootFilesystemSize will pass again.

BUG=chromium:851897
TEST=None

Change-Id: Ibc8c8c5762bf53878191884c501b110806f7f021
Reviewed-on: https://chromium-review.googlesource.com/1226282
Reviewed-by: Yu Zhao <yuzhao@chromium.org>
Tested-by: Alexis Savery <asavery@chromium.org>
Commit-Queue: Bernie Thompson <bhthompson@chromium.org>

[modify] https://crrev.com/dd1da5f0cafa2297660279610530a9a6481fd5e1/baseboard-strago/scripts/disk_layout.json