Amin Hassani has a fix [CL/757158]

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/crosutils/+/33e68f9e22a5433852d669c099630a1b410bc5f6

commit 33e68f9e22a5433852d669c099630a1b410bc5f6
Author: Amin Hassani <ahassani@google.com>
Date: Wed Nov 08 23:10:10 2017

build_image: Do not do fstrim for unmounted filesystems

CL:751809 removed a check to bypass zeroing unallocated blocks when the
filesystem is unmounted like squashfs or ubifs partitions. This CL puts
that check back.

BUG=chromium:780291,chromium:782453,b:69042809
TEST=precq runs

Change-Id: Ia81926c3d1821e1ee2ea84b7cabe846f8fdbde2d
Reviewed-on: https://chromium-review.googlesource.com/757158
Commit-Ready: Ian Coolidge <icoolidge@google.com>
Tested-by: Ian Coolidge <icoolidge@google.com>
Reviewed-by: Gwendal Grignou <gwendal@google.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/33e68f9e22a5433852d669c099630a1b410bc5f6/build_library/base_image_util.sh

The change above landed in 10111. However, looking at the builder, 10104 with CL:751809 only was successful in gonzo builder, and claimed the rootfs was trimmed.

The problem is some servers have a newer version of fstrim that does not complain when the filesystem does not support fstrim (v2.25) [cros-beefy461-c2] while others don't [ build69-m2]:

[commit 36c370cbf1481aa8724dff8b7b7fec4a8ba9930b in git://git.kernel.org/pub/scm/utils/util-linux/util-linux.git]:
...
     * silently ignore EOPNOTSUPP (for example LUKS without discard) and ENOTTY
       (for example vfat) errors
...

build_image runs inside the chroot, so the version of util-linux/fstrim inside the chroot is the only one that should really matter

Building Chromium OS for daisy on my laptop. It has util-linux 2.28.2 installed in chroot and still fails. It has nvme and sata m.2 drives (both support TRIM and are mounted with discard).

14:08:13: INFO: Searching for unused gconv files defined in /mnt/host/source/src/build/images/daisy/R65-10206.0.2017_12_11_1404-a1/rootfs/usr/lib/gconv/gconv-modules
14:08:13: INFO: Will search for 1131 strings in 33 files
14:08:14: INFO: Done. Using 59 gconv modules. Removed 187 unused modules (3668.4 KiB) and 2 unused dependencies (174.7 KiB)
insert_container_publickey.sh: INFO   : Container verification key was installed. Do not forget to resign the image!
INFO    : The target platform does not use bootloader templates.
Cleaning up /usr/local symlinks for /mnt/host/source/src/build/images/daisy/R65-10206.0.2017_12_11_1404-a1/stateful/dev_image
INFO    : Building dev-install package lists
INFO    : Zeroing freespace in /mnt/host/source/src/build/images/daisy/R65-10206.0.2017_12_11_1404-a1/rootfs
fstrim: /mnt/host/source/src/build/images/daisy/R65-10206.0.2017_12_11_1404-a1/rootfs: the discard operation is not supported
INFO    : Unmounting image from /mnt/host/source/src/build/images/daisy/R65-10206.0.2017_12_11_1404-a1/stateful and /mnt/host/source/src/build/images/daisy/R65-10206.0.2017_12_11_1404-a1/rootfs
Cleaning up /usr/local symlinks for /mnt/host/source/src/build/images/daisy/R65-10206.0.2017_12_11_1404-a1/stateful/dev_image
An error occurred in your build so your latest output directory is invalid.
Would you like to delete the output directory (y/N)? y
Deleted /mnt/host/source/src/build/images/daisy/R65-10206.0.2017_12_11_1404-a1

(cr) ((814581e...)) dbehr@pixel ~/trunk/src/scripts $ emerge -p util-linux

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R    ] sys-apps/util-linux-2.28.2
(cr) ((814581e...)) dbehr@pixel ~/trunk/src/scripts $ /sbin/fstrim --version
fstrim from util-linux 2.28.2

#6: which kernel is your laptop running on? On which type of filesystem have you installed chromium source?
It looks like the ioctl FSTRIM is failing on your machine.

I think I'm having a similar issue. I just re-sync'd and checked that the latest util-linux is installed (sys-apps/util-linux-2.28.2). Currently my kernel is 
4.10.0-40-generic #44~16.04.1-Ubuntu SMP (Ubuntu 16.04.3 LTS), but I have another system using a 4.14 kernel.

I've tried on a few different filesystems with the same result - Plain HDD with ext4, HDDs in RAID using LUKS and btrfs, plain SSD with ext4, directory on SSD using ecryptfs, SSD using LUKS, etc. I'm not an expert on TRIM, but AFAICT it shouldn't apply to HDDs. I am able to run `fstrim -v` outside the chroot on my SSDs (I had to add --allow-discards to the `cryptsetup luksOpen` command for LUKS).

Here are the last several lines of output from build_image on an Ubuntu system using SSD:
Started virtual/target-os-1-r3 (logged in /tmp/target-os-1-r3-RIqKAY)
Completed virtual/target-os-1-r3 (in 0m3.5s)
Pending 0/348, [Time 15:32:49 | Elapsed 10m48.8s | Load 13.43 20.25 19.65]
Merge complete
Done
INFO    : Generating license credits page. Time:

real    0m8.932s
user    0m8.400s
sys     0m0.352s
15:32:59: INFO: Searching for unused gconv files defined in /mnt/host/source/src/build/images/kevin/R64-10166.0.2017_12_12_1521-a1/rootfs/usr/lib/gconv/gconv-modules
15:33:01: INFO: Will search for 1131 strings in 34 files
15:33:02: INFO: Done. Using 60 gconv modules. Removed 186 unused modules (3659.1 KiB) and 2 unused dependencies (174.7 KiB)
insert_container_publickey.sh: INFO   : Container verification key was installed. Do not forget to resign the image!
INFO    : The target platform does not use bootloader templates.
Cleaning up /usr/local symlinks for /mnt/host/source/src/build/images/kevin/R64-10166.0.2017_12_12_1521-a1/stateful/dev_image
INFO    : Building dev-install package lists
INFO    : Zeroing freespace in /mnt/host/source/src/build/images/kevin/R64-10166.0.2017_12_12_1521-a1/rootfs
fstrim: /mnt/host/source/src/build/images/kevin/R64-10166.0.2017_12_12_1521-a1/rootfs: the discard operation is not supported
INFO    : Unmounting image from /mnt/host/source/src/build/images/kevin/R64-10166.0.2017_12_12_1521-a1/stateful and /mnt/host/source/src/build/images/kevin/R64-10166.0.2017_12_12_1521-a1/rootfs
Cleaning up /usr/local symlinks for /mnt/host/source/src/build/images/kevin/R64-10166.0.2017_12_12_1521-a1/stateful/dev_image
An error occurred in your build so your latest output directory is invalid.
Would you like to delete the output directory (y/N)? 

the point of fstrim here isn't for the underlying storage, it's to cheaply and cleanly throw away unused blocks (so they get zeroed)

are you saying you're building CrOS inside of an encrypted DM block that you created ?

Re #9: I'm building CrOS inside the CrOS SDK without any local changes, simply running thru the repo init, repo sync, cros_sdk --enter, setup_board, build_packages, and build_image routine.

I was attempting add some data points for the question raised by Gwendal in #7, though perhaps I was confused about what he was asking for. I thought there may have been some issue due to my hardware/storage setup (not specific to where I have CrOS checked out) but I've tried in a few different environments and so far the issue has been 100% reproducible.

you mentioned cryptsetup and i thought you meant you were rolling your own thing

is your sdk using the image approach or is it just an untarred directory ?  what does `ls -ld ~chromiumos/chroot*` show ?  and `mount | grep chroot` ?

@#7
Linux 4.13.8 #1 SMP PREEMPT Fri Dec 8 14:30:02 PST 2017 x86_64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz GenuineIntel GNU/Linux
/dev/nvme0n1p2 on / type ext4 (rw,noatime,discard,data=ordered)
/dev/sda1 on /data type ext4 (rw,noatime,nodiratime,discard,data=ordered)

Re #11: I think it's using the loopback image. Perhaps I should try --nouse-image?

Here's one of the environments I tried (SSD with ext4 filesystem encrypted using LUKS)
$ ls -ld chroot*
drwxr-xr-x 22 root root         4096 Dec 10 18:17 chroot
-rw-r--r--  1 root root 536870912001 Dec 14 23:40 chroot.img

$ mount | grep chroot
/dev/mapper/cros_mnt+ssd+src+git--projects+chromiumos--pub+chroot_000-chroot on /mnt/ssd/src/git-projects/chromiumos-pub/chroot type ext4 (rw,noatime,stripe=64,data=ordered)

using a LVM and dm-thin might be an explanation. Does it work better with --nouse-mage [for reference, cl/553462/]?

cros_sdk runs fstrim when entering the chroot if it thinks chroot.img has a lot of unused space.  After that, being on an image or not shouldn't make a difference because the path where build_image does its work points back outside the chroot.

Re #14: Unfortunately --nouse-image did not make a difference.

Re #15: Yes, when I enter the chroot I see:
10:44:30: NOTICE: /mnt/ssd/src/git-projects/chromiumos-pub/chroot.img is using 25 GiB more than needed.  Running fstrim.

That step works fine. I can also run `sudo fstrim -v /` inside the chroot and it works. The error only happens during build_image fstrim is run on the newly generated rootfs (see output in #8).

I tried with kernel 4.14 (chromeos-kernel-4_14 on eve + crouton - ubuntu 16.04). I had to set --nouse-image as LVM was not working properly, but fstrim and Zero'ing the base image worked as expected.

Re #18: Did you try building an image for any mainboard target? In my case I think I was only having problems when trying to build an image for kevin.

Looking deeper into: in [block/loop.c] lo_discard, we punch hole in the underlying file by calling file->f_op->fallocate(file...

If the underlying filesystem has fallocate, loopback will call it, but it will propagate an error back to fstrim. Check how can fallocate fail in the normal case. Using fstrim compiled with warning enabled could help.

I can see fallocate failing when ext4 calls ext4_punch_hole that wants to propagate the trim downward to an encrypted block, as it may require a reencryption. In that case, the filesystem should not be mounted with discard option, so be sure the discard/trim chain ends before it fails.

Hi Gwendal,
Thanks for looking into this more. I poked around util-linux but didn't much in fstrim that will spit out more errors. I enabled HAVE_ERR, HAVE_ERRX, HAVE_WARN, and HAVE_WARNX by setting environment variables as per https://kernel.googlesource.com/pub/scm/utils/util-linux/util-linux/+/master/Documentation/howto-debug.txt , but I'm not sure how useful it is (see attached file).

I also added a debug print after the FITRIM ioctl() in fstrim.c, compiled and replaced /usr/sbin/fstrim (and /usr/sbin/.lib/lt-fstrim, whatever that is), and it showed errno == 25.

LMK if that is sufficient to work with or if there are other ways I can improve debug output. You can also attach a patch for util-linux and I'll compile and run it.

Deleted: build_image_with_util_linux_debug_enabled.txt 52.9 KB

build_image_with_util_linux_debug_enabled.txt 52.9 KB View Download

seems to be cros-specific, and not buildbot-specific