So the failure is in BuildImages, and it does look like we're running out of space on the target.

First failing build seems to be: https://build.chromium.org/p/chromiumos.chromium/builders/amd64-generic-tot-asan-informational/builds/12177

I looked at the chromium and chromeos CLs and nothing obvious stood out as increasing the size of the binaries.

One (easy) option is to increase this size to 4gb, but it would be good to figure out what caused this increase? I'm not sure if we were really close to the threshold and some small random jump broke us.

Could you try to repo this locally and do a bisect?

You can also see that the chromiumos waterfall has some ASAN bots:
https://build.chromium.org/p/chromiumos/waterfall

The x86-generic asan bot (which ought to be deprecated) is able to build images just fine:
https://build.chromium.org/p/chromiumos/builders/x86-generic-asan

The amd64-generic asan bot is failing with the same symptom:
https://build.chromium.org/p/chromiumos/builders/amd64-generic-asan

You should rope in the chromeos sheriffs to help with this as these bots are their responsibility.

+chromeos sheriffs.

Yes, this problem seems to be only specific to amd64-generic asan bots.

Correction to #0, it's failing the BuildImage* step.

The first attempt fails at copying resources.pak:


chromeos-chrome-58.0.3021.0_alpha-r1: !!! copy /build/amd64-generic/tmp/portage/chromeos-base/chromeos-chrome-58.0.3021.0_alpha-r1/image/opt/google/chrome/resources.pak -> /mnt/host/source/src/build/images/amd64-generic/R58-9308.0.2017_02_22_1503-a1/rootfs/opt/google/chrome/resources.pak failed.
chromeos-chrome-58.0.3021.0_alpha-r1: !!! [Errno 28] No space left on device
chromeos-chrome-58.0.3021.0_alpha-r1: >>> Failed to install chromeos-base/chromeos-chrome-58.0.3021.0_alpha-r1 to /mnt/host/source/src/build/images/amd64-generic/R58-9308.0.2017_02_22_1503-a1/rootfs/, Log file:
chromeos-chrome-58.0.3021.0_alpha-r1: >>>  '/build/amd64-generic/tmp/portage/logs/chromeos-base:chromeos-chrome-58.0.3021.0_alpha-r1:20170222-230441.log'
=== Complete: job chromeos-chrome-58.0.3021.0_alpha-r1 (0m20.8s) ===
Failed chromeos-base/chromeos-chrome-58.0.3021.0_alpha-r1 (in 0m20.8s), retrying later.

However, it doesn't rank high up among the sorted list of packages by size:

Packages failed:
	chromeos-base/chromeos-chrome-58.0.3021.0_alpha-r1
ERROR   : Here are the biggest [partially-]extracted files (by disk usage):
1908736 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_sz_nexe
2052096 usr/bin/firewalld
2060288 usr/sbin/imageloader
2088960 usr/bin/update_engine_client
2097152 lib/modules/4.4.44-07203-g1adfab2/kernel/drivers/gpu/drm/radeon/radeon.ko
2138112 usr/bin/mmcli
2154496 usr/lib64/va/drivers/i965_drv_video.so
2170880 opt/google/chrome/pnacl/pnacl_public_x86_64_ld_nexe
2215936 usr/lib64/libcrypto.so.1.0.0
2273280 usr/share/fonts/tibt-jomolhari/Jomolhari-alpha3c-0605331.ttf
2306048 usr/sbin/lockbox-cache
2310144 usr/bin/lorgnette
2351104 lib/firmware/iwlwifi-8000C-14.ucode
2457600 usr/bin/chaps_client
2576384 opt/google/chrome/chrome_100_percent.pak
2580480 opt/google/mtpd/mtpd
2580480 usr/bin/buffet_client
2588672 usr/bin/permission_broker
2641920 usr/lib64/libpoppler.so.52.0.0
2654208 usr/bin/mist
2682880 usr/lib64/libmtp.so.9.3.0
2818048 usr/lib64/libndr-standard.so.0.0.1
2924544 usr/lib64/samba/libsmbd-base-samba4.so
3059712 usr/bin/dump_power_status
3100672 usr/bin/power_supply_info
3112960 usr/lib64/libshill-net-395517.so
3301376 sbin/crash_reporter
3592192 opt/google/chrome/nacl_helper_nonsfi
3719168 usr/bin/get_powerd_initial_backlight_level
3809280 opt/google/chrome/nacl_irt_x86_64.nexe
3813376 usr/share/fonts/ko-nanum/NanumMyeongjo.ttf
3858432 usr/bin/webservd
3911680 usr/bin/apmanager
3911680 usr/bin/metrics_daemon
4034560 opt/google/cros-disks/disks
4263936 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_pt-BR.zvoice
4304896 usr/share/fonts/ko-nanum/NanumGothicBold.ttf
4317184 usr/bin/peerd
4358144 usr/share/fonts/ko-nanum/NanumGothic.ttf
4399104 usr/lib64/libmm-glib.so.0.3.0
4489216 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-GB.zvoice
4653056 usr/share/fonts/ko-nanum/NanumMyeongjoBold.ttf
4849664 usr/share/chromeos-assets/input_methods/zhuyin/data.js
4861952 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-IN.zvoice
5120000 sbin/debugd
5177344 usr/share/chromeos-assets/input_methods/nacl_mozc/nacl_session_handler_x86_64.nexe
5414912 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-US.zvoice
5517312 usr/lib64/libbrillo-core-395517.so
5750784 usr/sbin/authpolicyd
5767168 usr/lib64/libpolicy-395517.so
5828608 usr/sbin/chapsd
6000640 usr/share/fonts/noto/NotoColorEmoji.ttf
6062080 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_ko-KR.zvoice
6152192 usr/share/chromeos-assets/input_methods/hangul/hangul_x86_64.nexe
6160384 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-US.zvoice
6205440 usr/bin/quipper
6303744 usr/bin/buffet
6352896 boot/vmlinuz-4.4.44-07203-g1adfab2
6467584 usr/share/chromeos-assets/input_methods/hangul/hanja.txt
6492160 usr/share/fonts/ja-ipafonts/ipag.ttc
6643712 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_nl-NL.zvoice
6660096 usr/lib64/libweave-395517.so
6819840 usr/sbin/tpm-manager
7090176 usr/sbin/cryptohome
7110656 usr/sbin/authpolicy_parser
7200768 opt/google/chrome/chrome_200_percent.pak
7417856 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_it-IT.zvoice
7512064 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_id-ID.zvoice
7598080 usr/lib64/dri/i965_dri.so
7598080 usr/lib64/dri/nouveau_vieux_dri.so
7598080 usr/lib64/dri/r200_dri.so
7598080 usr/lib64/dri/radeon_dri.so
8302592 usr/share/fonts/ja-ipafonts/ipam.ttc
8458240 usr/share/chromeos-assets/input_methods/pinyin/data.js
8474624 usr/sbin/ModemManager
8740864 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_de-DE.zvoice
9379840 usr/bin/powerd
9809920 sbin/session_manager
9973760 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_fr-FR.zvoice
10186752 opt/google/chrome/icudtl.dat
10473472 usr/sbin/cryptohomed
11149312 usr/sbin/update_engine
11603968 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_hi-IN.zvoice
12505088 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-ES.zvoice
14114816 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_llc_nexe
14225408 opt/google/chrome/libosmesa.so
15667200 usr/share/chromeos-assets/input_methods/nacl_mozc/zipped_data_oss
18702336 usr/share/fonts/notocjk/NotoSansCJK-Regular.ttc
19275776 usr/share/fonts/notocjk/NotoSansCJK-Bold.ttc

*** 24809472 opt/google/chrome/resources.pak#new ***

26275840 usr/lib64/dri/i915_dri.so
26275840 usr/lib64/dri/kms_swrast_dri.so
26275840 usr/lib64/dri/nouveau_dri.so
26275840 usr/lib64/dri/r300_dri.so
26275840 usr/lib64/dri/r600_dri.so
26275840 usr/lib64/dri/swrast_dri.so
29728768 usr/share/chromeos-assets/speech_synthesis/patts/tts_service_x86-64.nexe
40099840 usr/bin/shill
244199424 opt/google/chrome/nacl_helper
596746240 opt/google/chrome/chrome

Assigning to this weeks gardener; we need to fix this.

I think that will make a great starter bug for the deputy.

Is anyone looking at this? It's going to become harder to bisect this as time goes by, and we want to avoid handing it from gardener to gardener.

+gardeners.

Looking at the first failure
https://build.chromium.org/p/chromiumos.chromium/builders/amd64-generic-tot-asan-informational/builds/12177

it has 2 commits
CHUMP | chromiumos-overlay | manojgupta | 446119
CHUMP | chromite | ayatane | 445867

One of which is a toolchain change
https://chromium-review.googlesource.com/#/c/446119/

Notice that the image is full
ERROR   : Target image has run out of space:
ERROR   : Filesystem     1M-blocks  Used Available Use% Mounted on
ERROR   : /dev/loop0          1969  1969         0 100% /mnt/host/source/src/build/images/amd64-generic/R58-9308.0.2017_02_22_1503-a1/rootfs

And the binaries truly huge
26275840 usr/lib64/dri/i915_dri.so
26275840 usr/lib64/dri/kms_swrast_dri.so
26275840 usr/lib64/dri/nouveau_dri.so
26275840 usr/lib64/dri/r300_dri.so
26275840 usr/lib64/dri/r600_dri.so
26275840 usr/lib64/dri/swrast_dri.so
29728768 usr/share/chromeos-assets/speech_synthesis/patts/tts_service_x86-64.nexe
40099840 usr/bin/shill
244199424 opt/google/chrome/nacl_helper
596746240 opt/google/chrome/chrome

I wonder if the size before this toolchain change was still that large, unfortunately the artifacts for the last known good image might be gone and the old size wasn't recorded.
https://build.chromium.org/p/chromiumos.chromium/builders/amd64-generic-tot-asan-informational/builds/12176

Toolchain  team already filed a bug for this failure (https://bugs.chromium.org/p/chromium/issues/detail?id=695627).

Copying shawnn@ comment from there:

It looks like our rootfs size grew suddenly between .7 and .8:

https://chromeperf.appspot.com/report?sid=422068e0adb64425b2320337d23c7bb5202c4421147e3c6eaa33011defcd619c

This CL is the likely cause:

https://chromium-review.googlesource.com/#/c/443133/

That looks important, so we probably do want to bump the size of the partition.

 Issue 695627  has been merged into this issue.

Manoj, it is true that 
https://chromium-review.googlesource.com/#/c/443133/
increased all images by 20MB. But it did so before the toolchain change. By how much did the toolchain change increase ASAN image sizes? And is it normal that ASAN Chrome executable is 600MB now?

I don't particularly care about the ASAN image sizes (shipping image sizes). So, I am fine to increase the rootfs on those to 4GB to never see these failures again.

Toolchain should not have change anything at all. It applied a trivial upstream patch to fix an compiler error.

In any case, toolchain changes are not applicable immediately but have a special update process to go be effective. 
1) Toolchain changes go to chromiumos-sdk builder which creates compiler prebuilts. 2) The prebuilts are pushed later to builds in a separate change (e.g. look at https://chromium-review.googlesource.com/c/452541/)

But can you confirm that it is normal for ASAN chrome to be 600MB? If yes, then lets increase image size.

asan binaries have larger sizes so 600MB is probably not out of line.

+1 to increasing the rootfs size if that will make the problem go away since we do not ship ASAN images. It has now been several weeks since we have had any ASAN coverage running on a Chrome OS device.

CL to increase rootfs size at [1], not sure if we will land it though (pending further discussion).

1: https://chromium-review.googlesource.com/c/453921/

Don's concern is about ASAN images being tested on real hardware. As far as I can tell we are not doing this currently, the ASAN images are only used in vm_tests.

If we were to run them on hardware, it might be worth removing some of the graphics drivers (right now support for ATI etc.) But that would make them non-generic images. So I just don't see how we have generic+asan+64bit+2GB. One of them has to give. We want to keep asan. We don't care about 32bit anymore. So 2GB has to go.

Not that the images are ASAN, but that they have non-standard partition table.

Background:

The partition table can not be changed by a standard update, only during a USB recovery of a device (which fully wipes/repartitions the drive). In the lab, we MOSTLY provision devices via network, but use USB recovery on devices in a sufficiently bad state. 

So, my expectations are that:

A) You won't be able to do lab testing, since rootfs partitions are 2G and will break for network updates that try to install a rootfs file system bigger than that.

B) If these images ever escape into the USB recovery process, we'll end up with DUTs that seem normal except for stateful partitions that are 4G smaller than usual (2G per rootfs partition). That might not be enough for many lab operations, but nobody is likely to be able to easily understand why things are running out of space.

PS: As for growing rootfs partitions past 2G.... those partition sizes are effectively frozen at first FSI. We can grow them for new devices, but not existing hardware.

I added some more comments to the CL about ways to make it safer, mostly about making sure the images stay out of the lab.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/0d03f0f74706932433e8e9b789d46d920713633c

commit 0d03f0f74706932433e8e9b789d46d920713633c
Author: Jacob Dufault <jdufault@google.com>
Date: Thu Mar 16 21:23:50 2017

Increase rootfs size for amd64 asan builds to 4gb.

TEST=Manual inspection
BUG= chromium:697967 
Change-Id: Id7fed2d6c289847534963cbe368b72c1be122b51
Reviewed-on: https://chromium-review.googlesource.com/453921
Commit-Ready: Jacob Dufault <jdufault@chromium.org>
Tested-by: Jacob Dufault <jdufault@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/0d03f0f74706932433e8e9b789d46d920713633c/cbuildbot/config_dump.json
[modify] https://crrev.com/0d03f0f74706932433e8e9b789d46d920713633c/cbuildbot/chromeos_config.py

We are looking for this build now
https://build.chromium.org/p/chromiumos.chromium/builders/amd64-generic-tot-asan-informational/builds/12419

Still seeing:
ERROR   : Target image has run out of space:

Also:
return code: 1; command: /b/cbuild/shared_external/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpm1ReR3' 'FEATURES=separatedebug' 'CHROME_ORIGIN=LOCAL_SOURCE' -- ./build_image '--board=amd64-generic' --replace '--version=' '--builder_path=amd64-generic-tot-asan-informational/R59-9384.0.0-b12458' '--disk_layout=2gb-rootfs' test
cmd=['/b/cbuild/shared_external/chromite/bin/cros_sdk', 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpm1ReR3', 'FEATURES=separatedebug', 'CHROME_ORIGIN=LOCAL_SOURCE', '--', './build_image', '--board=amd64-generic', '--replace', '--version=', '--builder_path=amd64-generic-tot-asan-informational/R59-9384.0.0-b12458', '--disk_layout=2gb-rootfs', 'test'], cwd=/b/cbuild/shared_external, extra env={'PARALLEL_EMERGE_STATUS_FILE': '/tmp/tmpm1ReR3', 'FEATURES': 'separatedebug', 'CHROME_ORIGIN': 'LOCAL_SOURCE'}

The bot seems to still using a 2gb layout, I didn't have time to investigate why.

  INFO    : Using image type 2gb-rootfs

https://luci-logdog.appspot.com/v/?s=chromiumos%2Fbb%2Fchromiumos.chromium%2Famd64-generic-tot-asan-informational%2F12458%2F%2B%2Frecipes%2Fsteps%2FBuildImage%2F0%2Fstdout

Yeah, I saw that. I will pick up the investigation, we really really really need to fix this, so you can go ahead and assign this to me, but if you could find a moment to poke around a bit that might be helpful since I will pretty much be starting cold.

I've uploaded a CL here[1] to change the config type, but I am a bit nervous about the CL because amd64-generic-tot-asan-information has a buildslave_type = baremetal, and we can't run 4gb images on baremetal.

1: https://chromium-review.googlesource.com/c/457108/

Why not?

I was assuming baremetal => run on physical hardware. I'm guessing that's not the case?

baremetal => the *builder* runs on physical hardware, i.e. not a cloud instance.

This is (currently) necessary for builders that run VMTests.

It is unrelated to the DUT (Device Under Test) required to run HWTests.

As long as the builder is not configured to run HWTests it is fine.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/4a714c91b4a410e1e195ea2b9e5d0b78e94209e2

commit 4a714c91b4a410e1e195ea2b9e5d0b78e94209e2
Author: Jacob Dufault <jdufault@google.com>
Date: Wed Mar 22 00:07:15 2017

Increase rootfs size to 4gb for amd64-generic-tot-asan-informational.

TEST=Manual inspection
BUG= chromium:697967 

Change-Id: I421600e944f1929fba5393896424aa62bbecf8d1
Reviewed-on: https://chromium-review.googlesource.com/457108
Reviewed-by: Steven Bennetts <stevenjb@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Steven Bennetts <stevenjb@chromium.org>
Trybot-Ready: Steven Bennetts <stevenjb@chromium.org>
Tested-by: Steven Bennetts <stevenjb@chromium.org>

[modify] https://crrev.com/4a714c91b4a410e1e195ea2b9e5d0b78e94209e2/cbuildbot/config_dump.json
[modify] https://crrev.com/4a714c91b4a410e1e195ea2b9e5d0b78e94209e2/cbuildbot/chromeos_config.py

The builder is still failing:
https://build.chromium.org/p/chromiumos.chromium/builders/amd64-generic-tot-asan-informational/builds/12482


I've confirmed that it is passing --disk_layout=4gb-rootfs to build_image:

08:41:03: INFO: RunCommand: /b/cbuild/shared_external/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmplLuaJr' 'FEATURES=separatedebug' 'CHROME_ORIGIN=LOCAL_SOURCE' -- ./build_image '--board=amd64-generic' --replace '--version=' '--builder_path=amd64-generic-tot-asan-informational/R59-9390.0.0-b12482' '--disk_layout=4gb-rootfs' test in /b/cbuild/shared_external
...
INFO    : Using image type 4gb-rootfs


However further down I see '/dev/loop0 2.0G':

Setting up symlinks for /usr/local for /mnt/host/source/src/build/images/amd64-generic/R59-9390.0.2017_03_22_0841-a1/stateful/dev_image
INFO    : Image specified by /mnt/host/source/src/build/images/amd64-generic/R59-9390.0.2017_03_22_0841-a1 mounted at /mnt/host/source/src/build/images/amd64-generic/R59-9390.0.2017_03_22_0841-a1/rootfs successfully.
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      2.0G  3.0M  2.0G   1% /mnt/host/source/src/build/images/amd64-generic/R59-9390.0.2017_03_22_0841-a1/rootfs
Setting up symlinks for /usr/local for /mnt/host/source/src/build/images/amd64-generic/R59-9390.0.2017_03_22_0841-a1/stateful/dev_image


And then:

ERROR   : Target image has run out of space:
ERROR   : Filesystem     1M-blocks  Used Available Use% Mounted on
ERROR   : /dev/loop0          1969  1969         0 100% /mnt/host/source/src/build/images/amd64-generic/R59-9390.0.2017_03_22_0841-a1/rootfs

What happens if you build an image locally with the 4g rootfs option? Does it have 4G partitions with a 2G rootfs?

+vaper@, +semenzato@, who most recently touched  build_library/legacy_disk_layout.json

This is the snippet from  https://chromium.git.corp.google.com/chromiumos/platform/crosutils/+/0870b3c27b2d6c7a95c5343a4861f3f66dcf3dfe/build_library/legacy_disk_layout.json:

    # Very large rootfs, suitable for development with symbols,
    # etc. Cannot apply updates when running from USB (no slot B)
    "4gb-rootfs": [
      {
        "num": 5,
        "size": "2 MiB"
      },
      {
        # This partition is larger than the base partition, so the
        # installer will corrupt the disk during installation.
        "num": 3,
        "size": "4096 MiB",
        "fs_size": "4000 MiB"
      }
    ]

I don't understand the second comment, but it sounds.... ominous.

Also, when I build the image locally and boot from it, /dev/root is 3.9G.

I'm trying an asan build now, but I am wondering if the way we are mounting/installing the image on the builder is the problem? (This isn't an area I really now anything about).

The builder runs this command:

08:41:03: INFO: RunCommand: /b/cbuild/shared_external/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmplLuaJr' 'FEATURES=separatedebug' 'CHROME_ORIGIN=LOCAL_SOURCE' -- ./build_image '--board=amd64-generic' --replace '--version=' '--builder_path=amd64-generic-tot-asan-informational/R59-9390.0.0-b12482' '--disk_layout=4gb-rootfs' test in /b/cbuild/shared_external


Every other log file message until the error came from build_image. So.... it sounds like a build_image bug.

The build_image command line contains a lot of noise, but what's standing out to me is that it's a standard image, not dev/test/release.

Maybe the problem is related to that?

'test' appears in RunCommand just after '--disk_layout=4gb-rootfs'

I have no idea.

I ran the same build config as a tryjob, and it's green.

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/chromium_pfq_informational/builds/18

dgarrett@ RE comment #43 - I don't see 'asan' in the SetupBoard stage for that try job.

I was finally able to reproduce the failure locally.

It requires both a non-internal chromiumos checkout and a non-internal chromium checkout for LOCAL_SOURCE.

It fails both with and without --disk_layout=4gb-rootfs

I'm going to see if we can maybe address this by reducing the amount of stuff we copy (e.g. maybe fewer .zvoice files for speech synthesis in different languages) since this will only ever be used for testing.

If that fails I will try experimenting with legacy_disk_layout.json...

Potential workaround reducing the languages supported for ASAN builds to just en-US: https://chromium-review.googlesource.com/c/461298/

The workaround failed in VM Tests and may have other problems so I am abandoning that approach.

Someone is going to need to investigate build_image and figure out why the 4gb layout doesn't work correctly.

Marking as Available because I don't expect to be able to work on this any time soon.

It's almost certain that we have hard coded 2G in multiple locations, which should be fixed to use the 'new' layout information.

Passing to a current sheriff to find an owner.

[shadow gardener theory]: --disk_layout=4gb-rootfs is a no-op (see crbug.com/238304) and what really matters is DISK_LAYOUT_PATH mentioned in disk_layout_util.sh

plausible or not?  Going to run trybot jobs to determine this.

Reading through all the comments, I am unclear on the current status of this? Is someone actively investigating/fixing anything here?

I was investigating as shadow gardener, but didn't assign to myself since I was mostly doing guess and check.  Reproducing the failure locally as described in comment #45 sounded difficult, so I launched trybots with this command from chromiumos checkout (but outside the chroot)

cbuildbot -g 'If17746776c89acc3ec3e6108b1bf223aa87cd044 I3c70da9d844742b647d50eb33fec05da9d4ef3fa' --remote amd64-generic-tot-asan-informational

where the relevant changes are:

https://chromium-review.googlesource.com/c/471788/
https://chromium-review.googlesource.com/c/468370/

From my results:

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/tot_asan_informational

It looks like overlay-amd64-generic/scripts/disk_layout.json is what affects the rootfs size, and we also need to make the partition size greater than the rootfs size too.  My last build:

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/tot_asan_informational/builds/14/steps/BuildPackages/logs/stdio

encountered  crbug.com/710006 .  I hope to launch one more trybot before my shift ends ... 

What's the current status of this?  Is there any estimate as to when it might be fixed?

alemate is the gardener this week. And I am sure not next week. Punt, punt, punt.

This is potentially causing us to miss detectable memory corruption issues leading to an increase in crash rates and general pain and suffering. I am escalating this to a P0.

There is no obvious owner here.

dgarret@, I know you are as stretched thin as the rest of us, but your previous attempt to delegate apepars to have failed :) I'm putting this back on your plate because this is clearly complex enough that a gardener is not going to be able to solve this unless they have a quiet week, which doesn't seem likely any time soon.

gurchetansingh@ you seemed to be making progress here, do you have cycles to continue your investigation?

+sheriffs

I think a sheriff should tackle this.

akeshet@ - While I don't necessarily disagree, we've been bouncing this around from gardener to sheriff from shift to shift and it hasn't gotten fixed in nearly 2 months. It involves builder configuration scripts that few people seem to understand, and, most importantly I think, some dedicated time which has not been available to sheriffs or gardeners in a while. It's one of those things that falls into the cracks between development and infrastructure. If a sheriff can take it on, that would be great, but someone needs to stay on this until it is fixed, please.

@55, I can launch more trybots, sure.

FWIW, gurchetansingh's last tryjob was green: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/tot_asan_informational/builds/22/steps/UnitTest/logs/stdio

Here's the line that's confirming it built a 4GB rootfs:

/dev/loop0      3.9G  7.9M  3.9G   1% /mnt/host/source/src/build/images/amd64-generic/R60-9483.0.2017_04_21_1844-a1/rootfs

I think https://chromium-review.googlesource.com/c/471788/ is sufficient to fix the issue (I did a local build with similar changes and verified that the df line quoted above says 3.9G as well).

As noted previously, changing the disk image parameters will break auto-updates from earlier images that used different parameters. I don't think anyone should rely on amd64-generic to be able to AU successfully though (except maybe for AU testing)?

Can we make a change that would only apply to the asan builder?

I wouldn't expect us to run an amd64-generic image on a device that auto updates, but if we can avoid that without too much trouble that might be nice.

We will need 4GB rootfs for real boards soonish...

What new boards are coming online that will have 4GB rootfs?

If I would have wanted to mention them on a public issue I would have done it! ;^)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/da44a0cd3d9603e6f8f7920258745d40502e9717

commit da44a0cd3d9603e6f8f7920258745d40502e9717
Author: Gurchetan Singh <gurchetansingh@google.com>
Date: Tue Apr 25 21:03:52 2017

cbuildbot: pass in disk layout argument to image_to_vm.sh

It's needed for the amd64-generic-tot-asan-informational builder.

BUG= chromium:697967 
TEST=Not sure how to test this buildbot change, but with CL:468370
applied the following buildbot succeeded in the BuildImage stage:

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/tot_asan_informational/builds/28

Change-Id: I46b7f2ccc8a1e98ade2c3dd16e37128265c43a23
Reviewed-on: https://chromium-review.googlesource.com/486248
Commit-Ready: Gurchetan Singh <gurchetansingh@chromium.org>
Tested-by: Gurchetan Singh <gurchetansingh@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/da44a0cd3d9603e6f8f7920258745d40502e9717/cbuildbot/stages/build_stages.py
[modify] https://crrev.com/da44a0cd3d9603e6f8f7920258745d40502e9717/cbuildbot/commands.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/34fb2dc3c09e23bc24ca486fc975b515fea04f9c

commit 34fb2dc3c09e23bc24ca486fc975b515fea04f9c
Author: Gurchetan Singh <gurchetansingh@google.com>
Date: Fri Apr 28 08:16:03 2017

overlay-amd64-generic: Have a case for 4gb rootfs

It's needed for the amd64-generic-tot-asan-informational builder.
The ROOT-A partition (num: 3) and ROOT-B partition sizes (num: 5)
are taken from legacy_disk_layout.json. The stateful partition (num: 1)
size was doubled since many VM tests were failing with the ENOSPC errno.

BUG= chromium:697967 
TEST=Latest buildbot passes VM test stage with this change:

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/tot_asan_informational/builds/36

The unit test stage is still failing, but the cause looks like legitimate
ASAN failures.

Change-Id: I56e0ba7bf04220fca950027027f80ea349e2658f
Reviewed-on: https://chromium-review.googlesource.com/484814
Reviewed-by: Mattias Nissler <mnissler@chromium.org>
Tested-by: Mattias Nissler <mnissler@chromium.org>

[modify] https://crrev.com/34fb2dc3c09e23bc24ca486fc975b515fea04f9c/overlay-amd64-generic/scripts/disk_layout.json

Note: CL in comment 67 is a stop-gap to fix amd64-generic Chrome PFQ redness. I'll leave it to you to drive a decision on how to properly handle disk layout merging and ensuring we get sufficiently large stateful and root file systems both with and without 4gb-rootfs.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/140a1d7ee38de433192f49a55b20ef82ae5fea8d

commit 140a1d7ee38de433192f49a55b20ef82ae5fea8d
Author: Mattias Nissler <mnissler@chromium.org>
Date: Fri Apr 28 15:17:09 2017

Drop 2gb-rootfs disk layout override for amd64-generic-chromium-pfq

After https://chromium-review.googlesource.com/c/486248/,
image_to_vm.sh gets the disk layout to use from the bot config. It
turns out that amd64-generic-chromium-pfq had an incompatible
override. 2gb-rootfs doesn't have a large enough stable partition to
run VMTests. Drop the override so it picks up the default
2gb-rootfs-updatable, which has sufficient stateful spcae.

BUG= chromium:697967 
TEST=No more 'No space left on device' errors in VMTest step on amd64-generic-chromium-pfq

Change-Id: I940e62073009148942c941485c9d58b2946af170
Reviewed-on: https://chromium-review.googlesource.com/490288
Tested-by: Mattias Nissler <mnissler@chromium.org>
Trybot-Ready: Mattias Nissler <mnissler@chromium.org>
Reviewed-by: Mattias Nissler <mnissler@chromium.org>

[modify] https://crrev.com/140a1d7ee38de433192f49a55b20ef82ae5fea8d/cbuildbot/config_dump.json
[modify] https://crrev.com/140a1d7ee38de433192f49a55b20ef82ae5fea8d/cbuildbot/chromeos_config.py

Okay, build_image is now succeeding:

https://build.chromium.org/p/chromiumos/builders/amd64-generic-asan

We still have some unit test failures (some ASAN specific):

https://build.chromium.org/p/chromiumos/builders/amd64-generic-asan/builds/19590/steps/UnitTest/logs/stdio

https://build.chromium.org/p/chromiumos/builders/amd64-generic-asan/builds/19590/steps/VMTest%20%28attempt%202%29/logs/stdio

However, none of this issues look related to the image size, so assigning to  stevenjb@ who can better triage the remaining issues. 

For the disk layout merging issues we found, I filed  crbug.com/716607 .

That's fantastic, thanks a ton gurchetansingh@!

I'm actually going to close this and have the gardener open a fresh bug for the current failures. At least now we can identify and hopefully fix them!