Issue 676151  has been merged into this issue.

OK, so build #942 complains with these messages:

INFO    : Generating license credits page. Time:
licenses: Unhandled exception:
Traceback (most recent call last):
  File "/mnt/host/source/chromite/licensing/licenses", line 168, in <module>
    DoMain()
  File "/mnt/host/source/chromite/licensing/licenses", line 164, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/mnt/host/source/chromite/lib/commandline.py", line 834, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/mnt/host/source/chromite/licensing/licenses.py", line 223, in main
    licensing.GenerateHTMLLicenseOutput(opts.output)
  File "/mnt/host/source/chromite/licensing/licenses_lib.py", line 1056, in GenerateHTMLLicenseOutput
    self.EvaluateTemplate(file_template, env).encode('UTF-8'))
  File "/mnt/host/source/chromite/lib/osutils.py", line 130, in WriteFile
    f.writelines(cros_build_lib.iflatten_instance(content))
IOError: [Errno 28] No space left on device
ERROR   : Here are the biggest [partially-]extracted files (by disk usage):
1503232 usr/bin/core_collector32
1581056 opt/google/chrome/resources/chromeos/chromevox/braille/liblouis_nacl_x86_64.nexe
1613824 usr/lib64/locale/locale-archive
1654784 opt/google/chrome/resources/inspector/inspector.js
1683456 usr/sbin/flashrom
1798144 lib64/libc-2.19.so
1818624 opt/google/containers/android/vendor/lib/arm/libstagefright.so
1818624 opt/google/containers/android/vendor/lib/arm/nb/libstagefright.so
1843200 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_sz_nexe
1925120 usr/share/chromeos-assets/easy_unlock/easy_unlock_background.js
1945600 usr/share/chromeos-assets/easy_unlock/easy_unlock_pairing.js
1949696 opt/google/containers/android/vendor/lib/arm/libicui18n.so
1949696 opt/google/containers/android/vendor/lib/arm/nb/libicui18n.so
2121728 usr/lib64/va/drivers/i965_drv_video.so
2170880 opt/google/chrome/pnacl/pnacl_public_x86_64_ld_nexe
2215936 usr/lib64/libcrypto.so.1.0.0
2228224 opt/google/containers/android/vendor/lib/mediadrm/libwvdrmengine.so
2273280 usr/share/fonts/tibt-jomolhari/Jomolhari-alpha3c-0605331.ttf
2334720 usr/share/chromeos-assets/input_methods/input_tools/en_gesture/gesture_lexicon.bin
2445312 usr/lib64/Dptf.so
2568192 opt/google/containers/android/vendor/bin/houdini
2650112 lib/udev/hwdb.d/20-pci-vendor-model.hwdb
2670592 usr/lib64/libpoppler.so.52.0.0
2674688 usr/share/chromeos-assets/input_methods/input_tools/wubi/system_dictionary.bin
2818048 opt/google/chrome/chrome_100_percent.pak
3309568 opt/google/chrome/nacl_helper_nonsfi
3608576 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/system_dictionary.bin
3678208 opt/google/chrome/nacl_irt_x86_64.nexe
3813376 usr/share/fonts/ko-nanum/NanumMyeongjo.ttf
3985408 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/bigram.bin
4059136 opt/google/chrome/libosmesa.so
4075520 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_id-ID.zvoice
4096000 usr/share/chromeos-assets/quickoffice/scripts/mainQOWT.js
4190208 opt/google/containers/android/vendor/lib/arm/libpdfium.so
4210688 usr/share/chromeos-assets/input_methods/input_tools/en/system_dictionary.bin
4304896 usr/share/fonts/ko-nanum/NanumGothicBold.ttf
4317184 opt/google/containers/android/vendor/lib/arm/libskia.so
4317184 opt/google/containers/android/vendor/lib/arm/nb/libskia.so
4358144 usr/share/fonts/ko-nanum/NanumGothic.ttf
4640768 usr/share/chromeos-assets/input_methods/input_tools/pinyin/system_dictionary.bin
4653056 usr/share/fonts/ko-nanum/NanumMyeongjoBold.ttf
4657152 opt/google/containers/android/vendor/lib/libhoudini.so
4722688 usr/bin/shill
4763648 usr/share/chromeos-assets/input_methods/input_tools/ko/system_hanja_dictionary.bin
4993024 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_pt-BR.zvoice
5128192 usr/share/chromeos-assets/input_methods/input_tools/nacl_session_handler_x86_64.nexe
5451776 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/word_x86_64.nexe
5664768 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-IN.zvoice
5795840 boot/vmlinuz-4.4.21-06342-g6e72416
5795840 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-GB.zvoice
6000640 usr/share/fonts/noto/NotoColorEmoji.ttf
6189056 opt/google/chrome/libwidevinecdm.so
6279168 opt/google/containers/android/vendor/lib/dri/kms_swrast_dri.so
6287360 opt/google/containers/android/vendor/lib/arm/libart.so
6410240 opt/google/chrome/nacl_helper
6557696 opt/google/containers/android/vendor/lib/dri/i965_dri.so
6684672 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-US.zvoice
6688768 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-ES.zvoice
6709248 usr/lib64/dri/i965_dri.so
6860800 etc/udev/hwdb.bin
6938624 usr/share/chromeos-assets/input_methods/input_tools/pinyin/bigram.bin
7106560 opt/displaylink/DisplayLinkManager
7483392 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_fr-FR.zvoice
7512064 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_nl-NL.zvoice
7618560 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-US.zvoice
7634944 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_ko-KR.zvoice
7757824 usr/share/fonts/ja-motoyafonts/MTG4g5H.ttc
7815168 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_de-DE.zvoice
8089600 opt/google/chrome/chrome_200_percent.pak
8409088 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_it-IT.zvoice
8491008 usr/share/fonts/ja-motoyafonts/MTG4g2bH.ttc
9129984 usr/share/fonts/ascender_licensed/MSungB5HK-Regular.ttf
9195520 usr/share/chromeos-assets/input_methods/input_tools/zipped_data_chromeos
9252864 usr/sbin/chromeos-firmwareupdate
9670656 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_hi-IN.zvoice
9957376 opt/google/containers/android/vendor/pittpatt/models/recognition/face.face.y0-y0-71-N-tree_7-wmd.bin
9981952 usr/share/chromeos-assets/input_methods/input_tools/decoder_x86_64.nexe
9981952 usr/share/fonts/ja-motoyafonts/MTG4m5H.ttc
10149888 opt/google/chrome/icudtl.dat
10272768 usr/share/chromeos-assets/demo_app/screensaver-video.webm
10371072 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/sheet_x86_64.nexe
10850304 usr/share/fonts/ja-motoyafonts/MTG4m2bH.ttc
11481088 usr/share/fonts/ascender_licensed/MSungB5HK-Bold.ttf
11747328 usr/share/fonts/ascender_licensed/MSungGB18030-Regular.ttf
12197888 usr/lib64/libgs.so.9.19
13234176 usr/share/chromeos-assets/input_methods/input_tools/en/bigram.bin
14114816 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_llc_nexe
15663104 opt/google/containers/android/vendor/lib/arm/libLLVM.so
16338944 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/point_x86_64.nexe
17084416 usr/share/fonts/ascender_licensed/MSungGB18030-Bold.ttf
18702336 usr/share/fonts/notocjk/NotoSansCJK-Regular.ttc
19275776 usr/share/fonts/notocjk/NotoSansCJK-Bold.ttc
23240704 usr/lib64/dri/i915_dri.so
23240704 usr/lib64/dri/kms_swrast_dri.so
23240704 usr/lib64/dri/swrast_dri.so
26513408 usr/share/chromeos-assets/speech_synthesis/patts/tts_service_x86-64.nexe
28975104 opt/google/chrome/resources.pak
31993856 opt/google/chrome/pepper/libpepflashplayer.so
154906624 opt/google/chrome/chrome
467406848 opt/google/containers/android/system.raw.img

===

I downloaded the "image.zip" of build #940 (the last successful one), then I did:

  unzip reef-paladin%2FR57-9105.0.0-rc1%2Fimage.zip
  ./unpack_partitions.sh chromiumos_base_image.bin 3
  mkdir foo
  sudo mount -o loop,ro part_3 foo
  cd foo
  df -h
  sudo find "." -xdev -type f -printf '%b %P\n' |awk '$1 > 16 { $1 = $1 * 512; print }' | sort -n | tail -100 1>&2

Where that last command came by grepping and finding the builder error message in "src/scripts/build_library/base_image_util.sh"

===

For the "df -h" I got:

1.7G  1.7G   76M  96%

...aka 76M was free on the last passing build.

===

For the find, I got:

1302528 usr/lib64/libgio-2.0.so.0.3400.3
1343488 usr/sbin/mosys
1384448 opt/google/chrome/resources/about_os_credits.html
1388544 lib/firmware/iwlwifi-7265D-14.ucode
1392640 lib/firmware/iwlwifi-7265D-16.ucode
1392640 lib/firmware/iwlwifi-7265D-21.ucode
1396736 opt/google/chrome/keyboard_resources.pak
1417216 usr/share/cups/ipptool/document-letter.pdf
1429504 usr/share/chromeos-assets/genius_app/images/200/discover/edit_photo.webp
1449984 usr/lib64/libbase-core-395517.so
1458176 lib/udev/hwdb.d/20-OUI.hwdb
1474560 usr/bin/core_collector
1474560 usr/share/cups/ipptool/document-letter.ps
1482752 usr/bin/core_collector32
1482752 usr/sbin/rmi4update
1499136 usr/lib64/libxml2.so.2.9.4
1581056 opt/google/chrome/resources/chromeos/chromevox/braille/liblouis_nacl_x86_64.nexe
1613824 usr/lib64/locale/locale-archive
1654784 opt/google/chrome/resources/inspector/inspector.js
1683456 usr/sbin/flashrom
1740800 lib64/libc-2.23.so
1843200 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_sz_nexe
1925120 usr/share/chromeos-assets/easy_unlock/easy_unlock_background.js
1945600 usr/share/chromeos-assets/easy_unlock/easy_unlock_pairing.js
2121728 usr/lib64/va/drivers/i965_drv_video.so
2170880 opt/google/chrome/pnacl/pnacl_public_x86_64_ld_nexe
2215936 usr/lib64/libcrypto.so.1.0.0
2273280 usr/share/fonts/tibt-jomolhari/Jomolhari-alpha3c-0605331.ttf
2334720 usr/share/chromeos-assets/input_methods/input_tools/en_gesture/gesture_lexicon.bin
2445312 usr/lib64/Dptf.so
2650112 lib/udev/hwdb.d/20-pci-vendor-model.hwdb
2670592 usr/lib64/libpoppler.so.52.0.0
2674688 usr/share/chromeos-assets/input_methods/input_tools/wubi/system_dictionary.bin
2818048 opt/google/chrome/chrome_100_percent.pak
3309568 opt/google/chrome/nacl_helper_nonsfi
3608576 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/system_dictionary.bin
3678208 opt/google/chrome/nacl_irt_x86_64.nexe
3813376 usr/share/fonts/ko-nanum/NanumMyeongjo.ttf
3985408 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/bigram.bin
4059136 opt/google/chrome/libosmesa.so
4075520 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_id-ID.zvoice
4096000 usr/share/chromeos-assets/quickoffice/scripts/mainQOWT.js
4210688 usr/share/chromeos-assets/input_methods/input_tools/en/system_dictionary.bin
4304896 usr/share/fonts/ko-nanum/NanumGothicBold.ttf
4358144 usr/share/fonts/ko-nanum/NanumGothic.ttf
4640768 usr/share/chromeos-assets/input_methods/input_tools/pinyin/system_dictionary.bin
4653056 usr/share/fonts/ko-nanum/NanumMyeongjoBold.ttf
4722688 usr/bin/shill
4763648 usr/share/chromeos-assets/input_methods/input_tools/ko/system_hanja_dictionary.bin
4993024 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_pt-BR.zvoice
5128192 usr/share/chromeos-assets/input_methods/input_tools/nacl_session_handler_x86_64.nexe
5451776 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/word_x86_64.nexe
5664768 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-IN.zvoice
5795840 boot/vmlinuz-4.4.21-06340-g3b81685
5795840 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-GB.zvoice
6000640 usr/share/fonts/noto/NotoColorEmoji.ttf
6189056 opt/google/chrome/libwidevinecdm.so
6410240 opt/google/chrome/nacl_helper
6684672 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-US.zvoice
6688768 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-ES.zvoice
6709248 usr/lib64/dri/i965_dri.so
6938624 usr/share/chromeos-assets/input_methods/input_tools/pinyin/bigram.bin
7106560 opt/displaylink/DisplayLinkManager
7483392 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_fr-FR.zvoice
7512064 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_nl-NL.zvoice
7618560 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-US.zvoice
7634944 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_ko-KR.zvoice
7757824 usr/share/fonts/ja-motoyafonts/MTG4g5H.ttc
7815168 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_de-DE.zvoice
8089600 opt/google/chrome/chrome_200_percent.pak
8409088 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_it-IT.zvoice
8491008 usr/share/fonts/ja-motoyafonts/MTG4g2bH.ttc
9129984 usr/share/fonts/ascender_licensed/MSungB5HK-Regular.ttf
9195520 usr/share/chromeos-assets/input_methods/input_tools/zipped_data_chromeos
9252864 usr/sbin/chromeos-firmwareupdate
9670656 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_hi-IN.zvoice
9981952 usr/share/chromeos-assets/input_methods/input_tools/decoder_x86_64.nexe
9981952 usr/share/fonts/ja-motoyafonts/MTG4m5H.ttc
10149888 opt/google/chrome/icudtl.dat
10272768 usr/share/chromeos-assets/demo_app/screensaver-video.webm
10371072 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/sheet_x86_64.nexe
10850304 usr/share/fonts/ja-motoyafonts/MTG4m2bH.ttc
11481088 usr/share/fonts/ascender_licensed/MSungB5HK-Bold.ttf
11747328 usr/share/fonts/ascender_licensed/MSungGB18030-Regular.ttf
12197888 usr/lib64/libgs.so.9.19
13234176 usr/share/chromeos-assets/input_methods/input_tools/en/bigram.bin
14114816 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_llc_nexe
16338944 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/point_x86_64.nexe
17084416 usr/share/fonts/ascender_licensed/MSungGB18030-Bold.ttf
18702336 usr/share/fonts/notocjk/NotoSansCJK-Regular.ttc
19275776 usr/share/fonts/notocjk/NotoSansCJK-Bold.ttc
23240704 usr/lib64/dri/i915_dri.so
23240704 usr/lib64/dri/kms_swrast_dri.so
23240704 usr/lib64/dri/swrast_dri.so
26513408 usr/share/chromeos-assets/speech_synthesis/patts/tts_service_x86-64.nexe
28975104 opt/google/chrome/resources.pak
31993856 opt/google/chrome/pepper/libpepflashplayer.so
46718976 opt/google/containers/android/vendor.raw.img
154906624 opt/google/chrome/chrome
464564224 opt/google/containers/android/system.raw.img

===

Diffing those two shows that the passing image has:

46718976 opt/google/containers/android/vendor.raw.img

...but the failing image doesn't.  Instead, it has:

15663104 opt/google/containers/android/vendor/lib/arm/libLLVM.so
9957376 opt/google/containers/android/vendor/pittpatt/models/recognition/face.face.y0-y0-71-N-tree_7-wmd.bin
6557696 opt/google/containers/android/vendor/lib/dri/i965_dri.so
6287360 opt/google/containers/android/vendor/lib/arm/libart.so
6279168 opt/google/containers/android/vendor/lib/dri/kms_swrast_dri.so
4657152 opt/google/containers/android/vendor/lib/libhoudini.so
4317184 opt/google/containers/android/vendor/lib/arm/nb/libskia.so
4317184 opt/google/containers/android/vendor/lib/arm/libskia.so
4190208 opt/google/containers/android/vendor/lib/arm/libpdfium.so
2568192 opt/google/containers/android/vendor/bin/houdini
2228224 opt/google/containers/android/vendor/lib/mediadrm/libwvdrmengine.so
1949696 opt/google/containers/android/vendor/lib/arm/nb/libicui18n.so
1949696 opt/google/containers/android/vendor/lib/arm/libicui18n.so
1818624 opt/google/containers/android/vendor/lib/arm/nb/libstagefright.so
1818624 opt/google/containers/android/vendor/lib/arm/libstagefright.so
...

===

Postulating a guess I'd say that "src/private-overlays/project-cheets-private/scripts/board_specific_setup.sh"'s function pack_android_vendor_image() (in conjunction with the script deploy_vendor_image.sh) is to blame.  Specifically I believe that together the copy the _uncompressed_ files to the final image.  They compress to a temp location, then delete the uncompressed file, then move the compressed file to the final image.  ...but there's a period of time where the uncompressed files need to fit into the final image.  I think that's where we're getting into trouble.

Basically: even though there are 76M free at the end of the step, we are still over the limit at some time in the build. 

===

Why weren't we hitting this before?  I think we were just squeaking under the limit but that this file pushed us over:

6860800 etc/udev/hwdb.bin

That file also shows up in the list of big files and is new for the failing build.  Presumably this is a side effect of <https://chromium-review.googlesource.com/#/c/409017/>

Short term, we can presumably figure out how to avoid adding 6+ MB to the build from a udev upgrade.  Long term we need to fix the android build.

===

All of the above is subject to review and I could certainly have made mistakes since I'm just a silly sheriff.  Please correct any stupidity.

At some point I did try to have that temporary step be performed in a tmpfs, but it was abandoned due to some other thing freeing up space. I'll try to see if it's easy enough to unabandon for the long-term fix.

The failure in the original report happened on this buildbot:

BUILDBOT_SLAVENAME: build281-m2

We're seeing out-of-space failures on other builders too.  See issue 677279.

I think that the last line in comment #4 may refer to a different out-of-space situation.

Reef-paladin is failing again.

https://uberchromegw.corp.google.com/i/chromeos/builders/reef-paladin/builds/1101/steps/BuildImage/logs/stdio

and the previous two builds.

I don't remember if we have a separate bug for this, but it would be highly helpful to print a list of top large files in the SUCCESSFUL builds as well.  So we can compare and see what's pushing us over the edge.

Hi Luis,

we're hitting this in the CQ and this time we're not sure there's any quick workaround, although we're looking for one.

How difficult is it to fix your way?

Thanks

Following the same procedure as in comment 2:

Here's the list of big users in the failed build (1101)

Packages failed:
	chromeos-base/chromeos-init-0.0.25-r2630
	chromeos-base/chromeos-chrome-57.0.2977.0_rc-r1
ERROR   : Here are the biggest [partially-]extracted files (by disk usage):
1482752 usr/bin/core_collector32
1499136 usr/lib64/libxml2.so.2.9.4
1581056 opt/google/chrome/resources/chromeos/chromevox/braille/liblouis_nacl_x86_64.nexe
1671168 opt/google/chrome/resources/inspector/inspector.js
1675264 usr/lib64/locale/locale-archive
1683456 usr/sbin/flashrom
1740800 lib64/libc-2.23.so
1818624 opt/google/containers/android/vendor/lib/arm/libstagefright.so
1818624 opt/google/containers/android/vendor/lib/arm/nb/libstagefright.so
1843200 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_sz_nexe
1925120 usr/share/chromeos-assets/easy_unlock/easy_unlock_background.js
1945600 usr/share/chromeos-assets/easy_unlock/easy_unlock_pairing.js
1949696 opt/google/containers/android/vendor/lib/arm/libicui18n.so
1949696 opt/google/containers/android/vendor/lib/arm/nb/libicui18n.so
2121728 usr/lib64/va/drivers/i965_drv_video.so
2170880 opt/google/chrome/pnacl/pnacl_public_x86_64_ld_nexe
2215936 usr/lib64/libcrypto.so.1.0.0
2228224 opt/google/containers/android/vendor/lib/mediadrm/libwvdrmengine.so
2273280 usr/share/fonts/tibt-jomolhari/Jomolhari-alpha3c-0605331.ttf
2334720 usr/share/chromeos-assets/input_methods/input_tools/en_gesture/gesture_lexicon.bin
2445312 usr/lib64/Dptf.so
2568192 opt/google/containers/android/vendor/bin/houdini
2650112 lib/udev/hwdb.d/20-pci-vendor-model.hwdb
2670592 usr/lib64/libpoppler.so.52.0.0
2674688 usr/share/chromeos-assets/input_methods/input_tools/wubi/system_dictionary.bin
2818048 opt/google/chrome/chrome_100_percent.pak
3489792 opt/google/chrome/nacl_helper_nonsfi
3608576 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/system_dictionary.bin
3678208 opt/google/chrome/nacl_irt_x86_64.nexe
3813376 usr/share/fonts/ko-nanum/NanumMyeongjo.ttf
3985408 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/bigram.bin
4059136 opt/google/chrome/libosmesa.so
4075520 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_id-ID.zvoice
4096000 usr/share/chromeos-assets/quickoffice/scripts/mainQOWT.js
4190208 opt/google/containers/android/vendor/lib/arm/libpdfium.so
4210688 usr/share/chromeos-assets/input_methods/input_tools/en/system_dictionary.bin
4304896 usr/share/fonts/ko-nanum/NanumGothicBold.ttf
4317184 opt/google/containers/android/vendor/lib/arm/libskia.so
4317184 opt/google/containers/android/vendor/lib/arm/nb/libskia.so
4358144 usr/share/fonts/ko-nanum/NanumGothic.ttf
4640768 usr/share/chromeos-assets/input_methods/input_tools/pinyin/system_dictionary.bin
4653056 usr/share/fonts/ko-nanum/NanumMyeongjoBold.ttf
4657152 opt/google/containers/android/vendor/lib/libhoudini.so
4722688 usr/bin/shill
4763648 usr/share/chromeos-assets/input_methods/input_tools/ko/system_hanja_dictionary.bin
4993024 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_pt-BR.zvoice
5128192 usr/share/chromeos-assets/input_methods/input_tools/nacl_session_handler_x86_64.nexe
5451776 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/word_x86_64.nexe
5664768 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-IN.zvoice
5795840 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-GB.zvoice
5828608 boot/vmlinuz-4.4.35-06593-g28c3ec0
6000640 usr/share/fonts/noto/NotoColorEmoji.ttf
6189056 opt/google/chrome/libwidevinecdm.so
6283264 opt/google/containers/android/vendor/lib/dri/kms_swrast_dri.so
6287360 opt/google/containers/android/vendor/lib/arm/libart.so
6561792 opt/google/containers/android/vendor/lib/dri/i965_dri.so
6684672 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-US.zvoice
6688768 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-ES.zvoice
6729728 usr/lib64/dri/i965_dri.so
6836224 opt/google/chrome/nacl_helper
6938624 usr/share/chromeos-assets/input_methods/input_tools/pinyin/bigram.bin
7106560 opt/displaylink/DisplayLinkManager
7483392 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_fr-FR.zvoice
7512064 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_nl-NL.zvoice
7618560 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-US.zvoice
7634944 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_ko-KR.zvoice
7757824 usr/share/fonts/ja-motoyafonts/MTG4g5H.ttc
7815168 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_de-DE.zvoice
8093696 opt/google/chrome/chrome_200_percent.pak
8409088 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_it-IT.zvoice
8491008 usr/share/fonts/ja-motoyafonts/MTG4g2bH.ttc
9129984 usr/share/fonts/ascender_licensed/MSungB5HK-Regular.ttf
9195520 usr/share/chromeos-assets/input_methods/input_tools/zipped_data_chromeos
9252864 usr/sbin/chromeos-firmwareupdate
9670656 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_hi-IN.zvoice
9957376 opt/google/containers/android/vendor/pittpatt/models/recognition/face.face.y0-y0-71-N-tree_7-wmd.bin
9981952 usr/share/chromeos-assets/input_methods/input_tools/decoder_x86_64.nexe
9981952 usr/share/fonts/ja-motoyafonts/MTG4m5H.ttc
10149888 opt/google/chrome/icudtl.dat
10272768 usr/share/chromeos-assets/demo_app/screensaver-video.webm
10371072 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/sheet_x86_64.nexe
10850304 usr/share/fonts/ja-motoyafonts/MTG4m2bH.ttc
11481088 usr/share/fonts/ascender_licensed/MSungB5HK-Bold.ttf
11747328 usr/share/fonts/ascender_licensed/MSungGB18030-Regular.ttf
12197888 usr/lib64/libgs.so.9.19
13234176 usr/share/chromeos-assets/input_methods/input_tools/en/bigram.bin
14114816 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_llc_nexe
15663104 opt/google/containers/android/vendor/lib/arm/libLLVM.so
16338944 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/point_x86_64.nexe
17084416 usr/share/fonts/ascender_licensed/MSungGB18030-Bold.ttf
18702336 usr/share/fonts/notocjk/NotoSansCJK-Regular.ttc
19275776 usr/share/fonts/notocjk/NotoSansCJK-Bold.ttc
23265280 usr/lib64/dri/i915_dri.so
23265280 usr/lib64/dri/kms_swrast_dri.so
23265280 usr/lib64/dri/swrast_dri.so
26513408 usr/share/chromeos-assets/speech_synthesis/patts/tts_service_x86-64.nexe
28536832 opt/google/chrome/resources.pak
31993856 opt/google/chrome/pepper/libpepflashplayer.so
154849280 opt/google/chrome/chrome
473124864 opt/google/containers/android/system.raw.img
ERROR   : Target image has run out of space:
ERROR   : Filesystem     1M-blocks  Used Available Use% Mounted on
ERROR   : /dev/loop0          1723  1723         0 100% /mnt/host/source/src/build/images/reef/R57-9169.0.0-rc2/rootfs

I downloaded the image from the last successful build

into file

reef-paladin%2FR57-9168.0.0-rc2%2Fimage.zip

then used src/scripts/mount_gpt_image.sh, and df -h showed these:

/dev/loop0                                           1.7G  1.7G   75M  96% /tmp/m
/dev/loop1                                           1.2G   84M  1.1G   8% /tmp/s
/dev/loop2                                            12M   28K   12M   1% /tmp/m/usr/share/oem

Then I ran

sudo find "." -xdev -type f -printf '%b %P\n' |awk '$1 > 16 { $1 = $1 * 512; print }' | sort -n | tail -100 1>&2

1232896 opt/google/chrome/resources/inspector/unit_test_runner.js
1269760 opt/google/chrome/snapshot_blob.bin
1294336 usr/lib64/libtrunks.so
1302528 usr/lib64/libgio-2.0.so.0.3400.3
1343488 usr/sbin/mosys
1388544 lib/firmware/iwlwifi-7265D-14.ucode
1392640 lib/firmware/iwlwifi-7265D-16.ucode
1392640 lib/firmware/iwlwifi-7265D-21.ucode
1396736 opt/google/chrome/keyboard_resources.pak
1417216 usr/share/cups/ipptool/document-letter.pdf
1421312 opt/google/chrome/resources/about_os_credits.html
1429504 usr/share/chromeos-assets/genius_app/images/200/discover/edit_photo.webp
1449984 usr/lib64/libbase-core-395517.so
1474560 usr/bin/core_collector
1474560 usr/share/cups/ipptool/document-letter.ps
1482752 usr/bin/core_collector32
1499136 usr/lib64/libxml2.so.2.9.4
1581056 opt/google/chrome/resources/chromeos/chromevox/braille/liblouis_nacl_x86_64.nexe
1638400 usr/sbin/flashrom
1671168 opt/google/chrome/resources/inspector/inspector.js
1675264 usr/lib64/locale/locale-archive
1740800 lib64/libc-2.23.so
1843200 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_sz_nexe
1925120 usr/share/chromeos-assets/easy_unlock/easy_unlock_background.js
1945600 usr/share/chromeos-assets/easy_unlock/easy_unlock_pairing.js
2121728 usr/lib64/va/drivers/i965_drv_video.so
2170880 opt/google/chrome/pnacl/pnacl_public_x86_64_ld_nexe
2215936 usr/lib64/libcrypto.so.1.0.0
2273280 usr/share/fonts/tibt-jomolhari/Jomolhari-alpha3c-0605331.ttf
2334720 usr/share/chromeos-assets/input_methods/input_tools/en_gesture/gesture_lexicon.bin
2445312 usr/lib64/Dptf.so
2670592 usr/lib64/libpoppler.so.52.0.0
2674688 usr/share/chromeos-assets/input_methods/input_tools/wubi/system_dictionary.bin
2818048 opt/google/chrome/chrome_100_percent.pak
3489792 opt/google/chrome/nacl_helper_nonsfi
3608576 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/system_dictionary.bin
3678208 opt/google/chrome/nacl_irt_x86_64.nexe
3813376 usr/share/fonts/ko-nanum/NanumMyeongjo.ttf
3985408 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/bigram.bin
4059136 opt/google/chrome/libosmesa.so
4075520 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_id-ID.zvoice
4096000 usr/share/chromeos-assets/quickoffice/scripts/mainQOWT.js
4210688 usr/share/chromeos-assets/input_methods/input_tools/en/system_dictionary.bin
4304896 usr/share/fonts/ko-nanum/NanumGothicBold.ttf
4358144 usr/share/fonts/ko-nanum/NanumGothic.ttf
4640768 usr/share/chromeos-assets/input_methods/input_tools/pinyin/system_dictionary.bin
4653056 usr/share/fonts/ko-nanum/NanumMyeongjoBold.ttf
4722688 usr/bin/shill
4763648 usr/share/chromeos-assets/input_methods/input_tools/ko/system_hanja_dictionary.bin
4993024 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_pt-BR.zvoice
5128192 usr/share/chromeos-assets/input_methods/input_tools/nacl_session_handler_x86_64.nexe
5451776 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/word_x86_64.nexe
5664768 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-IN.zvoice
5795840 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-GB.zvoice
5828608 boot/vmlinuz-4.4.35-06593-gaa8a8b8
6000640 usr/share/fonts/noto/NotoColorEmoji.ttf
6189056 opt/google/chrome/libwidevinecdm.so
6684672 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-US.zvoice
6688768 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-ES.zvoice
6725632 usr/lib64/dri/i965_dri.so
6840320 opt/google/chrome/nacl_helper
6938624 usr/share/chromeos-assets/input_methods/input_tools/pinyin/bigram.bin
7106560 opt/displaylink/DisplayLinkManager
7483392 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_fr-FR.zvoice
7512064 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_nl-NL.zvoice
7618560 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-US.zvoice
7634944 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_ko-KR.zvoice
7757824 usr/share/fonts/ja-motoyafonts/MTG4g5H.ttc
7815168 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_de-DE.zvoice
8093696 opt/google/chrome/chrome_200_percent.pak
8409088 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_it-IT.zvoice
8491008 usr/share/fonts/ja-motoyafonts/MTG4g2bH.ttc
9129984 usr/share/fonts/ascender_licensed/MSungB5HK-Regular.ttf
9195520 usr/share/chromeos-assets/input_methods/input_tools/zipped_data_chromeos
9207808 usr/sbin/chromeos-firmwareupdate
9670656 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_hi-IN.zvoice
9981952 usr/share/chromeos-assets/input_methods/input_tools/decoder_x86_64.nexe
9981952 usr/share/fonts/ja-motoyafonts/MTG4m5H.ttc
10149888 opt/google/chrome/icudtl.dat
10272768 usr/share/chromeos-assets/demo_app/screensaver-video.webm
10371072 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/sheet_x86_64.nexe
10850304 usr/share/fonts/ja-motoyafonts/MTG4m2bH.ttc
11481088 usr/share/fonts/ascender_licensed/MSungB5HK-Bold.ttf
11747328 usr/share/fonts/ascender_licensed/MSungGB18030-Regular.ttf
12197888 usr/lib64/libgs.so.9.19
13234176 usr/share/chromeos-assets/input_methods/input_tools/en/bigram.bin
14114816 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_llc_nexe
16338944 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/point_x86_64.nexe
17084416 usr/share/fonts/ascender_licensed/MSungGB18030-Bold.ttf
18702336 usr/share/fonts/notocjk/NotoSansCJK-Regular.ttc
19275776 usr/share/fonts/notocjk/NotoSansCJK-Bold.ttc
23265280 usr/lib64/dri/i915_dri.so
23265280 usr/lib64/dri/kms_swrast_dri.so
23265280 usr/lib64/dri/swrast_dri.so
26513408 usr/share/chromeos-assets/speech_synthesis/patts/tts_service_x86-64.nexe
28532736 opt/google/chrome/resources.pak
31993856 opt/google/chrome/pepper/libpepflashplayer.so
46718976 opt/google/containers/android/vendor.raw.img
154947584 opt/google/chrome/chrome
466649088 opt/google/containers/android/system.raw.img

One immediately notices the increase in the android image size, from 466MB to 473MB.  A similar increase (7MB) caused the earlier occurrence of this error.

Took a quick look at file systems sizes in disk_layout.json (thank you Mike).

We have these in the public overlays:

$ find -name disk*json | xargs grep fs_size
./project-labstation/scripts/disk_layout.json:        "fs_size":"2000 MiB"
./overlay-beaglebone/scripts/disk_layout.json:        "fs_size":"350 MiB"
./overlay-beaglebone/scripts/disk_layout.json:        "fs_size":"350 MiB"
./project-moblab/scripts/disk_layout.json:        "fs_size":"2000 MiB"
./overlay-quawks/scripts/disk_layout.json:        "fs_size_min": "1750 MiB",
./overlay-quawks/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./baseboard-poppy/scripts/disk_layout.json:        "fs_size_min": "1750 MiB",
./baseboard-poppy/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./overlay-samus/scripts/disk_layout.json:        "fs_size_min": "1750 MiB",
./overlay-samus/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./overlay-clapper/scripts/disk_layout.json:        "fs_size_min": "1750 MiB",
./overlay-clapper/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./overlay-amd64-generic/scripts/disk_layout.json:        "fs_size":"2000 MiB"
./overlay-sama5d3/scripts/disk_layout.json:        "fs_size":"350 MiB"
./overlay-sama5d3/scripts/disk_layout.json:        "fs_size":"350 MiB"
./baseboard-kunimitsu/scripts/disk_layout.json:        "fs_size_min": "1750 MiB",
./baseboard-kunimitsu/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./overlay-veyron/scripts/disk_layout.json:        "fs_size_min": "1628 MiB",
./overlay-veyron/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./baseboard-reef/scripts/disk_layout.json:        "fs_size_min": "1750 MiB",
./baseboard-reef/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./baseboard-strago/scripts/disk_layout.json:        "fs_size_min": "1750 MiB",
./baseboard-strago/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./baseboard-glados/scripts/disk_layout.json:        "fs_size_min": "1750 MiB",
./baseboard-glados/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./overlay-x86-generic/scripts/disk_layout.json:        "fs_size":"2000 MiB"
./baseboard-gru/scripts/disk_layout.json:        "fs_size_min": "1628 MiB",
./baseboard-gru/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./baseboard-auron/scripts/disk_layout.json:        "fs_size_min": "1750 MiB",
./baseboard-auron/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./baseboard-oak/scripts/disk_layout.json:        "fs_size_min": "1628 MiB",
./baseboard-oak/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./overlay-arm-generic/scripts/disk_layout.json:        "fs_size":"2000 MiB"
./overlay-eve/scripts/disk_layout.json:        "fs_size_min": "1750 MiB",
./overlay-eve/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./overlay-squawks/scripts/disk_layout.json:        "fs_size_min": "1750 MiB",
./overlay-squawks/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./baseboard-jecht/scripts/disk_layout.json:        "fs_size_min": "1750 MiB",
./baseboard-jecht/scripts/disk_layout.json:        "fs_size_min": "0 MiB",
./overlay-gizmo/scripts/disk_layout.json:        "fs_size":"350 MiB"

Will now look for partition sizes.

private-overlays/project-cheets-private/chromeos-base/android-container/android-container-9999.ebuild:145 extracts the vendor files into the rootfs, then private-overlays/project-cheets-private/scripts/board_specific_setup.sh runs squashfs on those files and deletes them from the rootfs.

now i'm not entirely sure we can use the tmpfs trick, let me see if we can leave those files in another place and then maybe bind-mount it just for board_specific_setup.sh?

Rationale behind the tmpfs trick maybe-not-working:

* src_install() cannot write anywhere outside of ${D}, otherwise it would cause a sandbox error: http://www.funtoo.org/Ebuild_Functions#src_install
* Even if we were to move the unbundling of the vendor image elsewhere (say.. src_unpack()), other packages (like arc-mesa and the various arc-...-drivers) need to install their stuff _somewhere_, and the vendor/ directory seems like a good spot to do so.

Having said that, one thing we could do is *not* unsquashfs the vendor image in android-container's src_install(), rather just copy the raw vendor image. We let any package install stuff into vendor/, and then in both board_specific_setup.sh and deploy_vendor_image.sh we move everything to a temporary directory, delete all the junk from the rootfs and re-create the vendor image. this will cause the bulk of the houdini files compressed the whole time.

The image is still building, but something similar to this should save a non-trivial amount of space https://chrome-internal-review.googlesource.com/#/c/314713/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/bb019cdde40bd559f45a3bd3068d27062ee3959d

commit bb019cdde40bd559f45a3bd3068d27062ee3959d
Author: Luigi Semenzato <semenzato@chromium.org>
Date: Tue Jan 10 20:17:57 2017

TEMPORARY: bump file system size for reef

Just for fixing CQ temporarily.  See bug for details.

BUG= chromium:676152 
TEST=none

Change-Id: I8109aca7419284e66178b54a5b8164313de584fe
Reviewed-on: https://chromium-review.googlesource.com/426368
Tested-by: Luigi Semenzato <semenzato@chromium.org>
Reviewed-by: Sameer Nanda <snanda@chromium.org>
Reviewed-by: Luis Hector Chavez <lhchavez@chromium.org>
Commit-Queue: Luigi Semenzato <semenzato@chromium.org>

[modify] https://crrev.com/bb019cdde40bd559f45a3bd3068d27062ee3959d/baseboard-reef/scripts/disk_layout.json

 Issue 679774  has been merged into this issue.

I tried Luis's patch from comment #16, but the trybot still failed with "out of space".

The command was

$ cbuildbot --remote -g  426918  -g *314713 reef-paladin

try failure for semenzato:426918,*314713 on paladin @ r41813966f9d736fe7b0fb788dd2139b77849cac2

  File "/mnt/host/source/chromite/licensing/licenses_lib.py", line 1056, in GenerateHTMLLicenseOutput
    self.EvaluateTemplate(file_template, env).encode('UTF-8'))
  File "/mnt/host/source/chromite/lib/osutils.py", line 130, in WriteFile
    f.writelines(cros_build_lib.iflatten_instance(content))
IOError: [Errno 28] No space left on device
ERROR   : Here are the biggest [partially-]extracted files (by disk usage):
1482752 usr/bin/core_collector32
1499136 usr/lib64/libxml2.so.2.9.4
1581056 opt/google/chrome/resources/chromeos/chromevox/braille/liblouis_nacl_x86_64.nexe
1671168 opt/google/chrome/resources/inspector/inspector.js
1675264 usr/lib64/locale/locale-archive
1683456 usr/sbin/flashrom
1740800 lib64/libc-2.23.so
1818624 opt/google/containers/android/vendor/lib/arm/libstagefright.so
1818624 opt/google/containers/android/vendor/lib/arm/nb/libstagefright.so
1843200 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_sz_nexe
1925120 usr/share/chromeos-assets/easy_unlock/easy_unlock_background.js
1945600 usr/share/chromeos-assets/easy_unlock/easy_unlock_pairing.js
1949696 opt/google/containers/android/vendor/lib/arm/libicui18n.so
1949696 opt/google/containers/android/vendor/lib/arm/nb/libicui18n.so
2121728 usr/lib64/va/drivers/i965_drv_video.so
2170880 opt/google/chrome/pnacl/pnacl_public_x86_64_ld_nexe
2215936 usr/lib64/libcrypto.so.1.0.0
2228224 opt/google/containers/android/vendor/lib/mediadrm/libwvdrmengine.so
2273280 usr/share/fonts/tibt-jomolhari/Jomolhari-alpha3c-0605331.ttf
2334720 usr/share/chromeos-assets/input_methods/input_tools/en_gesture/gesture_lexicon.bin
2445312 usr/lib64/Dptf.so
2568192 opt/google/containers/android/vendor/bin/houdini
2650112 lib/udev/hwdb.d/20-pci-vendor-model.hwdb
2670592 usr/lib64/libpoppler.so.52.0.0
2674688 usr/share/chromeos-assets/input_methods/input_tools/wubi/system_dictionary.bin
2818048 opt/google/chrome/chrome_100_percent.pak
3489792 opt/google/chrome/nacl_helper_nonsfi
3608576 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/system_dictionary.bin
3678208 opt/google/chrome/nacl_irt_x86_64.nexe
3813376 usr/share/fonts/ko-nanum/NanumMyeongjo.ttf
3985408 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/bigram.bin
4059136 opt/google/chrome/libosmesa.so
4075520 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_id-ID.zvoice
4096000 usr/share/chromeos-assets/quickoffice/scripts/mainQOWT.js
4190208 opt/google/containers/android/vendor/lib/arm/libpdfium.so
4210688 usr/share/chromeos-assets/input_methods/input_tools/en/system_dictionary.bin
4304896 usr/share/fonts/ko-nanum/NanumGothicBold.ttf
4317184 opt/google/containers/android/vendor/lib/arm/libskia.so
4317184 opt/google/containers/android/vendor/lib/arm/nb/libskia.so
4358144 usr/share/fonts/ko-nanum/NanumGothic.ttf
4640768 usr/share/chromeos-assets/input_methods/input_tools/pinyin/system_dictionary.bin
4653056 usr/share/fonts/ko-nanum/NanumMyeongjoBold.ttf
4657152 opt/google/containers/android/vendor/lib/libhoudini.so
4722688 usr/bin/shill
4763648 usr/share/chromeos-assets/input_methods/input_tools/ko/system_hanja_dictionary.bin
4993024 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_pt-BR.zvoice
5128192 usr/share/chromeos-assets/input_methods/input_tools/nacl_session_handler_x86_64.nexe
5451776 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/word_x86_64.nexe
5664768 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-IN.zvoice
5795840 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-GB.zvoice
5828608 boot/vmlinuz-4.4.35-06593-g28c3ec0
6000640 usr/share/fonts/noto/NotoColorEmoji.ttf
6189056 opt/google/chrome/libwidevinecdm.so
6283264 opt/google/containers/android/vendor/lib/dri/kms_swrast_dri.so
6287360 opt/google/containers/android/vendor/lib/arm/libart.so
6561792 opt/google/containers/android/vendor/lib/dri/i965_dri.so
6684672 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-US.zvoice
6688768 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-ES.zvoice
6729728 usr/lib64/dri/i965_dri.so
6836224 opt/google/chrome/nacl_helper
6938624 usr/share/chromeos-assets/input_methods/input_tools/pinyin/bigram.bin
7106560 opt/displaylink/DisplayLinkManager
7483392 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_fr-FR.zvoice
7512064 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_nl-NL.zvoice
7618560 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-US.zvoice
7634944 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_ko-KR.zvoice
7757824 usr/share/fonts/ja-motoyafonts/MTG4g5H.ttc
7815168 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_de-DE.zvoice
8093696 opt/google/chrome/chrome_200_percent.pak
8409088 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_it-IT.zvoice
8491008 usr/share/fonts/ja-motoyafonts/MTG4g2bH.ttc
9129984 usr/share/fonts/ascender_licensed/MSungB5HK-Regular.ttf
9195520 usr/share/chromeos-assets/input_methods/input_tools/zipped_data_chromeos
9252864 usr/sbin/chromeos-firmwareupdate
9670656 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_hi-IN.zvoice
9957376 opt/google/containers/android/vendor/pittpatt/models/recognition/face.face.y0-y0-71-N-tree_7-wmd.bin
9981952 usr/share/chromeos-assets/input_methods/input_tools/decoder_x86_64.nexe
9981952 usr/share/fonts/ja-motoyafonts/MTG4m5H.ttc
10149888 opt/google/chrome/icudtl.dat
10272768 usr/share/chromeos-assets/demo_app/screensaver-video.webm
10371072 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/sheet_x86_64.nexe
10850304 usr/share/fonts/ja-motoyafonts/MTG4m2bH.ttc
11481088 usr/share/fonts/ascender_licensed/MSungB5HK-Bold.ttf
11747328 usr/share/fonts/ascender_licensed/MSungGB18030-Regular.ttf
12197888 usr/lib64/libgs.so.9.19
13234176 usr/share/chromeos-assets/input_methods/input_tools/en/bigram.bin
14114816 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_llc_nexe
15663104 opt/google/containers/android/vendor/lib/arm/libLLVM.so
16338944 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/point_x86_64.nexe
17084416 usr/share/fonts/ascender_licensed/MSungGB18030-Bold.ttf
18702336 usr/share/fonts/notocjk/NotoSansCJK-Regular.ttc
19275776 usr/share/fonts/notocjk/NotoSansCJK-Bold.ttc
23265280 usr/lib64/dri/i915_dri.so
23265280 usr/lib64/dri/kms_swrast_dri.so
23265280 usr/lib64/dri/swrast_dri.so
26513408 usr/share/chromeos-assets/speech_synthesis/patts/tts_service_x86-64.nexe
28536832 opt/google/chrome/resources.pak
31993856 opt/google/chrome/pepper/libpepflashplayer.so
154849280 opt/google/chrome/chrome
469475328 opt/google/containers/android/system.raw.img
ERROR   : Target image has run out of space:
ERROR   : Filesystem     1M-blocks  Used Available Use% Mounted on
ERROR   : /dev/loop0          1723  1723         0 100% /mnt/host/source/src/build/images/reef/R57-9171.0.2017_01_10_1933-a1/rootfs
INFO    : Unmounting image from /mnt/host/source/src/build/images/reef/R57-9171.0.2017_01_10_1933-a1/stateful and /mnt/host/source/src/build/images/reef/R57-9171.0.2017_01_10_1933-a1/rootfs
Cleaning up /usr/local symlinks for /mnt/host/source/src/build/images/reef/R57-9171.0.2017_01_10_1933-a1/stateful/dev_image
An error occurred in your build so your latest output directory is invalid.
Running in non-interactive mode so deleting output directory.
Deleted /mnt/host/source/src/build/images/reef/R57-9171.0.2017_01_10_1933-a1
19:36:55: ERROR: 
return code: 1; command: /b/cbuild/internal_master/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmplx7bLS' 'USE=chrome_internal' 'FEATURES=separatedebug' -- ./build_image '--board=reef' --replace '--version=' '--builder_path=trybot-reef-paladin/R57-9171.0.0-b1881' base test
cmd=['/b/cbuild/internal_master/chromite/bin/cros_sdk', 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmplx7bLS', 'USE=chrome_internal', 'FEATURES=separatedebug', '--', './build_image', '--board=reef', '--replace', '--version=', '--builder_path=trybot-reef-paladin/R57-9171.0.0-b1881', 'base', 'test'], cwd=/b/cbuild/internal_master, extra env={'PARALLEL_EMERGE_STATUS_FILE': '/tmp/tmplx7bLS', 'USE': 'chrome_internal', 'FEATURES': 'separatedebug'}

No longer an emergency since the FS size bump seems to have worked.

this is odd, it builds fine on my machine.

anyways, i now realized that reef uses android-container-nyc, so let me prepare an additional change for that board.

re #21:

The successful build *might* be due to https://chromium-review.googlesource.com/#/c/423195/ , which removes a few MB from the file system.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/project-cheets-private/+/a28c903dd196694cb6708e3a0f75d1e22303616f

commit a28c903dd196694cb6708e3a0f75d1e22303616f
Author: Luis Hector Chavez <lhchavez@google.com>
Date: Tue Jan 10 22:18:14 2017

Saved 40M. There's an additional delta of ~8MB worth of files that are written to the rootfs uncompressed and then removed and squashfs'd, but I don't think we can easily get rid of those.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/project-cheets-private/+/7cfedf4e0ea80de2ee0a98819e35b908710d3235

commit 7cfedf4e0ea80de2ee0a98819e35b908710d3235
Author: Luis Hector Chavez <lhchavez@google.com>
Date: Wed Jan 11 22:55:25 2017

In theory this is fixed.  (For now.)  Good job everybody.

Filed https://bugs.chromium.org/p/chromium/issues/detail?id=680834 to restore file collision protection.

ever since change in #25 reef is failing to build, so something else is wrong.

https://chrome-internal-review.googlesource.com/#/c/317535/ is waiting CQ to pick up and not being picked up.

a new CQ run hadn't started between the time it was set to Commit-Queue+1 and when you posted that comment

let's keep the conversation in  issue 680532  which was opened to track the new failure

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/overlays/project-cheets-private/+/5c5cdf5e1034896d1422415c1605bc2e2b94fa2b

commit 5c5cdf5e1034896d1422415c1605bc2e2b94fa2b
Author: Dominik Laskowski <domlaskowski@google.com>
Date: Tue Jan 17 21:17:42 2017