CC Infra groups because this might be a build system issue.

This is a space issue on the *loopback* device, not the builder. This generally means that the space allocated for image building is too small.

This is a CrOS board layout configuration bug, not an infra bug. +vapier@, since I've seen him deal with this sort of thing before.

the sheriffs need to investigate and find out where the extra size is coming from.  you can use the dashboard to track it down:
build_RootFilesystemSize -> $board -> bytes_rootfs_prod
https://chromeperf.appspot.com/report?sid=d2c8d8b3c8780bb331d2aaff94189c36d7a2dceb373903b03424f9982282032f

there's really large jumps there that look suspicious.

Will take a look.

It basically is slow growth. There was a step down, but we are looking at less than 10MB in 6 increments. There are 3 increments of more than 1MB.

The last statement we for the last 6 weeks roughtly.

We have been coasting between 950+MB and 1000MB for 1.5 years. At times it gets pushed down due to cleanups. The question is can we find some more bits to free or is it finally time to increase the size?

So, this is the test image. As an example, the test team added 3 short videos totaling <8MB recently as we didn't want them to access google storage anymore on the CQ to reduce flakiness. I think that was one of many little changes that helped bump against the limit. But as this is just a test image, we should not care about the 1GB limit. I wonder if there are separate limits for test and release images? But I think we should increase the test image limit generously so we can package all tests on the image and avoid network usage in the lab if possible.
 

https://uberchromegw.corp.google.com/i/chromeos/builders/nyan-chrome-pfq/builds/995/steps/BuildImage/logs/stdio
INFO    : The following images will be built  chromiumos_test_image.bin.
[...]
chromeos-chrome-51.0.2665.0_rc-r1: !!! copy /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/chrome -> /mnt/host/source/src/build/images/nyan/R51-7996.0.0-rc3/rootfs/opt/google/chrome/chrome failed.
chromeos-chrome-51.0.2665.0_rc-r1: !!! [Errno 28] No space left on device

For instance
localhost autotest # ls -alh ./cros/video/files/
total 7.6M
drwxr-xr-x 2 root root 4.0K Mar  1 20:49 .
drwxr-xr-x 3 root root 4.0K Mar  1 22:46 ..
-rwxr-xr-x 1 root root 3.1M Mar  1 22:18 720.mp4
-rwxr-xr-x 1 root root 2.4M Mar  1 22:18 720.webm
-rwxr-xr-x 1 root root 2.1M Mar  1 22:18 720_vp8.webm

This change tried to increase the limit to 1.2GB.
https://chromium-review.googlesource.com/#/c/291257/
Still trying to understand it.

Could we temporarily delete these videos or revert the CL until the PFQ greens?

No, we should make sure the size is increased.

For now, I vote for whatever we can do most quickly.


On Wed, Mar 2, 2016 at 2:01 PM, ihf@chromium.org via Monorail <
monorail@chromium.org> wrote:

We are actually out of space writing chrome, not tests as in  issue 516978 . Wrong partition.

Notice that in both instances build_packages is green, but it actually fails to build chromeos-chrome properly. Example:

chromeos-chrome-51.0.2665.0_rc-r1: [24482/24483] CXX obj/content/public/common/chrome_initial.content_switches.o
chromeos-chrome-51.0.2665.0_rc-r1: [24482/24483] CXX obj/chrome/app/chrome_initial.chrome_main.o
chromeos-chrome-51.0.2665.0_rc-r1: [24482/24483] CXX obj/chrome/app/chrome_initial.chrome_main_delegate.o
chromeos-chrome-51.0.2665.0_rc-r1: [24483/24483] LINK chrome
chromeos-chrome-51.0.2665.0_rc-r1: [24483/24483] LINK chrome
chromeos-chrome-51.0.2665.0_rc-r1: Copying Chrome tests into /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/work/test_src
chromeos-chrome-51.0.2665.0_rc-r1:  * Installing test targets: media_unittests sandbox_linux_unittests ppapi_example_video_decode video_decode_accelerator_unittest video_encode_accelerator_unittest gles2_conform_test gles2_conform_test_windowless khronos_glcts_test khronos_glcts_test_windowless libppapi_tests.so chrome_sandbox
chromeos-chrome-51.0.2665.0_rc-r1: armv7a-cros-linux-gnueabi-strip: '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/ppapi_example_video_decode': No such file
chromeos-chrome-51.0.2665.0_rc-r1: armv7a-cros-linux-gnueabi-strip: '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/libppapi_tests.so': No such file
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/lib32': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/lib64': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/mock_nacl_gdb': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/ppapi_nacl_tests_newlib.nmf': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/ppapi_nacl_tests_glibc.nmf': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/ppapi_nacl_tests_newlib_x32.nexe': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/ppapi_nacl_tests_newlib_x64.nexe': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/ppapi_nacl_tests_newlib_arm.nexe': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/ppapi_nacl_tests_glibc_x32.nexe': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/ppapi_nacl_tests_glibc_x64.nexe': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/ppapi_nacl_tests_glibc_arm.nexe': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/test_case.html': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/test_case.html.mock-http-headers': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/test_page.css': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/test_url_loader_data': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: rsync: link_stat "/home/chrome-bot/chrome_root/src/chrome/test/functional" failed: No such file or directory (2)
chromeos-chrome-51.0.2665.0_rc-r1: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/chrome/test/functional': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: rsync: link_stat "/home/chrome-bot/chrome_root/src/content/common/gpu/testdata" failed: No such file or directory (2)
chromeos-chrome-51.0.2665.0_rc-r1: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/content/common/gpu/testdata': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: rsync: change_dir "/home/chrome-bot/chrome_root/src/third_party/accessibility-developer-tools/gen" failed: No such file or directory (2)
chromeos-chrome-51.0.2665.0_rc-r1: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/third_party/accessibility-developer-tools/gen/axs_testing.js': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: rsync: link_stat "/home/chrome-bot/chrome_root/src/pdf/test" failed: No such file or directory (2)
chromeos-chrome-51.0.2665.0_rc-r1: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/pdf/test': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: Copying Page Cycler Data into /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/work/page_cycler_src
chromeos-chrome-51.0.2665.0_rc-r1: Copying Perf Data into /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/work/perf_data_src
chromeos-chrome-51.0.2665.0_rc-r1: Copying Telemetry Framework into /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/work/telemetry_src
chromeos-chrome-51.0.2665.0_rc-r1: rsync: link_stat "/home/chrome-bot/chrome_root/src/third_party/catapult/third_party/Paste/tests/urlparser_data/find_file/dir" failed: No such file or directory (2)
chromeos-chrome-51.0.2665.0_rc-r1: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/third_party/catapult/third_party/Paste/tests/urlparser_data/find_file/dir': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: rsync: link_stat "/home/chrome-bot/chrome_root/src/with" failed: No such file or directory (2)
chromeos-chrome-51.0.2665.0_rc-r1: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/with': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: rsync: change_dir "/home/chrome-bot/chrome_root/src/spaces" failed: No such file or directory (2)
chromeos-chrome-51.0.2665.0_rc-r1: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/spaces/test': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: rsync: link_stat "/home/chrome-bot/chrome_root/src/4.html" failed: No such file or directory (2)
chromeos-chrome-51.0.2665.0_rc-r1: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/4.html': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: rsync: link_stat "/home/chrome-bot/chrome_root/src/third_party/catapult/third_party/Paste/tests/urlparser_data/find_file/test" failed: No such file or directory (2)
chromeos-chrome-51.0.2665.0_rc-r1: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/third_party/catapult/third_party/Paste/tests/urlparser_data/find_file/test': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1: rsync: link_stat "/home/chrome-bot/chrome_root/src/3.html" failed: No such file or directory (2)
chromeos-chrome-51.0.2665.0_rc-r1: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
chromeos-chrome-51.0.2665.0_rc-r1: cp: cannot stat '/var/cache/chromeos-chrome/chrome-src-internal/src/3.html': No such file or directory
chromeos-chrome-51.0.2665.0_rc-r1:  * Installing telemetry binaries: bitmaptools clear_system_cache minidump_stackwalk
chromeos-chrome-51.0.2665.0_rc-r1: armv7a-cros-linux-gnueabi-strip:/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/bitmaptools: File format not recognized
chromeos-chrome-51.0.2665.0_rc-r1:  * Preparing tests
chromeos-chrome-51.0.2665.0_rc-r1:  * Compiling tests
chromeos-chrome-51.0.2665.0_rc-r1:  * Building tests (1):
chromeos-chrome-51.0.2665.0_rc-r1:  * myfaketest
chromeos-chrome-51.0.2665.0_rc-r1: DEBUG:root:Failed to import elasticsearch. Mock classes will be used and calls to Elasticsearch server will be no-op. Test run is not affected by the missing elasticsearch module.
chromeos-chrome-51.0.2665.0_rc-r1: * INFO:root:setup myfaketest.
chromeos-chrome-51.0.2665.0_rc-r1: INFO:root:Dependency chrome_test successfuly built
chromeos-chrome-51.0.2665.0_rc-r1: INFO:root:Dependency page_cycler_dep successfuly built
chromeos-chrome-51.0.2665.0_rc-r1: INFO:root:Dependency perf_data_dep successfuly built
chromeos-chrome-51.0.2665.0_rc-r1: INFO:root:Dependency telemetry_dep successfuly built
chromeos-chrome-51.0.2665.0_rc-r1: INFO:root:############################# SUMMARY #############################
chromeos-chrome-51.0.2665.0_rc-r1: INFO:root:Finished setup -- All tests built successfully
chromeos-chrome-51.0.2665.0_rc-r1: INFO:root:######################### END SUMMARY ##############################
chromeos-chrome-51.0.2665.0_rc-r1:  * Purging *.[do]
chromeos-chrome-51.0.2665.0_rc-r1:  * Purging *.pyc
chromeos-chrome-51.0.2665.0_rc-r1: >>> Source compiled.
chromeos-chrome-51.0.2665.0_rc-r1: >>> Test phase [not enabled]: chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1
chromeos-chrome-51.0.2665.0_rc-r1: 
chromeos-chrome-51.0.2665.0_rc-r1: >>> Install chromeos-chrome-51.0.2665.0_rc-r1 into /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/ category chromeos-base
chromeos-chrome-51.0.2665.0_rc-r1:  * Installing tests
chromeos-chrome-51.0.2665.0_rc-r1: /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/temp/environment: line 12894: get_paths: command not found
chromeos-chrome-51.0.2665.0_rc-r1:  * /home/chrome-bot/chrome_root/src/third_party/chromite/bin/deploy_chrome --board=nyan --build-dir=/var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release --gyp-defines=sysroot=/build/nyan linux_link_libbrlapi=1 use_brlapi=1 proprietary_codecs=1 system_libdir=lib pkg-config=/build/nyan/build/bin/pkg-config use_v4l2_codec=1 use_v4lplugin=0 use_vtable_verify=0 use_ozone=0 use_evdev_gestures=0 use_xkbcommon=0 internal_gles2_conform_tests=0 internal_khronos_glcts_tests=0 linux_use_bundled_binutils=0 linux_use_bundled_gold=0 linux_use_gold_flags=1 linux_use_debug_fission=0 remoting=1 chromeos=1 disable_nacl=0 icu_use_data_file_flag=1 use_cras=1 use_system_minigbm=1 asan=0 clang=0 clang_use_chrome_plugins=0 host_clang=0 target_arch=arm arm_float_abi=hard arm_neon=1 arm_fpu=neon branding=Chrome buildtype=Official internal_gles2_conform_tests=1 internal_khronos_glcts_tests=1 release_extra_cflags='-g' --nostrip --staging-dir=/build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image///opt/google/chrome --staging-flags=X accessibility arm autotest build_tests buildcheck chrome_debug chrome_internal chrome_remoting elibc_glibc gold hardfp highdpi kernel_linux nacl neon opengles runhooks userland_GNU v4l2_codec --staging-only --strict --strip-bin=armv7a-cros-linux-gnueabi-strip --strip-flags=--strip-unneeded --verbose
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:11: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/chrome  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/chrome
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/chrome-wrapper  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/chrome-wrapper
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/chrome_100_percent.pak  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/chrome_100_percent.pak
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/chrome_200_percent.pak  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/chrome_200_percent.pak
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/chrome_material_100_percent.pak  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/chrome_material_100_percent.pak
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/chrome_material_200_percent.pak  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/chrome_material_200_percent.pak
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/keyboard_resources.pak  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/keyboard_resources.pak
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/libexif.so  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/libexif.so
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/libwidevinecdmadapter.so  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/libwidevinecdmadapter.so
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/libwidevinecdm.so  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/libwidevinecdm.so
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/locales/  [d] ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/locales
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/resources/  [d] ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/resources
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/resources.pak  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/resources.pak
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/xdg-settings  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/xdg-settings
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/product_logo_48.png  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/product_logo_48.png
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/chrome_sandbox  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/chrome-sandbox
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/icudtl.dat  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/icudtl.dat
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/lib/libpeerconnection.so does not exist and is optional.  Skipping.
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/libffmpegsumo.so does not exist and is optional.  Skipping.
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/libosmesa.so  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/libosmesa.so
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/libpdf.so does not exist and is optional.  Skipping.
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/libppGoogleNaClPluginChrome.so does not exist and is optional.  Skipping.
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/mojo_shell does not exist and is optional.  Skipping.
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/nacl_helper_bootstrap  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/nacl_helper_bootstrap
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/nacl_irt_arm.nexe  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/nacl_irt_arm.nexe
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/nacl_helper  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/nacl_helper
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/nacl_helper_nonsfi  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/nacl_helper_nonsfi
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/natives_blob.bin  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/natives_blob.bin
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/pnacl/  [d] ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/pnacl
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: /var/cache/chromeos-chrome/chrome-src-internal/src/out_nyan/Release/snapshot_blob.bin  ->  /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome/snapshot_blob.bin
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:16: DEBUG: RunCommand: chmod -R a+r /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:17: DEBUG: RunCommand: find /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/opt/google/chrome -perm /110 -exec chmod a+x '{}' +
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:11: DEBUG: root@build153-m2.golo.chromium.org is not a Google or Chromium user.
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:11: DEBUG: Skipping stats upload.
chromeos-chrome-51.0.2665.0_rc-r1: >>> Completed installing chromeos-chrome-51.0.2665.0_rc-r1 into /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/image/
chromeos-chrome-51.0.2665.0_rc-r1: 
chromeos-chrome-51.0.2665.0_rc-r1:  * QA Notice: command not found:
chromeos-chrome-51.0.2665.0_rc-r1:  * 
chromeos-chrome-51.0.2665.0_rc-r1:  * 	/build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1/temp/environment: line 12894: get_paths: command not found
chromeos-chrome-51.0.2665.0_rc-r1:  * Generating license for chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1 in /build/nyan/tmp/portage/chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:20: INFO: Static license mapping for chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1: BSD-Google,Google-TOS
chromeos-chrome-51.0.2665.0_rc-r1: 03:12:20: INFO: chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1: using stock|cust license(s) Google-TOS,BSD-Google
=== Still running: job chromeos-chrome-51.0.2665.0_rc-r1 (60m28.1s) ===

First of all I am going to add a whole bunch of || die to the ebuild.

Building this locally right now, feel free to chime in:
https://chromium-review.googlesource.com/#/c/330201/

I don't know where the get_paths failures comes from. But it is not in all logs.

The builder does have plenty of free space.

Some of the binaries that are not there were built before, but disappeared.

I did kick off clobbered builds.

Building locally from source with the new ebuild.

With the new ebuild I think I have a local repro

[23821/23821] LINK chrome
Copying Chrome tests into /build/daisy/tmp/portage/chromeos-base/chromeos-chrome-9999/work/test_src
 * Installing test targets: media_unittests sandbox_linux_unittests ppapi_example_video_decode video_decode_accelerator_unittest video_encode_accelerator_unittest libppapi_tests.so chrome_sandbox
armv7a-cros-linux-gnueabi-strip: '/var/cache/chromeos-chrome/chrome-src/src/out_daisy/Release/ppapi_example_video_decode': No such file
 * ERROR: chromeos-base/chromeos-chrome-9999::chromiumos failed (compile phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line   93:  Called src_compile
 *   environment, line 3638:  Called install_chrome_test_resources '/build/daisy/tmp/portage/chromeos-base/chromeos-chrome-9999/work/test_src'
 *   environment, line 2466:  Called test_strip_install '/var/cache/chromeos-chrome/chrome-src/src/out_daisy/Release' '/build/daisy/tmp/portage/chromeos-base/chromeos-chrome-9999/work/test_src/out/Release' 'media_unittests' 'sandbox_linux_unittests' 'ppapi_example_video_decode' 'video_decode_accelerator_unittest' 'video_encode_accelerator_unittest' 'libppapi_tests.so' 'chrome_sandbox'
 *   environment, line 4481:  Called die
 * The specific snippet of code:
 *           $(tc-getSTRIP) --strip-debug --keep-file-symbols "${from}"/${f} -o "${dest}/$(basename ${f})" || die;

(cr) ihf@ql /var/cache/chromeos-chrome/chrome-src/src/out_daisy/Release $ find | grep ppapi_example_video_decode
./lib/libppapi_example_video_decode.so
./lib/libppapi_example_video_decode.so.TOC
./obj/ppapi/ppapi_example_video_decode.actions_depends.stamp
./obj/ppapi/ppapi_example_video_decode.ninja
./obj/ppapi/ppapi_example_video_decode_dev.ninja
./obj/ppapi/examples/video_decode/ppapi_example_video_decode.video_decode.o

i don't think that's reproducing the issue.  the log is showing a build_image failure while setting up the rootfs.  that log just shows your local disk is out of space :).

My local disk is not out of space.

Lets just say we have a lot of silent failures.

I am trying to look at builder artifacts, did they disappear due to the renaming?
** BUILD ARTIFACTS FOR THIS BUILD CAN BE FOUND AT:
**  nyan: https://storage.cloud.google.com/chromeos-image-archive/nyan-chrome-pfq/R51-8001.0.0-rc1/index.html
@@@STEP_LINK@Artifacts[nyan]: nyan-chrome-pfq/R51-8001.0.0-rc1@https://storage.cloud.google.com/chromeos-image-archive/nyan-chrome-pfq/R51-8001.0.0-rc1/index.html@@@

Another question for the cleanup. There is a telemetry find_dependencies directory and file which contain spaces
src/third_party/catapult/third_party/Paste/tests/urlparser_data/find_file/dir with spaces/test 4.html

I am stuck on how to escape this and pass it to install_test_resources. Ideas?
https://cs.corp.google.com/chromeos_public/src/third_party/chromiumos-overlay/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild?q=chromeos-chrome%20DEPS_LIST%20package:%5Echromeos_public$&l=1113&dr=C

(This failure can also seen on the bots, see comment 18.)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/9ea2c632d2610b2fe81cad84359af0c2af352556

commit 9ea2c632d2610b2fe81cad84359af0c2af352556
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Thu Mar 03 00:30:45 2016

chromeos-chrome: die early.

This adds assert and die to many shell operations.
It also removes fast_cp which is dead code.

BUG= chromium:591401 
TEST=emerge with local sources.

Change-Id: I3893dd3349599740bc4b64c7d043ee43b418b942
Reviewed-on: https://chromium-review.googlesource.com/330201
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>

[modify] https://crrev.com/9ea2c632d2610b2fe81cad84359af0c2af352556/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

I am curious where the bot will die with this change. Most likely at the rsync of the file with the spaces.

Here is a snapshot of trying to clean up things that go wrong:
https://chromium-review.googlesource.com/#/c/330127/1/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

Finally, I only noticed now. There are list of the largest files at the end of the failed build_images. All files look sane, except for chrome. It is 400+MB large and seems to fill the remaining space. This might confirm one of my suspicions that it either didn't get stripped or corrupted otherwise. Unfortunately I can't download the artifacts to check. My locally built chrome is only about 100MB in size (but daisy board).

Deleted: skate.txt 6.7 KB

skate.txt 6.7 KB View Download

Deleted: nyan.txt 6.6 KB

nyan.txt 6.6 KB View Download

Kicked off a fresh build for all three ARM boards from source.

your local builds are failing during install from source.  the build bots got passed that and were failing during build image because the target loopback device ran out of space.  i don't think the two are related.

This (or something similar?) is now the primary issue blocking all of the PFQ builders. 

The unhelpful error I am seeing now is:

chromeos-chrome-51.0.2665.0_rc-r1:  * ERROR: chromeos-base/chromeos-chrome-51.0.2665.0_rc-r1::chromiumos failed (compile phase):
chromeos-chrome-51.0.2665.0_rc-r1:  *   (no error message)


We really need to get some sort of a solution in as soon as possible. I'll try digging into the build logs.

the new error is:
chromeos-chrome-51.0.2665.0_rc-r1: armv7a-cros-linux-gnueabi-strip: '/var/cache/chromeos-chrome/chrome-src-internal/src/out_daisy_skate/Release/ppapi_example_video_decode': No such file

most likely related to one of the CLs ihf@ landed recently to try and fix this bug, but it's not the same error as this one (out of space).

It is one if the || die that I landed in 26.

Re 29: The chrome on loopback is 400MB instead of 100MB. Which stage could have caused this if not the chromeos-chrome.ebuild?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/425528bea361d396becfa36552fe99fc2abba1a9

commit 425528bea361d396becfa36552fe99fc2abba1a9
Author: Ilja Friedel <ihf@chromium.org>
Date: Thu Mar 03 20:00:22 2016

Revert "chromeos-chrome: die early."

This reverts commit 9ea2c632d2610b2fe81cad84359af0c2af352556.

The insights from the bot were limited. I will reland at a later time once the cleanup is ready.

BUG= chromium:591782 ,  chromium:591401 

Change-Id: Ie5357ca4213a60694380e9cc269062e34387ea8e
Reviewed-on: https://chromium-review.googlesource.com/330198
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>

[modify] https://crrev.com/425528bea361d396becfa36552fe99fc2abba1a9/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

+ some cros infa folks. Is anyone aware of any toolchain changes that might be responsible for the chrome binary size on some boards in the PFQ (ARM only?) growing by 4x? We're pretty mystified here...

Unfortunately other failures around the same time are making it difficult to isolate when exactly this was introduced. I'll see if I can narrow the window down...

 

OK, looking more closely at the PFQ master failures:

2595: autotest compile failure on:
 veyron_minnie-chromium
 arm-generic_freon-chromium
 arm-generic-chromium-pfq
 x86-generic-chromium-pfq

2598: 2595 failures *plus* ChromeSDK failure in all other boards (BuildImage succeeded on other boards)

2599: 2595 + 2598 failures *plus* BuildImage failure (this issue) on:
 nyan-chrome-pfq
 peach_pit-chrome-pfq
 x86-alex-chrome-pfq
 daisy_skate-chrome-pfq
 veyron_pinky-chrome-pfq

2600: same
2601: ChromeSDK failures resolved, others remain
2602-2605: Same
2605: All boards fail due to change in comment #28

So this was introduced in 2599, on at least the 5 boards where it shows up in 2599, and possibly some or all of the boards with autotest compile failures in 2595.

Incidentally TOT builders are now failing with something similar:
https://uberchromegw.corp.google.com/i/chromiumos.chromium/builders/x86-generic-tot-chromium-pfq-informational
https://uberchromegw.corp.google.com/i/chromiumos.chromium/builders/x86-generic-tot-chromium-pfq-informational/builds/7853

Log snippet:
chromeos-chrome-51.0.2666.0_alpha-r1: Copying Chrome tests into /build/x86-generic/tmp/portage/chromeos-base/chromeos-chrome-51.0.2666.0_alpha-r1/work/test_src
chromeos-chrome-51.0.2666.0_alpha-r1:  * Installing test targets: media_unittests sandbox_linux_unittests ppapi_example_video_decode video_decode_accelerator_unittest video_encode_accelerator_unittest libppapi_tests.so chrome_sandbox
chromeos-chrome-51.0.2666.0_alpha-r1: i686-pc-linux-gnu-strip: '/var/cache/chromeos-chrome/chrome-src/src/out_x86-generic/Release/ppapi_example_video_decode': No such file
chromeos-chrome-51.0.2666.0_alpha-r1:  * ERROR: chromeos-base/chromeos-chrome-51.0.2666.0_alpha-r1::chromiumos failed (compile phase):

Yup, I think it's https://chromium-review.googlesource.com/#/c/330201/, which has since been reverted.

2598, Chrome was at:  51.0.2664.1 
2599, Chrome was at:  51.0.2665.0 

Diff: https://chromium.googlesource.com/chromium/src/+log/50.0.2657.0..51.0.2665.0?pretty=fuller&n=10000

But I believe we can ignore changes prior to #378169

Maybe this?

https://codereview.chromium.org/1680943002/

That is an interesting change. It is actually in my more narrow range:
https://chromium.googlesource.com/chromium/src/+log/51.0.2664.1..51.0.2665.0?pretty=fuller&n=10000
If I only had a repro! My local daisy_skate build went through.

INFO    : Done. Image(s) created in /mnt/host/source/src/build/images/daisy_skate/R51-8004.0.2016_03_03_1356-a1

I'll try to examine the buildbot next. But maybe we can do a speculative revert of --gdb-index?

The gdb-index change is probably it. It increases size of debug binaries by storing a symbol index. 

Do you mind then if I speculatively revert the gdb-index CL? Our turnaround time for proving that it is indeed the cause is ~4 hours...

Please do. 

Speculatively reverting: https://codereview.chromium.org/1762873002/

chromeos-chrom build completed on the bot
https://uberchromegw.corp.google.com/i/chromeos/builders/daisy_skate-chrome-pfq/builds/2119/steps/BuildPackages/logs/stdio

chrome-bot@build259-m2:/b/cbuild/internal_master/chroot/build/daisy_skate$ file opt/google/chrome/chrome
opt/google/chrome/chrome: ELF 32-bit LSB shared object, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, BuildID[sha1]=0xfc1ea729c1d8f10a31eef0eb88985f658f56f3a6, not stripped
chrome-bot@build259-m2:/b/cbuild/internal_master/chroot/build/daisy_skate$ ls -alh opt/google/chrome/chrome
-rwxr-xr-x 1 root root 4.3G Mar  3 16:03 opt/google/chrome/chrome

chrome-bot@build259-m2:/b/cbuild/internal_master/chroot/build/daisy_skate$ ls -alh ./packages/chromeos-base/chromeos-chrome-51.0.2666.0*
-rw-r--r-- 1 root root  88M Mar  3 16:27 ./packages/chromeos-base/chromeos-chrome-51.0.2666.0_rc-r1.debug.tbz2
-rw-r--r-- 1 root root 2.0G Mar  3 16:26 ./packages/chromeos-base/chromeos-chrome-51.0.2666.0_rc-r1.tbz2

This is how it normally should look like
(cr) ihf@ql ~/trunk/chroot/build/daisy_skate $ ls -alh ./packages/chromeos-base/chromeos-chrome-*
-rw-r--r-- 1 root root 1.2G Mar  3 13:36 ./packages/chromeos-base/chromeos-chrome-9999.debug.tbz2
-rw-r--r-- 1 root root 368M Mar  3 13:36 ./packages/chromeos-base/chromeos-chrome-9999.tbz2

I am optimistic the revert will fix this issue. I'll try to pull the artifacts off the bot.

Before I could pull the files off the bot rebooted me. I am still confident that the revert fixes things. Assigning to deputy gardener to verify tomorrow as I will be OOO. If this persists until Monday I'll log into the bot again and grab the files then.

It looks like locally the revert fixed the problem. Still waiting on it to make it through the CQ.

Looks like we're still having PFQ issues.

ERROR   : Target image has run out of space:

One example failure: https://uberchromegw.corp.google.com/i/chromeos/builders/peach_pit-chrome-pfq/builds/1769

The PFQ still has not picked up the revert. I will see what we can do about that.

This has been resolved with the revert.

Bulk verified