build_image run out of space |
|||||||||||
Issue descriptionBuilder coral-paladin Build #1242 https://luci-milo.appspot.com/buildbot/chromeos/coral-paladin/1242 https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fcoral-paladin%2F1242%2F%2B%2Frecipes%2Fsteps%2FBuildImage%2F0%2Fstdout Started virtual/target-os-1.5-r1 (logged in /tmp/target-os-1.5-r1-71sU88) Completed virtual/target-os-1.5-r1 (in 0m2.2s) Pending 0/400, [Time 22:00:07 | Elapsed 1m55.8s | Load 14.78 11.09 11.3] Merge complete Done INFO : Generating license credits page. Time: licenses: Unhandled exception: Traceback (most recent call last): File "/mnt/host/source/chromite/licensing/licenses", line 168, in <module> DoMain() File "/mnt/host/source/chromite/licensing/licenses", line 164, in DoMain commandline.ScriptWrapperMain(FindTarget) File "/mnt/host/source/chromite/lib/commandline.py", line 910, in ScriptWrapperMain ret = target(argv[1:]) File "/mnt/host/source/chromite/licensing/licenses.py", line 223, in main licensing.GenerateHTMLLicenseOutput(opts.output) File "/mnt/host/source/chromite/licensing/licenses_lib.py", line 1056, in GenerateHTMLLicenseOutput self.EvaluateTemplate(file_template, env).encode('UTF-8')) File "/mnt/host/source/chromite/lib/osutils.py", line 130, in WriteFile f.writelines(cros_build_lib.iflatten_instance(content)) IOError: [Errno 28] No space left on device ^^^^^^^^^^ ERROR : Here are the biggest [partially-]extracted files (by disk usage): 1388544 lib/firmware/iwlwifi-7265D-14.ucode 1392640 lib/firmware/iwlwifi-7265D-16.ucode 1392640 lib/firmware/iwlwifi-7265D-21.ucode 1396736 opt/google/chrome/keyboard_resources.pak 1413120 opt/google/chrome/resources/inspector/audits2_worker/audits2_worker_module.js 1417216 usr/share/cups/ipptool/document-letter.pdf 1429504 usr/lib64/libgio-2.0.so.0.5000.3 1445888 usr/lib64/libbase-core-395517.so 1474560 opt/google/chrome/resources/about_os_credits.html 1474560 usr/share/cups/ipptool/document-letter.ps 1478656 usr/bin/core_collector 1486848 usr/bin/core_collector32 1511424 usr/lib64/libxml2.so.2.9.4 1515520 usr/share/chromeos-assets/easy_unlock/easy_unlock_background.js 1540096 usr/share/chromeos-assets/easy_unlock/easy_unlock_pairing.js 1581056 opt/google/chrome/resources/chromeos/chromevox/braille/liblouis_nacl_x86_64.nexe 1654784 opt/google/chrome/resources/inspector/inspector.js 1667072 opt/google/chrome/resources/inspector/unit_test_runner.js 1671168 usr/sbin/flashrom 1675264 usr/lib64/locale/locale-archive 1687552 opt/google/chrome/resources/inspector/integration_test_runner.js 1708032 usr/bin/perf 1708032 usr/bin/trace 1740800 lib64/libc-2.23.so 1908736 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_sz_nexe 2154496 usr/lib64/libcrypto.so.1.0.0 2170880 opt/google/chrome/pnacl/pnacl_public_x86_64_ld_nexe 2207744 usr/lib64/libprotobuf.so.13.0.0 2260992 usr/lib64/libvulkan_intel.so 2273280 usr/share/fonts/tibt-jomolhari/Jomolhari-alpha3c-0605331.ttf 2334720 usr/share/chromeos-assets/input_methods/input_tools/en_gesture/gesture_lexicon.bin 2461696 opt/google/chrome/nacl_helper 2650112 usr/lib64/libpoppler.so.52.0.0 2674688 usr/share/chromeos-assets/input_methods/input_tools/wubi/system_dictionary.bin 2818048 usr/lib64/libndr-standard.so.0.0.1 2908160 usr/lib64/samba/libsmbd-base-samba4.so 3043328 usr/bin/superiotool 3088384 usr/lib64/Dptf.so 3547136 opt/google/chrome/libosmesa.so 3608576 opt/google/chrome/nacl_helper_nonsfi 3608576 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/system_dictionary.bin 3743744 opt/google/chrome/nacl_irt_x86_64.nexe 3985408 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/bigram.bin 4100096 usr/share/chromeos-assets/quickoffice/scripts/mainQOWT.js 4210688 usr/share/chromeos-assets/input_methods/input_tools/en/system_dictionary.bin 4263936 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_pt-BR.zvoice 4300800 usr/share/fonts/ko-nanum/NanumGothicBold.ttf 4321280 usr/lib64/va/drivers/i965_drv_video.so 4358144 usr/share/fonts/ko-nanum/NanumGothic.ttf 4489216 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-GB.zvoice 4640768 usr/share/chromeos-assets/input_methods/input_tools/pinyin/system_dictionary.bin 4734976 opt/google/chrome/chrome_100_percent.pak 4763648 usr/share/chromeos-assets/input_methods/input_tools/ko/system_hanja_dictionary.bin 4861952 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-IN.zvoice 5013504 usr/bin/shill 5128192 usr/share/chromeos-assets/input_methods/input_tools/nacl_session_handler_x86_64.nexe 5414912 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-US.zvoice 5713920 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/word_x86_64.nexe 6053888 boot/vmlinuz-4.4.86-11736-g62e8131a1278 6062080 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_ko-KR.zvoice 6160384 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-US.zvoice 6643712 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_nl-NL.zvoice 6938624 usr/share/chromeos-assets/input_methods/input_tools/pinyin/bigram.bin 7114752 opt/google/containers/android/vendor/lib/dri/kms_swrast_dri.so 7266304 usr/share/fonts/noto/NotoColorEmoji.ttf 7323648 opt/google/chrome/libwidevinecdm.so 7401472 opt/displaylink/DisplayLinkManager 7417856 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_it-IT.zvoice 7430144 usr/lib64/dri/i965_dri.so 7512064 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_id-ID.zvoice 8740864 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_de-DE.zvoice 8871936 opt/google/containers/android/vendor/lib/dri/i965_dri.so 9195520 usr/share/chromeos-assets/input_methods/input_tools/zipped_data_chromeos 9351168 usr/sbin/chromeos-firmwareupdate 9973760 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_fr-FR.zvoice 9981952 usr/share/chromeos-assets/input_methods/input_tools/decoder_x86_64.nexe 10215424 opt/google/chrome/icudtl.dat 10272768 usr/share/chromeos-assets/demo_app/screensaver-video.webm 10371072 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/sheet_x86_64.nexe 10698752 opt/google/chrome/chrome_200_percent.pak 11603968 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_hi-IN.zvoice 12197888 usr/lib64/libgs.so.9.19 12505088 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-ES.zvoice 13234176 usr/share/chromeos-assets/input_methods/input_tools/en/bigram.bin 14114816 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_llc_nexe 16338944 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/point_x86_64.nexe 17534976 usr/share/fonts/notocjk/NotoSansCJK-Light.ttc 18776064 usr/share/fonts/notocjk/NotoSansCJK-Regular.ttc 19345408 usr/share/fonts/notocjk/NotoSansCJK-Bold.ttc 21925888 usr/lib64/dri/kms_swrast_dri.so 21925888 usr/lib64/dri/swrast_dri.so 24686592 usr/share/fonts/notocjk/NotoSerifCJK-Light.ttc 24842240 usr/share/fonts/notocjk/NotoSerifCJK-Regular.ttc 25915392 usr/share/fonts/notocjk/NotoSerifCJK-Bold.ttc 27471872 opt/google/chrome/resources.pak 29728768 usr/share/chromeos-assets/speech_synthesis/patts/tts_service_x86-64.nexe 32276480 opt/google/chrome/pepper/libpepflashplayer.so 46010368 opt/google/containers/android/vendor.raw.img 138694656 opt/google/chrome/chrome 451137536 opt/google/containers/android/system.raw.img ERROR : Target image has run out of space: ERROR : Filesystem 1M-blocks Used Available Use% Mounted on ERROR : /dev/loop0 1723 1723 0 100% /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2/rootfs INFO : Unmounting image from /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2/stateful and /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2/rootfs Cleaning up /usr/local symlinks for /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2/stateful/dev_image An error occurred in your build so your latest output directory is invalid. Running in non-interactive mode so deleting output directory. Deleted /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2 The CLs in this CQ build do not look like the cause of the out of space failure.
,
Sep 25 2017
"No common CLs in these failed builds" - what builds are these?
,
Sep 25 2017
Actions: 1) respawninstance the GCE: cros-beefy393-c2 2) Mark coral-paladin as experimental Waiting for next CQ round to verify whether respawn works. Re #2, these failed builds = all failed coral-paladin due to 'out of spaces' issue.
,
Sep 25 2017
Coral-paladin still failed: ERROR : Target image has run out of space: ERROR : Filesystem 1M-blocks Used Available Use% Mounted on ERROR : /dev/loop0 1723 1723 0 100% /mnt/host/source/src/build/images/coral/R63-9973.0.0-rc1/rootfs INFO : Unmounting image from /mnt/host/source/src/build/images/coral/R63-9973.0.0-rc1/stateful and /mnt/host/source/src/build/images/coral/R63-9973.0.0-rc1/rootfs Cleaning up /usr/local symlinks for /mnt/host/source/src/build/images/coral/R63-9973.0.0-rc1/stateful/dev_image An error occurred in your build so your latest output directory is invalid. What's this 1M blocks?
,
Sep 25 2017
,
Sep 25 2017
,
Sep 25 2017
Oops, it's already experimental. Downgrading to P1
,
Sep 25 2017
Surprisingly this round of coral-paladin passed build_packages: https://uberchromegw.corp.google.com/i/chromeos/builders/coral-paladin/builds/1248 We have put in any fix?
,
Sep 25 2017
Is this bug 767521?
,
Sep 25 2017
YH please take a look at this issue.
,
Sep 26 2017
coral-paladin build #1250 and #1251 succeeded. However build #1252 failed again with the same error. https://luci-milo.appspot.com/buildbot/chromeos/coral-paladin/1252
,
Sep 26 2017
Any progress? coral-paladin continues this failure, so it will keep staying in experimental state.
,
Sep 26 2017
I am increasing the partition size to see if that works. Tryjob is currently progressing. Thanks.
,
Sep 26 2017
Trybot build passed that build_image stage without issue. IMHO, should we try it few more times before pushing the change in? https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/paladin/builds/3844 Thanks.
,
Sep 26 2017
Nice! Yeah I prefer to try it few more times. Based on the history of coral-paladin (https://uberchromegw.corp.google.com/i/chromeos/builders/coral-paladin), it sometimes passed, sometimes failed. Seems the space it grabs varies.
,
Sep 26 2017
Sure thing. I will try it few more times and update. Thanks.
,
Sep 26 2017
The partition size, or the filesystem size?
,
Sep 27 2017
Yes, that's file system size. I'll revise the CL. Thanks. The few try job runs are all successful. https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/paladin/builds/3844 https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/paladin/builds/3846 https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/paladin/builds/3847 On the other hand, coral-paladin seems to be behaving well (the past 9 runs). So I am not sure do we really need this? Thanks.
,
Sep 27 2017
Good question, we can hold it until next time the same problem happened. Or just submit in case. @don do you have any preference?
,
Sep 27 2017
Actually, is this the same problem vapier has been addressing?
,
Sep 27 2017
if you mean the AU/firmware updater stuff, that doesn't apply to build_image i think we're just running up to the limit again. we bumped it for reef recently.
,
Sep 27 2017
Is there any owner for dealing with us approaching to 2G limit?
,
Sep 27 2017
nope!
,
Sep 28 2017
The build_image run out of space error also happened in guado_moblab-paladin in recent builds, not sure if it's related to this one. https://bugs.chromium.org/p/chromium/issues/detail?id=769655
,
Sep 28 2017
I will push the CL to align coral with reef. Thanks.
,
Oct 1 2017
,
Oct 9 2017
RE: #25 Looks like the problem is due to crbug.com/767521, which has been fixed. The increment of file system size for reef has also been reverted. So we should be able to stay the same. Therefore I'm closing the bug. Please re-open if in need. Thanks.
,
Jan 22 2018
,
Jan 23 2018
|
|||||||||||
►
Sign in to add a comment |
|||||||||||
Comment 1 by xixuan@chromium.org
, Sep 25 2017Owner: vbendeb@chromium.org