Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Starred by 1 user
Status: Fixed
Owner:
Closed: Oct 9
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment
build_image run out of space
Project Member Reported by chinyue@chromium.org, Sep 25 Back to list
Builder coral-paladin Build #1242
https://luci-milo.appspot.com/buildbot/chromeos/coral-paladin/1242

https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fcoral-paladin%2F1242%2F%2B%2Frecipes%2Fsteps%2FBuildImage%2F0%2Fstdout

Started virtual/target-os-1.5-r1 (logged in /tmp/target-os-1.5-r1-71sU88)
Completed virtual/target-os-1.5-r1 (in 0m2.2s)
Pending 0/400, [Time 22:00:07 | Elapsed 1m55.8s | Load 14.78 11.09 11.3]
Merge complete
Done
INFO    : Generating license credits page. Time:
licenses: Unhandled exception:
Traceback (most recent call last):
  File "/mnt/host/source/chromite/licensing/licenses", line 168, in <module>
    DoMain()
  File "/mnt/host/source/chromite/licensing/licenses", line 164, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/mnt/host/source/chromite/lib/commandline.py", line 910, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/mnt/host/source/chromite/licensing/licenses.py", line 223, in main
    licensing.GenerateHTMLLicenseOutput(opts.output)
  File "/mnt/host/source/chromite/licensing/licenses_lib.py", line 1056, in GenerateHTMLLicenseOutput
    self.EvaluateTemplate(file_template, env).encode('UTF-8'))
  File "/mnt/host/source/chromite/lib/osutils.py", line 130, in WriteFile
    f.writelines(cros_build_lib.iflatten_instance(content))
IOError: [Errno 28] No space left on device
^^^^^^^^^^

ERROR   : Here are the biggest [partially-]extracted files (by disk usage):
1388544 lib/firmware/iwlwifi-7265D-14.ucode
1392640 lib/firmware/iwlwifi-7265D-16.ucode
1392640 lib/firmware/iwlwifi-7265D-21.ucode
1396736 opt/google/chrome/keyboard_resources.pak
1413120 opt/google/chrome/resources/inspector/audits2_worker/audits2_worker_module.js
1417216 usr/share/cups/ipptool/document-letter.pdf
1429504 usr/lib64/libgio-2.0.so.0.5000.3
1445888 usr/lib64/libbase-core-395517.so
1474560 opt/google/chrome/resources/about_os_credits.html
1474560 usr/share/cups/ipptool/document-letter.ps
1478656 usr/bin/core_collector
1486848 usr/bin/core_collector32
1511424 usr/lib64/libxml2.so.2.9.4
1515520 usr/share/chromeos-assets/easy_unlock/easy_unlock_background.js
1540096 usr/share/chromeos-assets/easy_unlock/easy_unlock_pairing.js
1581056 opt/google/chrome/resources/chromeos/chromevox/braille/liblouis_nacl_x86_64.nexe
1654784 opt/google/chrome/resources/inspector/inspector.js
1667072 opt/google/chrome/resources/inspector/unit_test_runner.js
1671168 usr/sbin/flashrom
1675264 usr/lib64/locale/locale-archive
1687552 opt/google/chrome/resources/inspector/integration_test_runner.js
1708032 usr/bin/perf
1708032 usr/bin/trace
1740800 lib64/libc-2.23.so
1908736 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_sz_nexe
2154496 usr/lib64/libcrypto.so.1.0.0
2170880 opt/google/chrome/pnacl/pnacl_public_x86_64_ld_nexe
2207744 usr/lib64/libprotobuf.so.13.0.0
2260992 usr/lib64/libvulkan_intel.so
2273280 usr/share/fonts/tibt-jomolhari/Jomolhari-alpha3c-0605331.ttf
2334720 usr/share/chromeos-assets/input_methods/input_tools/en_gesture/gesture_lexicon.bin
2461696 opt/google/chrome/nacl_helper
2650112 usr/lib64/libpoppler.so.52.0.0
2674688 usr/share/chromeos-assets/input_methods/input_tools/wubi/system_dictionary.bin
2818048 usr/lib64/libndr-standard.so.0.0.1
2908160 usr/lib64/samba/libsmbd-base-samba4.so
3043328 usr/bin/superiotool
3088384 usr/lib64/Dptf.so
3547136 opt/google/chrome/libosmesa.so
3608576 opt/google/chrome/nacl_helper_nonsfi
3608576 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/system_dictionary.bin
3743744 opt/google/chrome/nacl_irt_x86_64.nexe
3985408 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/bigram.bin
4100096 usr/share/chromeos-assets/quickoffice/scripts/mainQOWT.js
4210688 usr/share/chromeos-assets/input_methods/input_tools/en/system_dictionary.bin
4263936 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_pt-BR.zvoice
4300800 usr/share/fonts/ko-nanum/NanumGothicBold.ttf
4321280 usr/lib64/va/drivers/i965_drv_video.so
4358144 usr/share/fonts/ko-nanum/NanumGothic.ttf
4489216 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-GB.zvoice
4640768 usr/share/chromeos-assets/input_methods/input_tools/pinyin/system_dictionary.bin
4734976 opt/google/chrome/chrome_100_percent.pak
4763648 usr/share/chromeos-assets/input_methods/input_tools/ko/system_hanja_dictionary.bin
4861952 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-IN.zvoice
5013504 usr/bin/shill
5128192 usr/share/chromeos-assets/input_methods/input_tools/nacl_session_handler_x86_64.nexe
5414912 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-US.zvoice
5713920 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/word_x86_64.nexe
6053888 boot/vmlinuz-4.4.86-11736-g62e8131a1278
6062080 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_ko-KR.zvoice
6160384 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-US.zvoice
6643712 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_nl-NL.zvoice
6938624 usr/share/chromeos-assets/input_methods/input_tools/pinyin/bigram.bin
7114752 opt/google/containers/android/vendor/lib/dri/kms_swrast_dri.so
7266304 usr/share/fonts/noto/NotoColorEmoji.ttf
7323648 opt/google/chrome/libwidevinecdm.so
7401472 opt/displaylink/DisplayLinkManager
7417856 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_it-IT.zvoice
7430144 usr/lib64/dri/i965_dri.so
7512064 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_id-ID.zvoice
8740864 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_de-DE.zvoice
8871936 opt/google/containers/android/vendor/lib/dri/i965_dri.so
9195520 usr/share/chromeos-assets/input_methods/input_tools/zipped_data_chromeos
9351168 usr/sbin/chromeos-firmwareupdate
9973760 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_fr-FR.zvoice
9981952 usr/share/chromeos-assets/input_methods/input_tools/decoder_x86_64.nexe
10215424 opt/google/chrome/icudtl.dat
10272768 usr/share/chromeos-assets/demo_app/screensaver-video.webm
10371072 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/sheet_x86_64.nexe
10698752 opt/google/chrome/chrome_200_percent.pak
11603968 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_hi-IN.zvoice
12197888 usr/lib64/libgs.so.9.19
12505088 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-ES.zvoice
13234176 usr/share/chromeos-assets/input_methods/input_tools/en/bigram.bin
14114816 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_llc_nexe
16338944 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/point_x86_64.nexe
17534976 usr/share/fonts/notocjk/NotoSansCJK-Light.ttc
18776064 usr/share/fonts/notocjk/NotoSansCJK-Regular.ttc
19345408 usr/share/fonts/notocjk/NotoSansCJK-Bold.ttc
21925888 usr/lib64/dri/kms_swrast_dri.so
21925888 usr/lib64/dri/swrast_dri.so
24686592 usr/share/fonts/notocjk/NotoSerifCJK-Light.ttc
24842240 usr/share/fonts/notocjk/NotoSerifCJK-Regular.ttc
25915392 usr/share/fonts/notocjk/NotoSerifCJK-Bold.ttc
27471872 opt/google/chrome/resources.pak
29728768 usr/share/chromeos-assets/speech_synthesis/patts/tts_service_x86-64.nexe
32276480 opt/google/chrome/pepper/libpepflashplayer.so
46010368 opt/google/containers/android/vendor.raw.img
138694656 opt/google/chrome/chrome
451137536 opt/google/containers/android/system.raw.img
ERROR   : Target image has run out of space:
ERROR   : Filesystem     1M-blocks  Used Available Use% Mounted on
ERROR   : /dev/loop0          1723  1723         0 100% /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2/rootfs
INFO    : Unmounting image from /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2/stateful and /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2/rootfs
Cleaning up /usr/local symlinks for /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2/stateful/dev_image
An error occurred in your build so your latest output directory is invalid.
Running in non-interactive mode so deleting output directory.
Deleted /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2


The CLs in this CQ build do not look like the cause of the out of space failure.

 
Cc: akes...@chromium.org xixuan@chromium.org
Owner: vbendeb@chromium.org
@vbendeb I guess 2 possibilities 1) the VM has some issues. 2) bad CL/bug in ToT.

No common CLs in these failed builds, the only CL in last successful coral build is: https://chromium-review.googlesource.com/c/chromiumos/platform/ec/+/670063

Will this CL lead to the big occupation of disk of builder? 
"No common CLs in these failed builds" - what builds are these?
Actions:
1) respawninstance the GCE: cros-beefy393-c2
2) Mark coral-paladin as experimental 

Waiting for next CQ round to verify whether respawn works.


Re #2, these failed builds = all failed coral-paladin due to 'out of spaces' issue.
Cc: dgarr...@chromium.org nxia@chromium.org
Coral-paladin still failed:

ERROR   : Target image has run out of space:
ERROR   : Filesystem     1M-blocks  Used Available Use% Mounted on
ERROR   : /dev/loop0          1723  1723         0 100% /mnt/host/source/src/build/images/coral/R63-9973.0.0-rc1/rootfs
INFO    : Unmounting image from /mnt/host/source/src/build/images/coral/R63-9973.0.0-rc1/stateful and /mnt/host/source/src/build/images/coral/R63-9973.0.0-rc1/rootfs
Cleaning up /usr/local symlinks for /mnt/host/source/src/build/images/coral/R63-9973.0.0-rc1/stateful/dev_image
An error occurred in your build so your latest output directory is invalid.

What's this 1M blocks?
Cc: keta...@chromium.org
Labels: -Pri-1 Pri-0
Labels: -Pri-0 Pri-1
Oops, it's already experimental. Downgrading to P1
Surprisingly this round of coral-paladin passed build_packages:
https://uberchromegw.corp.google.com/i/chromeos/builders/coral-paladin/builds/1248

We have put in any fix?
Is this bug 767521?
Owner: yueherngl@chromium.org
Status: Assigned
YH please take a look at this issue.
coral-paladin build #1250 and #1251 succeeded.

However build #1252 failed again with the same error.

https://luci-milo.appspot.com/buildbot/chromeos/coral-paladin/1252

Any progress? coral-paladin continues this failure, so it will keep staying in experimental state.
I am increasing the partition size to see if that works. Tryjob is currently progressing. Thanks. 
Trybot build passed that build_image stage without issue. IMHO, should we try it few more times before pushing the change in?

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/paladin/builds/3844

Thanks.

Nice!

Yeah I prefer to try it few more times. Based on the history of coral-paladin (https://uberchromegw.corp.google.com/i/chromeos/builders/coral-paladin), it sometimes passed, sometimes failed. Seems the space it grabs varies.
Sure thing. I will try it few more times and update. Thanks.
The partition size, or the filesystem size?
Yes, that's file system size. I'll revise the CL. Thanks.

The few try job runs are all successful.

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/paladin/builds/3844
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/paladin/builds/3846
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/paladin/builds/3847

On the other hand, coral-paladin seems to be behaving well (the past 9 runs). So I am not sure do we really need this?

Thanks.

Good question, we can hold it until next time the same problem happened. Or just submit in case. @don do you have any preference?
Cc: vapier@chromium.org
Actually, is this the same problem vapier has been addressing?
if you mean the AU/firmware updater stuff, that doesn't apply to build_image

i think we're just running up to the limit again.  we bumped it for reef recently.
Is there any owner for dealing with us approaching to 2G limit?
nope!
The build_image run out of space error also happened in guado_moblab-paladin in recent builds, not sure if it's related to this one.

https://bugs.chromium.org/p/chromium/issues/detail?id=769655

I will push the CL to align coral with reef. Thanks.
Cc: x...@chromium.org ihf@chromium.org
Status: Fixed
RE: #25

Looks like the problem is due to crbug.com/767521, which has been fixed. The increment of file system size for reef has also been reverted. So we should be able to stay the same. Therefore I'm closing the bug. Please re-open if in need.
Thanks.
Sign in to add a comment