New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 1 user

Issue metadata

Status: Fixed
Closed: Oct 2017
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Sign in to add a comment

build_image run out of space

Project Member Reported by, Sep 25 2017

Issue description

Builder coral-paladin Build #1242

Started virtual/target-os-1.5-r1 (logged in /tmp/target-os-1.5-r1-71sU88)
Completed virtual/target-os-1.5-r1 (in 0m2.2s)
Pending 0/400, [Time 22:00:07 | Elapsed 1m55.8s | Load 14.78 11.09 11.3]
Merge complete
INFO    : Generating license credits page. Time:
licenses: Unhandled exception:
Traceback (most recent call last):
  File "/mnt/host/source/chromite/licensing/licenses", line 168, in <module>
  File "/mnt/host/source/chromite/licensing/licenses", line 164, in DoMain
  File "/mnt/host/source/chromite/lib/", line 910, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/mnt/host/source/chromite/licensing/", line 223, in main
  File "/mnt/host/source/chromite/licensing/", line 1056, in GenerateHTMLLicenseOutput
    self.EvaluateTemplate(file_template, env).encode('UTF-8'))
  File "/mnt/host/source/chromite/lib/", line 130, in WriteFile
IOError: [Errno 28] No space left on device

ERROR   : Here are the biggest [partially-]extracted files (by disk usage):
1388544 lib/firmware/iwlwifi-7265D-14.ucode
1392640 lib/firmware/iwlwifi-7265D-16.ucode
1392640 lib/firmware/iwlwifi-7265D-21.ucode
1396736 opt/google/chrome/keyboard_resources.pak
1413120 opt/google/chrome/resources/inspector/audits2_worker/audits2_worker_module.js
1417216 usr/share/cups/ipptool/document-letter.pdf
1429504 usr/lib64/
1445888 usr/lib64/
1474560 opt/google/chrome/resources/about_os_credits.html
1474560 usr/share/cups/ipptool/
1478656 usr/bin/core_collector
1486848 usr/bin/core_collector32
1511424 usr/lib64/
1515520 usr/share/chromeos-assets/easy_unlock/easy_unlock_background.js
1540096 usr/share/chromeos-assets/easy_unlock/easy_unlock_pairing.js
1581056 opt/google/chrome/resources/chromeos/chromevox/braille/liblouis_nacl_x86_64.nexe
1654784 opt/google/chrome/resources/inspector/inspector.js
1667072 opt/google/chrome/resources/inspector/unit_test_runner.js
1671168 usr/sbin/flashrom
1675264 usr/lib64/locale/locale-archive
1687552 opt/google/chrome/resources/inspector/integration_test_runner.js
1708032 usr/bin/perf
1708032 usr/bin/trace
1740800 lib64/
1908736 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_sz_nexe
2154496 usr/lib64/
2170880 opt/google/chrome/pnacl/pnacl_public_x86_64_ld_nexe
2207744 usr/lib64/
2260992 usr/lib64/
2273280 usr/share/fonts/tibt-jomolhari/Jomolhari-alpha3c-0605331.ttf
2334720 usr/share/chromeos-assets/input_methods/input_tools/en_gesture/gesture_lexicon.bin
2461696 opt/google/chrome/nacl_helper
2650112 usr/lib64/
2674688 usr/share/chromeos-assets/input_methods/input_tools/wubi/system_dictionary.bin
2818048 usr/lib64/
2908160 usr/lib64/samba/
3043328 usr/bin/superiotool
3088384 usr/lib64/
3547136 opt/google/chrome/
3608576 opt/google/chrome/nacl_helper_nonsfi
3608576 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/system_dictionary.bin
3743744 opt/google/chrome/nacl_irt_x86_64.nexe
3985408 usr/share/chromeos-assets/input_methods/input_tools/zhuyin/bigram.bin
4100096 usr/share/chromeos-assets/quickoffice/scripts/mainQOWT.js
4210688 usr/share/chromeos-assets/input_methods/input_tools/en/system_dictionary.bin
4263936 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_pt-BR.zvoice
4300800 usr/share/fonts/ko-nanum/NanumGothicBold.ttf
4321280 usr/lib64/va/drivers/
4358144 usr/share/fonts/ko-nanum/NanumGothic.ttf
4489216 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-GB.zvoice
4640768 usr/share/chromeos-assets/input_methods/input_tools/pinyin/system_dictionary.bin
4734976 opt/google/chrome/chrome_100_percent.pak
4763648 usr/share/chromeos-assets/input_methods/input_tools/ko/system_hanja_dictionary.bin
4861952 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-IN.zvoice
5013504 usr/bin/shill
5128192 usr/share/chromeos-assets/input_methods/input_tools/nacl_session_handler_x86_64.nexe
5414912 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-US.zvoice
5713920 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/word_x86_64.nexe
6053888 boot/vmlinuz-4.4.86-11736-g62e8131a1278
6062080 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_ko-KR.zvoice
6160384 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_en-US.zvoice
6643712 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_nl-NL.zvoice
6938624 usr/share/chromeos-assets/input_methods/input_tools/pinyin/bigram.bin
7114752 opt/google/containers/android/vendor/lib/dri/
7266304 usr/share/fonts/noto/NotoColorEmoji.ttf
7323648 opt/google/chrome/
7401472 opt/displaylink/DisplayLinkManager
7417856 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_it-IT.zvoice
7430144 usr/lib64/dri/
7512064 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_id-ID.zvoice
8740864 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_de-DE.zvoice
8871936 opt/google/containers/android/vendor/lib/dri/
9195520 usr/share/chromeos-assets/input_methods/input_tools/zipped_data_chromeos
9351168 usr/sbin/chromeos-firmwareupdate
9973760 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_fr-FR.zvoice
9981952 usr/share/chromeos-assets/input_methods/input_tools/decoder_x86_64.nexe
10215424 opt/google/chrome/icudtl.dat
10272768 usr/share/chromeos-assets/demo_app/screensaver-video.webm
10371072 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/sheet_x86_64.nexe
10698752 opt/google/chrome/chrome_200_percent.pak
11603968 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_hi-IN.zvoice
12197888 usr/lib64/
12505088 usr/share/chromeos-assets/speech_synthesis/patts/voice_lstm_es-ES.zvoice
13234176 usr/share/chromeos-assets/input_methods/input_tools/en/bigram.bin
14114816 opt/google/chrome/pnacl/pnacl_public_x86_64_pnacl_llc_nexe
16338944 usr/share/chromeos-assets/quickoffice/_platform_specific/x86_64/point_x86_64.nexe
17534976 usr/share/fonts/notocjk/NotoSansCJK-Light.ttc
18776064 usr/share/fonts/notocjk/NotoSansCJK-Regular.ttc
19345408 usr/share/fonts/notocjk/NotoSansCJK-Bold.ttc
21925888 usr/lib64/dri/
21925888 usr/lib64/dri/
24686592 usr/share/fonts/notocjk/NotoSerifCJK-Light.ttc
24842240 usr/share/fonts/notocjk/NotoSerifCJK-Regular.ttc
25915392 usr/share/fonts/notocjk/NotoSerifCJK-Bold.ttc
27471872 opt/google/chrome/resources.pak
29728768 usr/share/chromeos-assets/speech_synthesis/patts/tts_service_x86-64.nexe
32276480 opt/google/chrome/pepper/
46010368 opt/google/containers/android/vendor.raw.img
138694656 opt/google/chrome/chrome
451137536 opt/google/containers/android/system.raw.img
ERROR   : Target image has run out of space:
ERROR   : Filesystem     1M-blocks  Used Available Use% Mounted on
ERROR   : /dev/loop0          1723  1723         0 100% /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2/rootfs
INFO    : Unmounting image from /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2/stateful and /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2/rootfs
Cleaning up /usr/local symlinks for /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2/stateful/dev_image
An error occurred in your build so your latest output directory is invalid.
Running in non-interactive mode so deleting output directory.
Deleted /mnt/host/source/src/build/images/coral/R63-9972.0.0-rc2

The CLs in this CQ build do not look like the cause of the out of space failure.


Comment 1 by, Sep 25 2017

@vbendeb I guess 2 possibilities 1) the VM has some issues. 2) bad CL/bug in ToT.

No common CLs in these failed builds, the only CL in last successful coral build is:

Will this CL lead to the big occupation of disk of builder? 

Comment 2 by, Sep 25 2017

"No common CLs in these failed builds" - what builds are these?

Comment 3 by, Sep 25 2017

1) respawninstance the GCE: cros-beefy393-c2
2) Mark coral-paladin as experimental 

Waiting for next CQ round to verify whether respawn works.

Re #2, these failed builds = all failed coral-paladin due to 'out of spaces' issue.

Comment 4 by, Sep 25 2017

Coral-paladin still failed:

ERROR   : Target image has run out of space:
ERROR   : Filesystem     1M-blocks  Used Available Use% Mounted on
ERROR   : /dev/loop0          1723  1723         0 100% /mnt/host/source/src/build/images/coral/R63-9973.0.0-rc1/rootfs
INFO    : Unmounting image from /mnt/host/source/src/build/images/coral/R63-9973.0.0-rc1/stateful and /mnt/host/source/src/build/images/coral/R63-9973.0.0-rc1/rootfs
Cleaning up /usr/local symlinks for /mnt/host/source/src/build/images/coral/R63-9973.0.0-rc1/stateful/dev_image
An error occurred in your build so your latest output directory is invalid.

What's this 1M blocks?

Comment 5 by, Sep 25 2017

Labels: -Pri-1 Pri-0
Labels: -Pri-0 Pri-1
Oops, it's already experimental. Downgrading to P1

Comment 8 by, Sep 25 2017

Surprisingly this round of coral-paladin passed build_packages:

We have put in any fix?
Is this bug 767521?
Status: Assigned (was: Untriaged)
YH please take a look at this issue.
coral-paladin build #1250 and #1251 succeeded.

However build #1252 failed again with the same error.

Any progress? coral-paladin continues this failure, so it will keep staying in experimental state.
I am increasing the partition size to see if that works. Tryjob is currently progressing. Thanks. 
Trybot build passed that build_image stage without issue. IMHO, should we try it few more times before pushing the change in?



Yeah I prefer to try it few more times. Based on the history of coral-paladin (, it sometimes passed, sometimes failed. Seems the space it grabs varies.
Sure thing. I will try it few more times and update. Thanks.
The partition size, or the filesystem size?
Yes, that's file system size. I'll revise the CL. Thanks.

The few try job runs are all successful.

On the other hand, coral-paladin seems to be behaving well (the past 9 runs). So I am not sure do we really need this?


Good question, we can hold it until next time the same problem happened. Or just submit in case. @don do you have any preference?
Actually, is this the same problem vapier has been addressing?
if you mean the AU/firmware updater stuff, that doesn't apply to build_image

i think we're just running up to the limit again.  we bumped it for reef recently.
Is there any owner for dealing with us approaching to 2G limit?
The build_image run out of space error also happened in guado_moblab-paladin in recent builds, not sure if it's related to this one.

I will push the CL to align coral with reef. Thanks.

Comment 26 by, Oct 1 2017

Status: Fixed (was: Assigned)
RE: #25

Looks like the problem is due to, which has been fixed. The increment of file system size for reef has also been reverted. So we should be able to stay the same. Therefore I'm closing the bug. Please re-open if in need.
Status: Archived (was: Fixed)
Status: Fixed (was: Archived)

Sign in to add a comment