New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 695627 link

Starred by 2 users

Issue metadata

Status: Duplicate
Merged: issue 697967
Owner:
Last visit > 30 days ago
Closed: Mar 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

amd64-generic-asan builders failing in BuildImage due to lack of space.

Project Member Reported by cmt...@chromium.org, Feb 23 2017

Issue description

The amd64-generic-asan builder, on the external waterfall (https://uberchromegw.corp.google.com/i/chromiumos/waterfall) started consistently failing in BuildImage yesterday.  When I look at the error logs, I see

chromeos-firewall-1-r1: IOError: [Errno 28] No space left on device

and (at the end):

ERROR   : Target image has run out of space:
ERROR   : Filesystem     1M-blocks  Used Available Use% Mounted on
ERROR   : /dev/loop0          1969  1969         0 100% /mnt/host/source/src/build/images/amd64-generic/R58-9311.0.2017_02_23_1227-a1/rootfs
INFO    : Unmounting image from /mnt/host/source/src/build/images/amd64-generic/R58-9311.0.2017_02_23_1227-a1/stateful and /mnt/host/source/src/build/images/amd64-generic/R58-9311.0.2017_02_23_1227-a1/rootfs
Cleaning up /usr/local symlinks for /mnt/host/source/src/build/images/amd64-generic/R58-9311.0.2017_02_23_1227-a1/stateful/dev_image
An error occurred in your build so your latest output directory is invalid.
Running in non-interactive mode so deleting output directory.
Deleted /mnt/host/source/src/build/images/amd64-generic/R58-9311.0.2017_02_23_1227-a1
12:30:02: ERROR: 
return code: 1; command: /b/cbuild/external_master/chromite/bin/cros_sdk 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpoEZD6b' 'FEATURES=separatedebug' 'CHROME_ORIGIN=LOCAL_SOURCE' -- ./build_image '--board=amd64-generic' --replace '--version=' '--builder_path=amd64-generic-asan/R58-9311.0.0-b17767' '--disk_layout=2gb-rootfs' test
cmd=['/b/cbuild/external_master/chromite/bin/cros_sdk', 'PARALLEL_EMERGE_STATUS_FILE=/tmp/tmpoEZD6b', 'FEATURES=separatedebug', 'CHROME_ORIGIN=LOCAL_SOURCE', '--', './build_image', '--board=amd64-generic', '--replace', '--version=', '--builder_path=amd64-generic-asan/R58-9311.0.0-b17767', '--disk_layout=2gb-rootfs', 'test'], cwd=/b/cbuild/external_master, extra env={'PARALLEL_EMERGE_STATUS_FILE': '/tmp/tmpoEZD6b', 'FEATURES': 'separatedebug', 'CHROME_ORIGIN': 'LOCAL_SOURCE'}


This probably means the rootfs file system on the image needs to be bigger, but it needs to be investigated.

I'm not sure who the correct owner for this is, so I'm filing it against the current sheriffs.

 

Comment 1 by sha...@chromium.org, Feb 23 2017

Cc: -llozano@chromium.org dmazz...@chromium.org
It looks like our rootfs size grew suddenly between .7 and .8:

https://chromeperf.appspot.com/report?sid=422068e0adb64425b2320337d23c7bb5202c4421147e3c6eaa33011defcd619c

This CL is the likely cause:

https://chromium-review.googlesource.com/#/c/443133/

That looks important, so we probably do want to bump the size of the partition.

Comment 2 by sha...@chromium.org, Feb 23 2017

Cc: llozano@chromium.org
Cc: za...@chromium.org littlecvr@chromium.org leecy@chromium.org scollyer@chromium.org
Components: Infra>Client>ChromeOS
Owner: leecy@chromium.org
Assigning to current sherrif(s) to find an owner.

Comment 4 by ihf@google.com, Mar 13 2017

Mergedinto: 697967
Status: Duplicate (was: Untriaged)

Sign in to add a comment