New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 787476 link

Starred by 2 users
Status: Archived
Owner:
█ nxia@chromium.org
Last visit > 30 days ago
Closed: Feb 2018
Cc:
█ dshi@chromium.org
akes...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

shard chromeos-server118.mtv is down (again) due to lack of disk (again)

Project Member Reported by akes...@chromium.org, Nov 21 2017 
https://viceroy.corp.google.com/chromeos/deputy-view?utc_end=1511278390#_VG_Qa50v5OG

Several alerts firing.

chromeos-test@chromeos-server118$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             60G   12K   60G   1% /dev
tmpfs            12G  768K   12G   1% /run
/dev/xvda1       59G   56G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none             60G  2.2M   60G   1% /run/shm
none            100M     0  100M   0% /run/user
/dev/xvda3      2.0T   91G  1.8T   5% /usr/local/google
tmpfs           100K     0  100K   0% /var/lib/lxd/shmounts
tmpfs           100K     0  100K   0% /var/lib/lxd/devlxd
srcfsd          1.0T     0  1.0T   0% /google/src
objfsd          2.0T   91G  1.8T   5% /google/obj


Is our partition layout strange on this machine? Why do we keep running out of space on it? We seem to be running on the small /dev/xvda1 partition.

Comment 1 by akes...@chromium.org, Nov 21 2017 

For comparison, on server98 we have a much larger root partition

chromeos-test@chromeos-server98:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             60G   12K   60G   1% /dev
tmpfs            12G  518M   12G   5% /run
/dev/xvda1      2.1T  183G  1.8T  10% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none             60G   19M   60G   1% /run/shm
none            100M     0  100M   0% /run/user
tmpfs           100K     0  100K   0% /var/lib/lxd/shmounts
tmpfs           100K     0  100K   0% /var/lib/lxd/devlxd

Comment 2 by akes...@chromium.org, Nov 21 2017

Status: Assigned (was: Untriaged)
I deleted containers/ logs/ and results/ and rebooted this shard. But really, we should turn down this shard (migrate its boards elsewhere) and reparatition it.

Comment 3 by nxia@chromium.org, Nov 21 2017

Cc: akes...@chromium.org
we have chromeos-server119.mtv.corp.google.com available, I can migrate boards to 119 ?

Comment 4 by nxia@chromium.org, Nov 21 2017

Status: Started (was: Assigned)
start migrating " board:reef, board:pyro, board:ultima" to  chromeos-server119.mtv.corp.google.com

Comment 5 by nxia@chromium.org, Nov 28 2017 

going to remove 118 from prod and re-provision it

Comment 6 by nxia@chromium.org, Feb 15 2018

Status: Archived (was: Started)
chromeos-server118 is deprecated
►

Sign in to add a comment