Check whether there're more devserver out of disk space |
||||||||||
Issue descriptionSomething is going with this devserver and it is almost full... chromeos-test@chromeos4-devserver2:~/images$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/chromeos4--devserver2--vg-root 5.4T 5.1T 15G 100% / If this is the new normal then I'm going to need to order more equipment.
,
Feb 27 2017
I'll tack on I do not see it in viceroy as well https://viceroy.corp.google.com/chromeos/machines?hostname=chromeos4-devserver2&duration=30d&refresh=-1
,
Feb 27 2017
this devserver should be good now. I will change the intent of this bug to 'check whether we still have out-of-disk-space devservers'.
,
Feb 28 2017
Based on all devserver's disk usage and my checking on the top-N devservers, I believe there's no devserver suffering full disk problem. 172.25.65.217 usage is: 0.2 100.115.219.133 usage is: 0.16 100.115.219.134 usage is: 0.15 100.115.219.132 usage is: 0.14 100.115.219.131 usage is: 0.14 100.115.219.130 usage is: 0.03 172.24.184.161 usage is: 0.01 100.107.126.160 usage is: 0.01 172.27.215.252 usage is: 0.01 100.115.219.136 usage is: 0.01 100.115.219.135 usage is: 0.01 100.115.185.228 usage is: 0.01 172.22.39.163 usage is: 0.01 100.107.126.164 usage is: 0.01 100.107.126.165 usage is: 0.01 172.24.184.160 usage is: 0.01 100.120.7.236 usage is: 0.01 100.115.185.227 usage is: 0.01 100.115.185.226 usage is: 0.01 100.107.126.162 usage is: 0.01 100.107.126.163 usage is: 0.01 100.115.99.246 usage is: 0.01 100.115.99.247 usage is: 0.01 100.115.245.198 usage is: 0.01 100.115.245.197 usage is: 0.01 100.115.99.249 usage is: 0.01 100.107.225.252 usage is: 0.01 172.25.65.235 usage is: 0.01 172.27.215.245 usage is: 0.01 100.107.227.251 usage is: 0.01 100.107.227.252 usage is: 0.01 172.27.215.246 usage is: 0.01 100.115.219.129 usage is: 0.01 172.27.215.249 usage is: 0.01 100.115.99.236 usage is: 0.01 100.107.126.159 usage is: 0.01 172.25.65.106 usage is: 0.01 100.115.24.253 usage is: 0.01 100.107.126.137 usage is: 0.01 100.107.126.136 usage is: 0.01 100.107.225.251 usage is: 0.01 100.115.245.200 usage is: 0.01 172.22.39.162 usage is: 0.01 100.120.7.235 usage is: 0.01 172.22.39.161 usage is: 0.01 100.115.99.251 usage is: 0.01 100.115.99.250 usage is: 0.01 172.22.39.164 usage is: 0.01 100.115.99.252 usage is: 0.01 Furthermore I find a list of devservers which cannot be sshed: '100.107.127.227', '100.107.127.228', '100.107.127.229', '172.24.190.169', '172.27.215.248'], one of which is involved in apache's error log: Issue 696783. Could I have englab to check whether we can remove these devservers from prod?
,
Feb 28 2017
englab-sys-cros' devservers are start with 100.115.x.x so those might be under the control of dschimmel's team
,
Feb 28 2017
Hi, You may remove 100.107.127.227, 100.107.127.228, 100.107.127.229 and 172.27.215.248. I do not know this one 172.24.190.169 Thanks David
,
Feb 28 2017
172.24.190.169 = chromeos-server59.hot.corp.google.com. I'm assuming there was an experiment using ganeti instances as devservers....
,
Feb 28 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/a61a765bf01230409dfece7e3e6903fad75466aa commit a61a765bf01230409dfece7e3e6903fad75466aa Author: xixuan <xixuan@chromium.org> Date: Tue Feb 28 07:07:58 2017
,
Feb 28 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/a61a765bf01230409dfece7e3e6903fad75466aa commit a61a765bf01230409dfece7e3e6903fad75466aa Author: xixuan <xixuan@chromium.org> Date: Tue Feb 28 07:07:58 2017
,
Feb 28 2017
,
Feb 28 2017
This is a good candidate for an alert. We want the alert only for in-lab devservers (IP address starts with 100 rather than 172).
,
Mar 1 2017
Hi, @dschimmels, is 100.115.24.253 still used?
,
Mar 1 2017
To clarify my 100.115.x.x statement 2081 uses 100.115.128.1 to 100.115.255.254 2081 Devservers specifically use 100.115.185.225 - 100.115.185.254 100.115.219.129 - 100.115.219.158 100.115.245.193 - 100.115.245.222
,
Mar 1 2017
I will remove 100.115.24.253 if no one have comments in 10 minutes... since this devserver is not well configured (need password to ssh in devserver push). I assume it cannot be used in such configuration.
,
Mar 1 2017
100.115.24.253 is vrlab-autotest-dev1.mtv Nicolas can you help these guys figure out why your devserver requires a password?
,
Mar 1 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/c3f9c12164e62775421bdfac513e70099a2da99d commit c3f9c12164e62775421bdfac513e70099a2da99d Author: Dan Shi <dshi@google.com> Date: Wed Mar 01 19:54:15 2017
,
Mar 1 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/c3f9c12164e62775421bdfac513e70099a2da99d commit c3f9c12164e62775421bdfac513e70099a2da99d Author: Dan Shi <dshi@google.com> Date: Wed Mar 01 19:54:15 2017
,
Mar 1 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/fd4f4cda16007548afb231f9d88a00c82b4b2251 commit fd4f4cda16007548afb231f9d88a00c82b4b2251 Author: xixuan <xixuan@chromium.org> Date: Wed Mar 01 20:05:20 2017
,
Mar 1 2017
Update: 100.115.24.253 is set back to use port 8082 (its 8080 is still listening by redirection), sysmon is running again, and puppet update is set.
,
Apr 3 2017
,
May 30 2017
,
Aug 1 2017
,
Jan 22 2018
|
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by xixuan@chromium.org
, Feb 27 2017