chromeos6-row1-rack18-host11 runs out of space and fails verify:
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos6-row1-rack18-host11/562548-verify/20173001084742/
status.log:
START ---- verify timestamp=1485794888 localtime=Jan 30 08:48:08
GOOD ---- verify.ssh timestamp=1485794891 localtime=Jan 30 08:48:11
GOOD ---- verify.update timestamp=1485794895 localtime=Jan 30 08:48:15
GOOD ---- verify.brd_config timestamp=1485794896 localtime=Jan 30 08:48:16
GOOD ---- verify.ser_config timestamp=1485794896 localtime=Jan 30 08:48:16
GOOD ---- verify.job timestamp=1485794896 localtime=Jan 30 08:48:16
GOOD ---- verify.servod timestamp=1485794900 localtime=Jan 30 08:48:20
GOOD ---- verify.pwr_button timestamp=1485794900 localtime=Jan 30 08:48:20
GOOD ---- verify.lid_open timestamp=1485794900 localtime=Jan 30 08:48:20
GOOD ---- verify.PASS timestamp=1485794900 localtime=Jan 30 08:48:20
GOOD ---- verify.ssh timestamp=1485794902 localtime=Jan 30 08:48:22
GOOD ---- verify.power timestamp=1485794902 localtime=Jan 30 08:48:22
GOOD ---- verify.ext4 timestamp=1485794903 localtime=Jan 30 08:48:23
FAIL ---- verify.writable timestamp=1485794903 localtime=Jan 30 08:48:23 Can't create a file in /var/tmp
GOOD ---- verify.tpm timestamp=1485794904 localtime=Jan 30 08:48:24
GOOD ---- verify.good_au timestamp=1485794905 localtime=Jan 30 08:48:25
GOOD ---- verify.fwstatus timestamp=1485794905 localtime=Jan 30 08:48:25
GOOD ---- verify.rwfw timestamp=1485794909 localtime=Jan 30 08:48:29
GOOD ---- verify.python timestamp=1485794910 localtime=Jan 30 08:48:30
FAIL ---- verify.cros timestamp=1485794910 localtime=Jan 30 08:48:30 Not enough free space on /usr/local/autotest - 0.000GB free, want 0.700GB
END FAIL ---- verify timestamp=1485794910 localtime=Jan 30 08:48:30
When repair comes around, it dies trying to remove a file on /var/...
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos6-row1-rack18-host11/562552-repair/20173001084835/
status.log:
START ---- repair timestamp=1485794925 localtime=Jan 30 08:48:45
GOOD ---- verify.ssh timestamp=1485794926 localtime=Jan 30 08:48:46
GOOD ---- verify.brd_config timestamp=1485794927 localtime=Jan 30 08:48:47
GOOD ---- verify.ser_config timestamp=1485794927 localtime=Jan 30 08:48:47
GOOD ---- verify.job timestamp=1485794928 localtime=Jan 30 08:48:48
GOOD ---- verify.servod timestamp=1485794931 localtime=Jan 30 08:48:51
GOOD ---- verify.pwr_button timestamp=1485794932 localtime=Jan 30 08:48:52
GOOD ---- verify.lid_open timestamp=1485794932 localtime=Jan 30 08:48:52
GOOD ---- verify.update timestamp=1485794936 localtime=Jan 30 08:48:56
GOOD ---- verify.PASS timestamp=1485794936 localtime=Jan 30 08:48:56
END FAIL ---- repair timestamp=1485794942 localtime=Jan 30 08:49:02
autoserv.DEBUG:
01/30 08:48:56.893 INFO | repair:0327| Verifying this condition: All host verification checks pass
01/30 08:48:56.893 INFO | server_job:0183| GOOD ---- verify.PASS timestamp=1485794936 localtime=Jan 30 08:48:56
01/30 08:48:56.948 DEBUG| ssh_host:0272| Running (ssh) 'test ! -e /var/log/messages || cp -f /var/log/messages /var/tmp/messages.autotest_start'
01/30 08:48:56.955 INFO | abstract_ssh:0809| Starting master ssh connection '/usr/bin/ssh -a -x -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_2DL0tQssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 chromeos6-row1-rack18-host11'
01/30 08:48:56.955 DEBUG| base_utils:0185| Running '/usr/bin/ssh -a -x -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_2DL0tQssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 chromeos6-row1-rack18-host11'
01/30 08:48:58.057 ERROR| base_utils:0280| [stderr] cp: cannot remove '/var/tmp/messages.autotest_start': Read-only file system
01/30 08:48:58.059 INFO | remote:0073| Failed to copy /var/log/messages at startup: command execution error
* Command:
/usr/bin/ssh -a -x -o ControlPath=/tmp/_autotmp_2DL0tQssh-
master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
-o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
chromeos6-row1-rack18-host11 "export LIBC_FATAL_STDERR_=1; if type
\"logger\" > /dev/null 2>&1; then logger -tag \"autotest\"
\"server[stack::create_target_machine|create_host|job_start] ->
ssh_run(test ! -e /var/log/messages || cp -f /var/log/messages
/var/tmp/messages.autotest_start)\";fi; test ! -e /var/log/messages || cp
-f /var/log/messages /var/tmp/messages.autotest_start"
Exit status: 1
Duration: 0.20854306221
stderr:
cp: cannot remove '/var/tmp/messages.autotest_start': Read-only file system
Comment 1 by jrbarnette@chromium.org
, Jul 21 2017Status: Duplicate (was: Untriaged)