New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 686791 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 734764
Owner:
Last visit > 30 days ago
Closed: Jul 2017
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

dut verify fails WritableVerifier but repair dies before repairing it

Project Member Reported by kevcheng@chromium.org, Jan 30 2017

Issue description

chromeos6-row1-rack18-host11 runs out of space and fails verify:
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos6-row1-rack18-host11/562548-verify/20173001084742/

status.log:
START	----	verify	timestamp=1485794888	localtime=Jan 30 08:48:08	
	GOOD	----	verify.ssh	timestamp=1485794891	localtime=Jan 30 08:48:11	
	GOOD	----	verify.update	timestamp=1485794895	localtime=Jan 30 08:48:15	
	GOOD	----	verify.brd_config	timestamp=1485794896	localtime=Jan 30 08:48:16	
	GOOD	----	verify.ser_config	timestamp=1485794896	localtime=Jan 30 08:48:16	
	GOOD	----	verify.job	timestamp=1485794896	localtime=Jan 30 08:48:16	
	GOOD	----	verify.servod	timestamp=1485794900	localtime=Jan 30 08:48:20	
	GOOD	----	verify.pwr_button	timestamp=1485794900	localtime=Jan 30 08:48:20	
	GOOD	----	verify.lid_open	timestamp=1485794900	localtime=Jan 30 08:48:20	
	GOOD	----	verify.PASS	timestamp=1485794900	localtime=Jan 30 08:48:20	
	GOOD	----	verify.ssh	timestamp=1485794902	localtime=Jan 30 08:48:22	
	GOOD	----	verify.power	timestamp=1485794902	localtime=Jan 30 08:48:22	
	GOOD	----	verify.ext4	timestamp=1485794903	localtime=Jan 30 08:48:23	
	FAIL	----	verify.writable	timestamp=1485794903	localtime=Jan 30 08:48:23	Can't create a file in /var/tmp
	GOOD	----	verify.tpm	timestamp=1485794904	localtime=Jan 30 08:48:24	
	GOOD	----	verify.good_au	timestamp=1485794905	localtime=Jan 30 08:48:25	
	GOOD	----	verify.fwstatus	timestamp=1485794905	localtime=Jan 30 08:48:25	
	GOOD	----	verify.rwfw	timestamp=1485794909	localtime=Jan 30 08:48:29	
	GOOD	----	verify.python	timestamp=1485794910	localtime=Jan 30 08:48:30	
	FAIL	----	verify.cros	timestamp=1485794910	localtime=Jan 30 08:48:30	Not enough free space on /usr/local/autotest - 0.000GB free, want 0.700GB
END FAIL	----	verify	timestamp=1485794910	localtime=Jan 30 08:48:30	

When repair comes around, it dies trying to remove a file on /var/...
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos6-row1-rack18-host11/562552-repair/20173001084835/

status.log:
START	----	repair	timestamp=1485794925	localtime=Jan 30 08:48:45	
	GOOD	----	verify.ssh	timestamp=1485794926	localtime=Jan 30 08:48:46	
	GOOD	----	verify.brd_config	timestamp=1485794927	localtime=Jan 30 08:48:47	
	GOOD	----	verify.ser_config	timestamp=1485794927	localtime=Jan 30 08:48:47	
	GOOD	----	verify.job	timestamp=1485794928	localtime=Jan 30 08:48:48	
	GOOD	----	verify.servod	timestamp=1485794931	localtime=Jan 30 08:48:51	
	GOOD	----	verify.pwr_button	timestamp=1485794932	localtime=Jan 30 08:48:52	
	GOOD	----	verify.lid_open	timestamp=1485794932	localtime=Jan 30 08:48:52	
	GOOD	----	verify.update	timestamp=1485794936	localtime=Jan 30 08:48:56	
	GOOD	----	verify.PASS	timestamp=1485794936	localtime=Jan 30 08:48:56	
END FAIL	----	repair	timestamp=1485794942	localtime=Jan 30 08:49:02	

autoserv.DEBUG:
01/30 08:48:56.893 INFO |            repair:0327| Verifying this condition: All host verification checks pass
01/30 08:48:56.893 INFO |        server_job:0183| 	GOOD	----	verify.PASS	timestamp=1485794936	localtime=Jan 30 08:48:56	
01/30 08:48:56.948 DEBUG|          ssh_host:0272| Running (ssh) 'test ! -e /var/log/messages || cp -f /var/log/messages /var/tmp/messages.autotest_start'
01/30 08:48:56.955 INFO |      abstract_ssh:0809| Starting master ssh connection '/usr/bin/ssh -a -x   -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_2DL0tQssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 chromeos6-row1-rack18-host11'
01/30 08:48:56.955 DEBUG|        base_utils:0185| Running '/usr/bin/ssh -a -x   -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_2DL0tQssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 chromeos6-row1-rack18-host11'
01/30 08:48:58.057 ERROR|        base_utils:0280| [stderr] cp: cannot remove '/var/tmp/messages.autotest_start': Read-only file system
01/30 08:48:58.059 INFO |            remote:0073| Failed to copy /var/log/messages at startup: command execution error
* Command: 
    /usr/bin/ssh -a -x    -o ControlPath=/tmp/_autotmp_2DL0tQssh-
    master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
    -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
    ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
    chromeos6-row1-rack18-host11 "export LIBC_FATAL_STDERR_=1; if type
    \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\"
    \"server[stack::create_target_machine|create_host|job_start] ->
    ssh_run(test ! -e /var/log/messages || cp -f /var/log/messages
    /var/tmp/messages.autotest_start)\";fi; test ! -e /var/log/messages || cp
    -f /var/log/messages /var/tmp/messages.autotest_start"
Exit status: 1
Duration: 0.20854306221

stderr:
cp: cannot remove '/var/tmp/messages.autotest_start': Read-only file system
 
Mergedinto: 734764
Status: Duplicate (was: Untriaged)

Sign in to add a comment