New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 696772 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug

Blocking:
issue 695529



Sign in to add a comment

ssh failure after initial reboot

Project Member Reported by akes...@chromium.org, Feb 27 2017

Issue description

Build:
https://luci-milo.appspot.com/buildbot/chromeos/veyron_speedy-paladin/4498

Test:
http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=103483372

Provision/repair:
http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row4-rack11-host6/2698350-repair/
http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row4-rack11-host6/2698091-provision/

Relevant provision snippet:
2017/02/25 22:42:23.930 DEBUG|      auto_updater:1055| Start pre-setup for rootfs update...
2017/02/25 22:42:23.930 INFO |     remote_access:0395| Rebooting chromeos4-row4-rack11-host6...
2017/02/25 22:42:23.931 DEBUG|    cros_build_lib:0564| RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmp_3wqey/testing_rsa root@chromeos4-row4-rack11-host6 -- cat /proc/sys/kernel/random/boot_id
2017/02/25 22:42:24.227 DEBUG|    cros_build_lib:0613| (stdout):
19e2dda6-b81b-4903-b518-6d520aff9652

...
2017/02/25 22:50:29.419 ERROR|    cros_build_lib:0661| Reboot has not completed after 480 seconds; giving up.
2017/02/25 22:50:29.420 DEBUG|    cros_build_lib:0564| RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmp_3wqey/testing_rsa root@chromeos4-row4-rack11-host6 -- rm -rf /mnt/stateful_partition/unencrypted/preserve/cros-update/tmp.hyPyfniyLG
2017/02/25 22:52:32.598 ERROR|     remote_access:0856| Error connecting to device chromeos4-row4-rack11-host6
2017/02/25 22:52:32.598 DEBUG|       cros_update:0253| Error happens in CrOS auto-update: SSHConnectionError('ssh: connect to host chromeos4-row4-rack11-host6 port 22: Connection timed out\r\n',)


repair status log:
START	----	repair	timestamp=1488092720	localtime=Feb 25 23:05:20	
	GOOD	----	verify.ssh	timestamp=1488092722	localtime=Feb 25 23:05:22	
	GOOD	----	verify.brd_config	timestamp=1488092722	localtime=Feb 25 23:05:22	
	GOOD	----	verify.ser_config	timestamp=1488092723	localtime=Feb 25 23:05:23	
	GOOD	----	verify.job	timestamp=1488092724	localtime=Feb 25 23:05:24	
	GOOD	----	verify.servod	timestamp=1488092730	localtime=Feb 25 23:05:30	
	GOOD	----	verify.pwr_button	timestamp=1488092730	localtime=Feb 25 23:05:30	
	GOOD	----	verify.lid_open	timestamp=1488092730	localtime=Feb 25 23:05:30	
	GOOD	----	verify.update	timestamp=1488092733	localtime=Feb 25 23:05:33	
	GOOD	----	verify.PASS	timestamp=1488092733	localtime=Feb 25 23:05:33	
	GOOD	----	verify.ssh	timestamp=1488092744	localtime=Feb 25 23:05:44	
	GOOD	----	verify.power	timestamp=1488092744	localtime=Feb 25 23:05:44	
	GOOD	----	verify.fwstatus	timestamp=1488092744	localtime=Feb 25 23:05:44	
	FAIL	----	verify.good_au	timestamp=1488092745	localtime=Feb 25 23:05:45	Last AU on this DUT failed
	START	----	repair.firmware	timestamp=1488092745	localtime=Feb 25 23:05:45	
		FAIL	----	repair.firmware	timestamp=1488092745	localtime=Feb 25 23:05:45	Firmware repair is not applicable to host chromeos4-row4-rack11-host6.
	END FAIL	----	repair.firmware	timestamp=1488092745	localtime=Feb 25 23:05:45	
	GOOD	----	verify.writable	timestamp=1488092746	localtime=Feb 25 23:05:46	
	GOOD	----	verify.tpm	timestamp=1488092747	localtime=Feb 25 23:05:47	
	GOOD	----	verify.ext4	timestamp=1488092748	localtime=Feb 25 23:05:48	
	GOOD	----	verify.rwfw	timestamp=1488092749	localtime=Feb 25 23:05:49	
	GOOD	----	verify.python	timestamp=1488092749	localtime=Feb 25 23:05:49	
	GOOD	----	verify.cros	timestamp=1488092755	localtime=Feb 25 23:05:55	
	START	----	repair.powerwash	timestamp=1488092755	localtime=Feb 25 23:05:55	
		START	----	reboot	timestamp=1488092756	localtime=Feb 25 23:05:56	
			GOOD	----	reboot.start	timestamp=1488092756	localtime=Feb 25 23:05:56	
			GOOD	----	reboot.verify	timestamp=1488092816	localtime=Feb 25 23:06:56	
		END GOOD	----	reboot	kernel=3.14.0	localtime=Feb 25 23:06:57	timestamp=1488092817	
		GOOD	----	verify.ssh	timestamp=1488093618	localtime=Feb 25 23:20:18	
		GOOD	----	verify.tpm	timestamp=1488093619	localtime=Feb 25 23:20:19	
		GOOD	----	verify.good_au	timestamp=1488093620	localtime=Feb 25 23:20:20	
		GOOD	----	verify.ext4	timestamp=1488093621	localtime=Feb 25 23:20:21	
		GOOD	----	verify.power	timestamp=1488093621	localtime=Feb 25 23:20:21	
		GOOD	----	verify.rwfw	timestamp=1488093622	localtime=Feb 25 23:20:22	
		GOOD	----	verify.python	timestamp=1488093622	localtime=Feb 25 23:20:22	
		GOOD	----	verify.cros	timestamp=1488093628	localtime=Feb 25 23:20:28	
	END GOOD	----	repair.powerwash	timestamp=1488093628	localtime=Feb 25 23:20:28	
	GOOD	----	verify.PASS	timestamp=1488093628	localtime=Feb 25 23:20:28	
	INFO	----	repair	timestamp=1488093628	localtime=Feb 25 23:20:28	Can't repair label 'pool:cq'.
	INFO	----	repair	timestamp=1488093628	localtime=Feb 25 23:20:28	Can't repair label 'cleanup-reboot'.
	INFO	----	repair	timestamp=1488093628	localtime=Feb 25 23:20:28	Can't repair label 'cros-version:veyron_speedy-paladin/R58-9318.0.0-rc2'.
	INFO	----	repair	timestamp=1488093628	localtime=Feb 25 23:20:28	Can't repair label 'board:veyron_speedy'.
END GOOD	----	repair	timestamp=1488093628	localtime=Feb 25 23:20:28	chromeos4-row4-rack11-host6 repaired successfully


Basic understanding "Failed at the initialization reboot"
 
Note: the failure of repair.firmware is a red herring, benign and unrelated to whatever happened here.
Blocking: 695529

Comment 3 by aut...@google.com, Mar 1 2017

Cc: dgarr...@chromium.org
+ dgarrett FYI 
Labels: -current-issue
Owner: xixuan@chromium.org
+ xixuan to find the dupe bug for this
Status: Assigned (was: Untriaged)
This bug is Untriaged and very old.  Because it has an owner, the status will be set to assigned to avoid closing a bug someone is using.  If this bug still needs triage, change it back to Untriaged.

Comment 6 by xixuan@chromium.org, Mar 19 2018

Status: WontFix (was: Assigned)

Sign in to add a comment