New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 910159 link

Starred by 2 users
Status: WontFix
Owner:
ayatane@chromium.org
Closed: Nov 29
Cc:
█ sonnysasaka@chromium.org
█ djkurtz@chromium.org
█ martinroth@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

grunt: Provision failed after firmware update

Project Member Reported by dlaurie@google.com, Nov 29 
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8928519857375806768

The logs appear to show that the firmware update completed successfully but then it failed to respond to SSH.

I can't seem to get to deeper logs that would actually show the firmware updater running, perhaps because the host did not come back up.

provision     FAIL: DUT firmware requires update from Google_Grunt.11031.34.0 to Google_Grunt.11031.36.0, completed successfully
provision     FAIL: Error on chromeos2-row3-rack9-host13 prior to update: No answer to ssh

Comment 1 by mikenichols@chromium.org, Nov 29

Components: -Infra>Client>ChromeOS>CI Infra>Client>ChromeOS>Test

Comment 2 by jclinton@chromium.org, Nov 29

Cc: djkurtz@chromium.org
Owner: ayatane@chromium.org
Status: Assigned (was: Untriaged)
Allen, can you maybe see if we have more details about what happened here? It looks like it's currently marked ready: http://cautotest-prod/afe/#tab_id=view_host&object_id=4736 so it's not like this firmware update bricked the device. Do we need to allow more time for Grunt devices? Or, if this looks like a platform problem, please assign to djkurtz@ to find a Grunt platform owner.

Comment 3 by ayatane@chromium.org, Nov 29 

START	----	provision	timestamp=1543504355	localtime=Nov 29 07:12:35	
	INFO	----	----	timestamp=1543504357	localtime=Nov 29 07:12:37	Beginning verify for servo host chromeos2-row3-rack9-host1-servo port 9999 serial None
	GOOD	----	verify.servo_ssh	timestamp=1543504359	localtime=Nov 29 07:12:39	
	GOOD	----	verify.update	timestamp=1543504363	localtime=Nov 29 07:12:43	
	GOOD	----	verify.brd_config	timestamp=1543504364	localtime=Nov 29 07:12:44	
	GOOD	----	verify.ser_config	timestamp=1543504364	localtime=Nov 29 07:12:44	
	GOOD	----	verify.job	timestamp=1543504365	localtime=Nov 29 07:12:45	
	GOOD	----	verify.servod	timestamp=1543504369	localtime=Nov 29 07:12:49	
	GOOD	----	verify.pwr_button	timestamp=1543504369	localtime=Nov 29 07:12:49	
	GOOD	----	verify.lid_open	timestamp=1543504369	localtime=Nov 29 07:12:49	
	GOOD	----	verify.PASS	timestamp=1543504369	localtime=Nov 29 07:12:49	
	START	provision_AutoUpdate	provision_AutoUpdate	timestamp=1543504370	localtime=Nov 29 07:12:50	
		START	----	----	timestamp=1543504400	localtime=Nov 29 07:13:20	
			GOOD	----	sysinfo.before	timestamp=1543504401	localtime=Nov 29 07:13:21	
		END GOOD	----	----	timestamp=1543504401	localtime=Nov 29 07:13:21	
		START	----	reboot	timestamp=1543504454	localtime=Nov 29 07:14:14	
			GOOD	----	reboot.start	timestamp=1543504454	localtime=Nov 29 07:14:14	
			GOOD	----	reboot.verify	timestamp=1543504487	localtime=Nov 29 07:14:47	
		END GOOD	----	reboot	kernel=4.14.83-08187-g08ab73f6ddf2	localtime=Nov 29 07:14:52	timestamp=1543504492	
		START	----	reboot	timestamp=1543504647	localtime=Nov 29 07:17:27	
			GOOD	----	reboot.start	timestamp=1543504647	localtime=Nov 29 07:17:27	
			GOOD	----	reboot.verify	timestamp=1543504708	localtime=Nov 29 07:18:28	
		END GOOD	----	reboot	kernel=4.14.83-08179-g6b161ff16455	localtime=Nov 29 07:18:33	timestamp=1543504713	
		GOOD	provision_AutoUpdate	provision_AutoUpdate	timestamp=1543504738	localtime=Nov 29 07:18:58	completed successfully
	END GOOD	provision_AutoUpdate	provision_AutoUpdate	timestamp=1543504738	localtime=Nov 29 07:18:58	
	INFO	----	----	timestamp=1543504738	localtime=Nov 29 07:18:58	Beginning verify for host chromeos2-row3-rack9-host1 board grunt model grunt
	GOOD	----	verify.ssh	timestamp=1543504739	localtime=Nov 29 07:18:59	
	GOOD	----	verify.devmode	timestamp=1543504739	localtime=Nov 29 07:18:59	
	GOOD	----	verify.hwid	timestamp=1543504741	localtime=Nov 29 07:19:01	
	GOOD	----	verify.power	timestamp=1543504742	localtime=Nov 29 07:19:02	
	GOOD	----	verify.ext4	timestamp=1543504743	localtime=Nov 29 07:19:03	
	GOOD	----	verify.writable	timestamp=1543504744	localtime=Nov 29 07:19:04	
	GOOD	----	verify.tpm	timestamp=1543504745	localtime=Nov 29 07:19:05	
	GOOD	----	verify.good_au	timestamp=1543504746	localtime=Nov 29 07:19:06	
	GOOD	----	verify.fwstatus	timestamp=1543504746	localtime=Nov 29 07:19:06	
	FAIL	----	verify.rwfw	timestamp=1543504749	localtime=Nov 29 07:19:09	DUT firmware requires update from Google_Grunt.11031.34.0 to Google_Grunt.11031.36.0
	GOOD	----	verify.python	timestamp=1543504750	localtime=Nov 29 07:19:10	
	GOOD	----	verify.cros	timestamp=1543504758	localtime=Nov 29 07:19:18	
	GOOD	----	verify.ec_reset	timestamp=1543504758	localtime=Nov 29 07:19:18	
END FAIL	----	provision	timestamp=1543504758	localtime=Nov 29 07:19:18	
INFO	----	----	timestamp=1543504758	job_abort_reason=	localtime=Nov 29 07:19:18	
INFO	----	----	timestamp=1543504762	localtime=Nov 29 07:19:22	Start crashcollection record
INFO	----	New Crash Dump	timestamp=1543504762	localtime=Nov 29 07:19:22	/usr/local/autotest/results/hosts/chromeos2-row3-rack9-host1/1147288-provision/20182911071226/sysinfo/var/spool/crash/cros_camera_service.20181129.040129.1765.dmp
INFO	----	----	timestamp=1543504762	localtime=Nov 29 07:19:22	End crashcollection record

Comment 4 by ayatane@chromium.org, Nov 29

Status: WontFix (was: Assigned)
Unfortunately expected behavior of how stable firmware version is managed:

The automated firmware update sometimes causes Provision failures:
Provision runs verify
Verify determines that firmware update is required
The just provisioned build doesn’t bundle the required firmware
Verify fails, which causes provisioning to fail
Repair fixes the problem, but there was still a failed provisioning task

Comment 5 by ayatane@chromium.org, Nov 30 

Issue 910416 has been merged into this issue.
►

Sign in to add a comment