New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 597480 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Mar 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Powerwash not triggered by TPM failure

Reported by jrbarnette@chromium.org, Mar 24 2016

Issue description

Observe this repair failure:
    https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos2-row4-rack6-host3/52705964-repair/

It fails because of a failure in the 'tpm' verifier.
(The failure is the wrong exception because of  bug 597444 ,
but it's still a failure).

Failures in 'tpm' are supposed to trigger the 'powerwash' repair
action.  But, the source code doesn't say that:
    repair_actions = [
        # ...
        (AutoUpdateRepair, 'au', ['ssh', 'good_au'], ['cros']),
        (PowerWashRepair, 'powerwash', ['ssh'], ['good_au', 'cros']),
        (ServoInstallRepair, 'usb', [], ['ssh', 'good_au', 'cros']),
    ]

Somewhere along the way, a merge got lost.

 
It's a three line fix:
        (AutoUpdateRepair, 'au', ['ssh', 'good_au', 'tpm'], ['cros']),
        (PowerWashRepair, 'powerwash', ['ssh'], ['good_au', 'tpm', 'cros']),
        (ServoInstallRepair, 'usb', [], ['ssh', 'good_au', 'tpm', 'cros']),

That is 'tpm' blocks AutoUpdateRepair, and triggers both
PowerWashRepair and ServoInstallRepair
Owner: xixuan@chromium.org
Status: Verified (was: Available)
Devices are coming back online.

Sample successful repair:

START	----	repair	timestamp=1458947802	localtime=Mar 25 16:16:42	
	GOOD	----	verify.ssh	timestamp=1458947819	localtime=Mar 25 16:16:59	
	GOOD	----	verify.power	timestamp=1458947819	localtime=Mar 25 16:16:59	
	GOOD	----	verify.cros	timestamp=1458947821	localtime=Mar 25 16:17:01	
	GOOD	----	verify.good_au	timestamp=1458947821	localtime=Mar 25 16:17:01	
	FAIL	----	verify.tpm	timestamp=1458947822	localtime=Mar 25 16:17:02	Cannot load the TPM SRC public key
	START	----	repair.powerwash	timestamp=1458947822	localtime=Mar 25 16:17:02	
		START	----	reboot	timestamp=1458947822	localtime=Mar 25 16:17:02	
			GOOD	----	reboot.start	timestamp=1458947822	localtime=Mar 25 16:17:02	
			GOOD	----	reboot.verify	timestamp=1458947857	localtime=Mar 25 16:17:37	
		END GOOD	----	reboot	kernel=3.14.0	localtime=Mar 25 16:17:37	timestamp=1458947857	
		START	----	reboot	timestamp=1458947867	localtime=Mar 25 16:17:47	
			GOOD	----	reboot.start	timestamp=1458947867	localtime=Mar 25 16:17:47	
			GOOD	----	reboot.verify	timestamp=1458947885	localtime=Mar 25 16:18:05	
		END GOOD	----	reboot	kernel=3.14.0	localtime=Mar 25 16:18:05	timestamp=1458947885	
		START	----	reboot	timestamp=1458948065	localtime=Mar 25 16:21:05	
			GOOD	----	reboot.start	timestamp=1458948065	localtime=Mar 25 16:21:05	
			GOOD	----	reboot.verify	timestamp=1458948099	localtime=Mar 25 16:21:39	
		END GOOD	----	reboot	kernel=3.14.0	localtime=Mar 25 16:21:40	timestamp=1458948100	
		GOOD	----	verify.ssh	timestamp=1458948131	localtime=Mar 25 16:22:11	
		GOOD	----	verify.tpm	timestamp=1458948131	localtime=Mar 25 16:22:11	
		GOOD	----	verify.good_au	timestamp=1458948131	localtime=Mar 25 16:22:11	
		GOOD	----	verify.cros	timestamp=1458948134	localtime=Mar 25 16:22:14	
	END GOOD	----	repair.powerwash	timestamp=1458948134	localtime=Mar 25 16:22:14	
	GOOD	----	verify.PASS	timestamp=1458948134	localtime=Mar 25 16:22:14	
	START	----	reboot	timestamp=1458948134	localtime=Mar 25 16:22:14	
		GOOD	----	reboot.start	timestamp=1458948134	localtime=Mar 25 16:22:14	
		GOOD	----	reboot.verify	timestamp=1458948153	localtime=Mar 25 16:22:33	
	END GOOD	----	reboot	kernel=3.14.0	localtime=Mar 25 16:22:34	timestamp=1458948154	
END GOOD	----	repair	timestamp=1458948154	localtime=Mar 25 16:22:34	{'host_attributes': {'job_repo_url': 'http://100.107.160.1:8082/static/samus-release/R50-7978.18.0/autotest/packages', 'powerunit_hostname': 'chromeos2-row1-rack5-rpm1', 'hydra_hostname': 'chromeos-197-hydra1.cros', 'powerunit_outlet': '.A9'}, 'hostname': 'chromeos2-row1-rack5-host4'} repaired successfully

Comment 3 by benhenry@google.com, Apr 27 2016

Components: Infra>Client>ChromeOS
Labels: -Infra-ChromeOS

Sign in to add a comment