New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 652171 link

Starred by 1 user
Status: Archived
Owner:
akes...@chromium.org
Closed: Nov 2016
Cc:
cywang@chromium.org
haoweiw@chromium.org
johndhong@chromium.org
█ jrbarnette@chromium.org
tbroch@chromium.org
█ kevcheng@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Provision Fail: AC power is not plugged in, completed successfully, servod not running on chromeos4-row9-rack11-host12-servo port 9999

Project Member Reported by cn...@chromium.org, Oct 3 2016 
ARC++ ChromeOS veyron_minnie-paladin test failed on provision error:

FAIL: AC power is not plugged in, completed successfully, servod not running on chromeos4-row9-rack11-host12-servo port 9999

Builder page: http://shortn/_UcoxIKVwcY
Error log: http://shortn/_nDzQkvQGXA

Comment 1 by cn...@chromium.org, Oct 3 2016

Cc: cywang@chromium.org

Comment 2 by akes...@chromium.org, Oct 3 2016 

$ dut-status -f chromeos4-row9-rack11-host12
chromeos4-row9-rack11-host12
    2016-10-02 22:57:58  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host12/207030-repair/
    2016-10-02 22:39:36  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host12/206950-provision/
    2016-10-02 20:39:16  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/78996045-chromeos-test/
    2016-10-02 20:38:49  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host12/206573-reset/
    2016-10-02 20:36:50  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/78996046-chromeos-test/
    2016-10-02 20:20:47  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host12/206527-provision/
    2016-10-02 16:56:57  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/78986821-chromeos-test/
    2016-10-02 16:56:32  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host12/206134-reset/
    2016-10-02 16:54:36  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/78986817-chromeos-test/
    2016-10-02 16:38:38  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host12/206102-provision/
    2016-10-02 14:39:44  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/78975938-chromeos-test/
    2016-10-02 14:39:06  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host12/205586-reset/
    2016-10-02 14:37:01  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/78975904-chromeos-test/
    2016-10-02 14:16:54  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host12/205511-provision/
    2016-10-02 12:11:52  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/78953715-chromeos-test/
    2016-10-02 12:11:34  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host12/205246-reset/
    2016-10-02 12:09:43  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/78953711-chromeos-test/
    2016-10-02 11:55:55  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host12/205215-provision/
    2016-10-02 09:20:41  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/78946762-chromeos-test/
    2016-10-02 09:20:05  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host12/204933-reset/
    2016-10-02 09:17:27  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/78946757-chromeos-test/
    2016-10-02 09:03:26  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host12/204901-provision/
    2016-10-02 02:35:41  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/78910465-chromeos-test/
    2016-10-02 02:35:21  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host12/203694-reset/
    2016-10-02 02:33:18  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/78910444-chromeos-test/
    2016-10-02 02:18:29  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row9-rack11-host12/203650-provision/


Next step, examine repair log.

Comment 3 by akes...@chromium.org, Oct 3 2016 

START	----	repair	timestamp=1475474281	localtime=Oct 02 22:58:01	
	GOOD	----	verify.ssh	timestamp=1475474282	localtime=Oct 02 22:58:02	
	GOOD	----	verify.brd_config	timestamp=1475474282	localtime=Oct 02 22:58:02	
	GOOD	----	verify.ser_config	timestamp=1475474282	localtime=Oct 02 22:58:02	
	FAIL	----	verify.job	timestamp=1475474283	localtime=Oct 02 22:58:03	servod not running on chromeos4-row9-rack11-host12-servo port 9999
	START	----	repair.restart	timestamp=1475474283	localtime=Oct 02 22:58:03	
		GOOD	----	verify.ssh	timestamp=1475474304	localtime=Oct 02 22:58:24	
		GOOD	----	verify.brd_config	timestamp=1475474304	localtime=Oct 02 22:58:24	
		GOOD	----	verify.ser_config	timestamp=1475474305	localtime=Oct 02 22:58:25	
		GOOD	----	verify.job	timestamp=1475474305	localtime=Oct 02 22:58:25	
		GOOD	----	verify.servod	timestamp=1475474311	localtime=Oct 02 22:58:31	
		GOOD	----	verify.pwr_button	timestamp=1475474311	localtime=Oct 02 22:58:31	
		GOOD	----	verify.lid_open	timestamp=1475474311	localtime=Oct 02 22:58:31	
	END GOOD	----	repair.restart	timestamp=1475474311	localtime=Oct 02 22:58:31	
	GOOD	----	verify.update	timestamp=1475474315	localtime=Oct 02 22:58:35	
	GOOD	----	verify.PASS	timestamp=1475474315	localtime=Oct 02 22:58:35	
	GOOD	----	verify.ssh	timestamp=1475474321	localtime=Oct 02 22:58:41	
	FAIL	----	verify.power	timestamp=1475474321	localtime=Oct 02 22:58:41	AC power is not plugged in
	START	----	repair.rpm	timestamp=1475474321	localtime=Oct 02 22:58:41	
		FAIL	----	repair.rpm	timestamp=1475474321	localtime=Oct 02 22:58:41	Client call exception: <Fault 1: "<class 'autotest_lib.site_utils.rpm_control_system.rpm_infrastructure_exception.RPMInfrastructureException'>:Can not retrieve complete rpm informationfrom AFE for chromeos4-row9-rack11-host12, please make sure powerunit_hostname and powerunit_outlet are in the host's attributes.">
	END FAIL	----	repair.rpm	timestamp=1475474321	localtime=Oct 02 22:58:41	
	GOOD	----	verify.cros	timestamp=1475474325	localtime=Oct 02 22:58:45	
	GOOD	----	verify.good_au	timestamp=1475474326	localtime=Oct 02 22:58:46	
	GOOD	----	verify.writable	timestamp=1475474326	localtime=Oct 02 22:58:46	
	GOOD	----	verify.tpm	timestamp=1475474327	localtime=Oct 02 22:58:47	
	GOOD	----	verify.python	timestamp=1475474327	localtime=Oct 02 22:58:47	
	START	----	repair.au	timestamp=1475474327	localtime=Oct 02 22:58:47	
		GOOD	----	verify.ssh	timestamp=1475475798	localtime=Oct 02 23:23:18	
		GOOD	----	verify.python	timestamp=1475475799	localtime=Oct 02 23:23:19	
		GOOD	----	verify.cros	timestamp=1475475803	localtime=Oct 02 23:23:23	
		GOOD	----	verify.power	timestamp=1475475803	localtime=Oct 02 23:23:23	
	END GOOD	----	repair.au	timestamp=1475475803	localtime=Oct 02 23:23:23	
	GOOD	----	verify.PASS	timestamp=1475475803	localtime=Oct 02 23:23:23	
	INFO	----	repair	timestamp=1475475803	localtime=Oct 02 23:23:23	Can't repair label 'pool:cq'.
	INFO	----	repair	timestamp=1475475803	localtime=Oct 02 23:23:23	Can't repair label 'arc'.
	INFO	----	repair	timestamp=1475475803	localtime=Oct 02 23:23:23	Can't repair label 'board:veyron_minnie'.
	INFO	----	repair	timestamp=1475475803	localtime=Oct 02 23:23:23	Can't repair label 'cros-version:veyron_minnie-paladin/R55-8858.0.0-rc2'.
END GOOD	----	repair	timestamp=1475475803	localtime=Oct 02 23:23:23	chromeos4-row9-rack11-host12 repaired successfully

Comment 4 by akes...@chromium.org, Oct 3 2016

Cc: haoweiw@chromium.org jrbarnette@chromium.org johndhong@chromium.org
Is there some reason to believe we have no working rpm for this DUT?

Comment 5 by cywang@chromium.org, Oct 3 2016

Cc: kevcheng@chromium.org
EC could not reflect the correct status of 'Line Power', .i.e. 'AC power' before rebooting the DUT. That's why it works after reboot, but we have rpm issue.

Comment 6 by cywang@chromium.org, Oct 3 2016

Cc: tbroch@chromium.org

Comment 7 by johndhong@chromium.org, Oct 3 2016 

RPM issue should be fixed now for that rack

* No RPM entry in database *
johndhong@phobrz:~/chromiumos$ manage_powerunit_info list -m chromeos4-row9-rack11-host12
INFO:root:Connected to cautotest
chromeos4-row9-rack11-host12,,,,

* Adding entry into database *
johndhong@phobrz:~/chromiumos$ manage_powerunit_info -f /tmp/dut_rpm.csv upload
INFO:root:Connected to cautotest
Upload rpm mapping from /tmp/dut_rpm.csv, are you sure? (y/N) y
INFO:root:Adding host attribues to chromeos4-row9-rack11-host1: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm2', 'hydra_hostname': None, 'powerunit_outlet': '.A1'}
DEBUG:root:No host named chromeos4-row9-rack11-host2
INFO:root:Adding host attribues to chromeos4-row9-rack11-host3: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm2', 'hydra_hostname': None, 'powerunit_outlet': '.A3'}
DEBUG:root:No host named chromeos4-row9-rack11-host4
DEBUG:root:No host named chromeos4-row9-rack11-host5
DEBUG:root:No host named chromeos4-row9-rack11-host6
INFO:root:Adding host attribues to chromeos4-row9-rack11-host7: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm2', 'hydra_hostname': None, 'powerunit_outlet': '.A7'}
DEBUG:root:No host named chromeos4-row9-rack11-host8
DEBUG:root:No host named chromeos4-row9-rack11-host9
DEBUG:root:No host named chromeos4-row9-rack11-host10
INFO:root:Adding host attribues to chromeos4-row9-rack11-host11: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm2', 'hydra_hostname': None, 'powerunit_outlet': '.A11'}
INFO:root:Adding host attribues to chromeos4-row9-rack11-host12: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A12'}
INFO:root:Adding host attribues to chromeos4-row9-rack11-host13: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm2', 'hydra_hostname': None, 'powerunit_outlet': '.A13'}
INFO:root:Adding host attribues to chromeos4-row9-rack11-host14: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A14'}
INFO:root:Adding host attribues to chromeos4-row9-rack11-host15: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm2', 'hydra_hostname': None, 'powerunit_outlet': '.A15'}
INFO:root:Adding host attribues to chromeos4-row9-rack11-host16: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A16'}
INFO:root:Adding host attribues to chromeos4-row9-rack11-host17: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm2', 'hydra_hostname': None, 'powerunit_outlet': '.A17'}
INFO:root:Adding host attribues to chromeos4-row9-rack11-host18: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A18'}
INFO:root:Adding host attribues to chromeos4-row9-rack11-host19: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm2', 'hydra_hostname': None, 'powerunit_outlet': '.A19'}
INFO:root:Adding host attribues to chromeos4-row9-rack11-host20: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A20'}
INFO:root:Adding host attribues to chromeos4-row9-rack11-host21: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm2', 'hydra_hostname': None, 'powerunit_outlet': '.A21'}
INFO:root:Adding host attribues to chromeos4-row9-rack11-host22: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A22'}
INFO:root:Adding host attribues to chromeos4-row10-rack11-host1: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A1'}
DEBUG:root:No host named chromeos4-row10-rack11-host2
INFO:root:Adding host attribues to chromeos4-row10-rack11-host3: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A3'}
DEBUG:root:No host named chromeos4-row10-rack11-host4
INFO:root:Adding host attribues to chromeos4-row10-rack11-host5: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A5'}
DEBUG:root:No host named chromeos4-row10-rack11-host6
INFO:root:Adding host attribues to chromeos4-row10-rack11-host7: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A7'}
DEBUG:root:No host named chromeos4-row10-rack11-host8
INFO:root:Adding host attribues to chromeos4-row10-rack11-host9: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A9'}
DEBUG:root:No host named chromeos4-row10-rack11-host10
INFO:root:Adding host attribues to chromeos4-row10-rack11-host11: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A11'}
DEBUG:root:No host named chromeos4-row10-rack11-host12
INFO:root:Adding host attribues to chromeos4-row10-rack11-host13: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A13'}
DEBUG:root:No host named chromeos4-row10-rack11-host14
INFO:root:Adding host attribues to chromeos4-row10-rack11-host15: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A15'}
DEBUG:root:No host named chromeos4-row10-rack11-host16
INFO:root:Adding host attribues to chromeos4-row10-rack11-host17: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A17'}
DEBUG:root:No host named chromeos4-row10-rack11-host18
INFO:root:Adding host attribues to chromeos4-row10-rack11-host19: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A19'}
DEBUG:root:No host named chromeos4-row10-rack11-host20
INFO:root:Adding host attribues to chromeos4-row10-rack11-host21: {'powerunit_hostname': 'chromeos4-row9_10-rack11-rpm1', 'hydra_hostname': None, 'powerunit_outlet': '.A21'}
DEBUG:root:No host named chromeos4-row10-rack11-host22


* Confirmed RPM entry into database *
johndhong@phobrz:~/chromiumos$ manage_powerunit_info list -m chromeos4-row9-rack11-host12
INFO:root:Connected to cautotest
chromeos4-row9-rack11-host12,chromeos4-row9_10-rack11-rpm1,.A12,,

Comment 8 by autumn@chromium.org, Oct 4 2016

Status: Assigned (was: Untriaged)

Comment 9 by autumn@chromium.org, Oct 4 2016

Labels: -current-issue

Comment 10 by akes...@chromium.org, Nov 3 2016

Status: Fixed (was: Assigned)

Comment 11 by dchan@google.com, Jan 21 2017

Labels: VerifyIn-57

Comment 12 by dchan@google.com, Mar 4 2017

Labels: VerifyIn-58

Comment 13 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 14 by dchan@google.com, May 30 2017

Labels: VerifyIn-60

Comment 15 by dchan@chromium.org, Aug 1 2017

Labels: VerifyIn-61

Comment 16 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)
►

Sign in to add a comment