Moblab: Tests are waiting in Queue after firmware_UpdateFirmwareVersion(faft_bios) test fail. |
||||||
Issue description
Chromeos Version: guado_moblab-release/R55-8872.73.0(MOBLAB2)
What steps will reproduce the problem?
(1) Create tunnel to access moblab2
Creating tunnel with: ssh -M -S /home/tgillella/.ssh/connections/moblab-sock-9082-80 -fNT -o ExitOnForwardFailure=yes -L 9082:localhost:80 moblab@100.107.195.212
(2) Create faft BIOS and EC suites using new user interface (moblab->run suite) without selecting RO and RW firmwares.
(After step2, tests are running successfully)
What is the expected result?
Faft_bios and faft_ec tests execution should be complete.
What happens instead?
firmware_UpdateFirmwareVersion test from faft_bios suite failed and after that all the tests are waiting in queue.
Please use labels and text to provide additional information.
Test device(ultima) is crashed and device is not waking up.
firmware_UpdateFirmwareVersion test logs attached.
01/17 23:44:13.960 INFO | faft_checkers:0105| Expected 'tried_fwb' value '0' == real value '0'
01/17 23:44:13.961 INFO | faft_checkers:0105| Expected 'mainfw_act' value 'A' == real value 'A'
01/17 23:44:13.961 INFO | faft_checkers:0105| Expected 'fwb_tries' value '0' == real value '0'
01/17 23:44:13.961 INFO | firmware_test:1140| -[FAFT]-[ end state_checker ]----------------
01/17 23:44:20.772 INFO |firmware_UpdateFir:0037| Update success, now version is 2
01/17 23:44:45.660 INFO | mode_switcher:0238| -[ModeSwitcher]-[ start reboot_to_mode('rec', None, True) ]-
01/17 23:44:46.394 INFO | servo:0512| Setting power_state to off
01/17 23:45:05.445 INFO | servo:0512| Setting power_state to rec
01/17 23:45:08.574 WARNI| test:0606| Autotest caught exception when running test:
Traceback (most recent call last):
File "/usr/local/autotest/client/common_lib/test.py", line 600, in _exec
_call_test_function(self.execute, *p_args, **p_dargs)
File "/usr/local/autotest/client/common_lib/test.py", line 804, in _call_test_function
return func(*args, **dargs)
File "/usr/local/autotest/client/common_lib/test.py", line 461, in execute
dargs)
File "/usr/local/autotest/client/common_lib/test.py", line 347, in _call_run_once_with_retry
postprocess_profiled_run, args, dargs)
File "/usr/local/autotest/client/common_lib/test.py", line 376, in _call_run_once
self.run_once(*args, **dargs)
File "/usr/local/autotest/server/site_tests/firmware_UpdateFirmwareVersion/firmware_UpdateFirmwareVersion.py", line 91, in run_once
self.reboot_and_reset_tpm()
File "/usr/local/autotest/server/cros/faft/firmware_test.py", line 944, in reboot_and_reset_tpm
self.switcher.reboot_to_mode(to_mode='rec')
File "/usr/local/autotest/server/cros/faft/utils/mode_switcher.py", line 242, in reboot_to_mode
self._enable_rec_mode_and_reboot(usb_state='dut')
File "/usr/local/autotest/server/cros/faft/utils/mode_switcher.py", line 356, in _enable_rec_mode_and_reboot
psc.power_on(psc.REC_ON)
File "/usr/local/autotest/server/cros/servo/servo.py", line 108, in power_on
self._servo.set_nocheck('power_state', rec_mode)
File "/usr/local/autotest/server/cros/servo/servo.py", line 518, in set_nocheck
raise error.TestFail(err_msg)
TestFail: Setting 'power_state' to 'rec' :: Timeout waiting for response.
01/18 00:00:51.391 ERROR| base_utils:0280| [stderr] ssh: connect to host 192.168.231.102 port 22: No route to host^M
,
Jan 18 2017
+tgillella please provide the test result directory should be under /mnt/moblab/results The moblab IP is 100.107.195.212
,
Jan 18 2017
Here are the bios run summary: Latest suite test result for Moblab: 100.107.195.212 Suite location: /mnt/moblab/results/917-moblab/hostless Test suite: ultima-release/R57-9173.0.0-test_suites/control.faft_bios 918-moblab 1274 GOOD firmware_FwScreenCloseLid.dev 919-moblab 1144 GOOD firmware_FwScreenPressPower.dev 920-moblab 391 GOOD firmware_UpdateFirmwareDataKeyVersion 921-moblab 426 GOOD firmware_EventLog 922-moblab 461 GOOD firmware_UpdateKernelVersion.dev 923-moblab 251 GOOD firmware_RONormalBoot.dev 924-moblab 498 GOOD firmware_TPMKernelVersion 925-moblab 250 GOOD firmware_FAFTSetup 926-moblab 54 GOOD firmware_TPMVersionCheck.normal 927-moblab 36 GOOD firmware_Mosys.normal 928-moblab 507 GOOD firmware_FWtries.dev 929-moblab 410 GOOD firmware_CorruptKernelA.dev 930-moblab 177 GOOD firmware_CorruptKernelA.normal 931-moblab 425 GOOD firmware_RecoveryButton.dev 932-moblab 356 GOOD firmware_TryFwB.dev 933-moblab 492 GOOD firmware_CorruptBothFwBodyAB.dev 934-moblab 213 GOOD firmware_RecoveryButton.normal 935-moblab 567 GOOD firmware_CorruptBothFwSigAB.dev 936-moblab 520 GOOD firmware_CorruptBothKernelAB.dev 937-moblab 473 GOOD firmware_CorruptBothFwSigAB.normal 938-moblab 365 GOOD firmware_RollbackFirmware.normal 939-moblab 480 GOOD firmware_DevBootUSB.dev 940-moblab 121 GOOD firmware_TryFwB.normal 941-moblab 338 GOOD firmware_CorruptBothFwBodyAB.normal 942-moblab 442 GOOD firmware_UpdateKernelSubkeyVersion.dev 943-moblab 780 GOOD firmware_LegacyRecovery.normal 944-moblab 63 GOOD firmware_RONormalBoot.normal 945-moblab 403 GOOD firmware_CorruptFwBodyA.dev 946-moblab 179 GOOD firmware_FWtries.normal 947-moblab 612 FAIL firmware_InvalidUSB.normal 948-moblab 535 GOOD firmware_CorruptKernelB.dev 949-moblab 380 GOOD firmware_RollbackKernel.dev 950-moblab 1630 FAIL firmware_RollbackKernel.normal 951-moblab 0 UNKONWN Missing Test 952-moblab 1834 FAIL firmware_DevScreenTimeout.dev 953-moblab 1047 FAIL firmware_CorruptFwBodyB.dev 954-moblab 1194 FAIL firmware_TPMExtend 955-moblab 162 GOOD firmware_CorruptFwBodyA.normal 956-moblab 276 GOOD firmware_CorruptKernelB.normal 957-moblab 1828 FAIL firmware_TPMVersionCheck.dev 958-moblab 1046 FAIL firmware_RollbackFirmware.dev 959-moblab 1046 FAIL firmware_CorruptFwSigB.dev 960-moblab 66 FAIL firmware_SelfSignedBoot.dev 961-moblab 1044 FAIL firmware_UpdateKernelDataKeyVersion.dev 962-moblab 363 GOOD firmware_CorruptBothKernelAB.normal 963-moblab 1047 FAIL firmware_CorruptFwSigA.dev 964-moblab 318 FAIL firmware_UserRequestRecovery.normal 965-moblab 1204 FAIL firmware_UpdateFirmwareVersion 966-moblab 0 UNKONWN Missing Test 967-moblab 0 UNKONWN Missing Test 968-moblab 0 UNKONWN Missing Test 969-moblab 0 UNKONWN Missing Test 970-moblab 0 UNKONWN Missing Test Suite Duration: 27728 seconds or 7.70 hours looks like it failed a few test and by 965 the suite no longer can continue.
,
Jan 18 2017
In a lot of the failed jobs, it said "sudo start servod BOARD=ultima PORT=9999" failed, and some error like: "No hosts within board:ultima satisfy servo" Do we have servo config right for the board ultima?
,
Jan 19 2017
You can ignore servod start failure in test log. The test somehow always try to start servod but servod is already running in moblab.
,
Jan 23 2017
Here are the bios run summary: Latest suite test result for Moblab: 100.107.195.212 Suite location: /mnt/moblab/results/1191-moblab/hostless Test suite: asuka-release/R56-9000.67.0-test_suites/control.faft_bios 1192-moblab 566 GOOD platform_ServoPowerStateController 1193-moblab 808 GOOD firmware_FwScreenCloseLid.dev 1194-moblab 862 GOOD firmware_FwScreenPressPower.dev 1195-moblab 436 GOOD firmware_UpdateFirmwareDataKeyVersion 1196-moblab 513 GOOD firmware_EventLog 1197-moblab 341 GOOD firmware_UpdateKernelVersion.dev 1198-moblab 219 GOOD firmware_RONormalBoot.dev 1199-moblab 394 GOOD firmware_TPMKernelVersion 1200-moblab 302 GOOD firmware_FAFTSetup 1201-moblab 573 GOOD firmware_CorruptBothFwSigAB.dev 1202-moblab 103 GOOD firmware_Mosys.normal 1203-moblab 383 GOOD firmware_FWtries.dev 1204-moblab 316 GOOD firmware_CorruptKernelA.dev 1205-moblab 186 GOOD firmware_CorruptKernelA.normal 1206-moblab 379 GOOD firmware_RecoveryButton.dev 1207-moblab 273 GOOD firmware_TryFwB.dev 1208-moblab 524 GOOD firmware_CorruptBothFwBodyAB.dev 1209-moblab 239 GOOD firmware_RecoveryButton.normal 1210-moblab 424 GOOD firmware_CorruptBothKernelAB.dev 1211-moblab 372 FAIL firmware_CorruptBothFwSigAB.normal 1212-moblab 442 FAIL firmware_RollbackFirmware.normal 1213-moblab 433 GOOD firmware_DevBootUSB.dev 1214-moblab 154 GOOD firmware_TryFwB.normal 1215-moblab 436 FAIL firmware_CorruptBothFwBodyAB.normal 1216-moblab 739 FAIL firmware_LegacyRecovery.normal 1217-moblab 64 GOOD firmware_RONormalBoot.normal 1218-moblab 344 GOOD firmware_CorruptFwBodyA.dev 1219-moblab 344 GOOD firmware_FWtries.normal 1220-moblab 54 GOOD firmware_TPMVersionCheck.normal 1221-moblab 109 ERROR firmware_InvalidUSB.normal 1222-moblab 378 GOOD firmware_CorruptKernelB.dev 1223-moblab 296 GOOD firmware_RollbackKernel.dev 1224-moblab 1622 FAIL firmware_RollbackKernel.normal 1225-moblab 224 ERROR firmware_UserRequestRecovery.dev 1226-moblab 1438 FAIL firmware_DevScreenTimeout.dev 1227-moblab 1048 FAIL firmware_CorruptFwBodyB.dev 1228-moblab 448 GOOD firmware_TPMExtend 1229-moblab 204 GOOD firmware_CorruptFwBodyA.normal 1230-moblab 601 GOOD firmware_CorruptKernelB.normal 1231-moblab 200 GOOD firmware_TPMVersionCheck.dev 1232-moblab 1609 FAIL firmware_RollbackFirmware.dev firmware_RollbackFirmware.dev test failed and device is with recovery screen(0x17 /0x17 RW firmware version rollback detected). After this all the remaining tests are aborted. Attached failed(firmware_RollbackFirmware.dev) test logs from moblab.
,
Mar 1 2017
Is this still an issue? Since Moblab M55 is released with FAFT passed, I will close it. Reopen if you still see it as an issue. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by dchan@google.com
, Jan 18 2017