New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 788589 link

Starred by 1 user
Status: Duplicate
Owner: ----
Closed: Nov 2017
Cc:
akes...@chromium.org
athilenius@chromium.org
█ slavamn@chromium.org
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

kefka-release: cannot recover from reboot at post check of stateful update // pre-setup of rootfs update

Project Member Reported by drinkcat@chromium.org, Nov 27 2017 
https://luci-milo.appspot.com/buildbot/chromeos/kefka-release/1711

[Test-Logs]: autoupdate_EndToEndTest.paygen_au_dev_delta: retry_count: 1, FAIL: Unhandled DevServerException: CrOS auto-update failed for host chromeos2-row4-rack8-host7: 0) ChromiumOSUpdateError: chromeos2-row4-rack8-host7 cannot recover from reboot at post check of stateful update, 1) SSHConnectionError: ssh: connect to host 100.115.227.26 port 22: Connection timed out,

2017/11/26 15:01:44.730 DEBUG|      auto_updater:1283| Start post check for stateful update...
2017/11/26 15:01:44.730 INFO |     remote_access:0401| Rebooting chromeos2-row4-rack8-host7...
2017/11/26 15:01:44.731 DEBUG|    cros_build_lib:0593| RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmpCHROZy/testing_rsa root@chromeos2-row4-rack8-host7 -- cat /proc/sys/kernel/random/boot_id
2017/11/26 15:01:45.561 DEBUG|    cros_build_lib:0642| (stdout):
aafd7b20-718f-4b17-b62b-a52a6759f5fa

2017/11/26 15:01:45.562 DEBUG|    cros_build_lib:0644| (stderr):
Warning: Permanently added 'chromeos2-row4-rack8-host7,100.115.227.26' (ED25519) to the list of known hosts.
Warning: Permanently added 'chromeos2-row4-rack8-host7,100.115.227.26' (ED25519) to the list of known hosts.

2017/11/26 15:01:45.562 DEBUG|    cros_build_lib:0593| RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmpCHROZy/testing_rsa root@chromeos2-row4-rack8-host7 -- reboot
...
2017/11/26 15:11:56.598 DEBUG|       cros_update:0339| Error happens in CrOS auto-update: ChromiumOSUpdateError('chromeos2-row4-rack8-host7 cannot recover from reboot at post check of stateful update',)

https://luci-milo.appspot.com/buildbot/chromeos/kefka-release/1710
[Test-Logs]: provision: FAIL: Unhandled DevServerException: CrOS auto-update failed for host chromeos2-row4-rack8-host7: 0) ChromiumOSUpdateError: chromeos2-row4-rack8-host7 cannot recover from reboot at pre-setup of rootfs update, 1) SSHConnectionError: ssh: connect to host 100.115.227.26 port 22: Connection timed out,

https://luci-milo.appspot.com/buildbot/chromeos/kefka-release/1709
[Test-Logs]: provision: FAIL: Unhandled DevServerException: CrOS auto-update failed for host chromeos2-row8-rack4-host22: 0) ChromiumOSUpdateError: chromeos2-row8-rack4-host22 cannot recover from reboot at pre-setup of rootfs update, 1) SSHConnectionError: ssh: connect to host 100.115.230.209 port 22: Connection timed out,

https://luci-milo.appspot.com/buildbot/chromeos/kefka-release/1708
[Test-Logs]: provision: FAIL: Unhandled DevServerException: CrOS auto-update failed for host chromeos2-row4-rack8-host7: 0) ChromiumOSUpdateError: chromeos2-row4-rack8-host7 cannot recover from reboot at pre-setup of rootfs update, 1) SSHConnectionError: ssh: connect to host 100.115.227.26 port 22: Connection timed out,

https://luci-milo.appspot.com/buildbot/chromeos/kefka-release/1706
[Test-Logs]: autoupdate_EndToEndTest.paygen_au_dev_delta: retry_count: 1, FAIL: Unhandled DevServerException: CrOS auto-update failed for host chromeos2-row8-rack4-host22: 0) ChromiumOSUpdateError: chromeos2-row8-rack4-host22 cannot recover from reboot at pre-setup of rootfs update, 1) SSHConnectionError: ssh: connect to host 100.115.230.209 port 22: Connection timed out,
[Test-Logs]: provision: FAIL: Unhandled DevServerException: CrOS auto-update failed for host chromeos2-row4-rack8-host7: 0) ChromiumOSUpdateError: chromeos2-row4-rack8-host7 cannot recover from reboot at post check of stateful update, 1) SSHConnectionError: ssh: connect to host 100.115.227.26 port 22: Connection timed out,

Most of the failures seem to be on chromeos2-row4-rack8-host7 (but occasionally on chromeos2-row8-rack4-host22 as well...)

Comment 1 by drinkcat@chromium.org, Nov 27 2017

Mergedinto: 788584
Status: Duplicate (was: Untriaged)
Looking at:
https://luci-milo.appspot.com/buildbot/chromeos/kefka-release/1710
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/158680468-chromeos-test/chromeos2-row4-rack8-host7/autoupdate_logs/

Reboot happens at:
2017/11/25 20:31:05.894 INFO |     remote_access:0401| Rebooting chromeos2-row4-rack8-host7...
...
2017/11/25 20:31:06.489 DEBUG|    cros_build_lib:0593| RunCommand: ssh -p 22 '-oConnectionAttempts=4' '-oUserKnownHostsFile=/dev/null' '-oProtocol=2' '-oConnectTimeout=30' '-oServerAliveCountMax=3' '-oStrictHostKeyChecking=no' '-oServerAliveInterval=10' '-oNumberOfPasswordPrompts=0' '-oIdentitiesOnly=yes' -i /tmp/ssh-tmpNh5BsN/testing_rsa root@chromeos2-row4-rack8-host7 -- reboot

From repair job that follows, we can see:
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos2-row4-rack8-host7/1452940-repair/20172511205111

2017-11-25T20:32:03.673043-08:00 ERR kernel: [   11.700789] usb 2-2: device not accepting address 2, error -62
2017-11-25T20:32:24.993093-08:00 ERR kernel: [   33.020894] usb 2-2: device not accepting address 3, error -62
2017-11-25T20:32:38.025316-08:00 ERR kernel: [   46.052346] usb 2-2: device not accepting address 4, error -62
2017-11-25T20:32:50.985055-08:00 ERR kernel: [   59.011675] usb 2-2: device not accepting address 5, error -62

After a reboot (probably servo-initiated):
2017-11-25T21:09:36.580077-08:00 INFO kernel: [    1.244366] usb 2-2: new SuperSpeed USB device number 2 using xhci_hcd
2017-11-25T21:09:36.580114-08:00 INFO kernel: [    1.256652] usb 2-2: New USB device found, idVendor=13b1, idProduct=0041
2017-11-25T21:09:36.580116-08:00 INFO kernel: [    1.256669] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=6
2017-11-25T21:09:36.580117-08:00 INFO kernel: [    1.256681] usb 2-2: Product: Linksys USB3GIGV1
2017-11-25T21:09:36.580125-08:00 INFO kernel: [    1.256690] usb 2-2: Manufacturer: Linksys
2017-11-25T21:09:36.580128-08:00 INFO kernel: [    1.256698] usb 2-2: SerialNumber: 000001000000
2017-11-25T21:09:48.457053-08:00 INFO kernel: [   15.485014] usb 2-2: reset SuperSpeed USB device number 2 using xhci_hcd
2017-11-25T21:09:48.499053-08:00 INFO kernel: [   15.527397] r8152 2-2:1.0 eth1: v1.08.3

Duping into 788584 .
►

Sign in to add a comment