New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 757037 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Last visit > 30 days ago
Closed: Aug 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Provision failure: RootfsUpdateError('Update failed with unexpected update status: UPDATE_STATUS_IDLE',)

Project Member Reported by nxia@chromium.org, Aug 18 2017

Issue description

https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/15776

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/135981535-chromeos-test/chromeos4-row4-rack13-host1/debug/


08/18 10:55:03.471 DEBUG|             utils:0212| Running 'ssh 100.115.219.136 'curl "http://100.115.219.136:8082/kill_au_proc?pid=22615&host_name=chromeos4-row4-rack13-host1"''
08/18 10:55:04.192 DEBUG|        dev_server:2153| Exception raised on auto_update attempt #1:
 Traceback (most recent call last):
  File "/home/chromeos-test/chromiumos/src/platform/dev/cros_update.py", line 259, in TriggerAU
    self._RootfsUpdate(chromeos_AU)
  File "/home/chromeos-test/chromiumos/src/platform/dev/cros_update.py", line 170, in _RootfsUpdate
    cros_updater.UpdateRootfs()
  File "/home/chromeos-test/chromiumos/chromite/lib/auto_updater.py", line 717, in UpdateRootfs
    raise RootfsUpdateError(error_msg % e)
RootfsUpdateError: Failed to perform rootfs update: RootfsUpdateError('Update failed with unexpected update status: UPDATE_STATUS_IDLE',)



 

Comment 1 by nxia@chromium.org, Aug 18 2017

I checked the provision failures, most of them contain crash info. was there a bad CL causing the crashes?

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/135981535-chromeos-test/chromeos4-row4-rack13-host1/crashinfo.chromeos4-row4-rack13-host1/

Comment 2 by nxia@chromium.org, Aug 18 2017

Cc: chirantan@chromium.org julanhsu@chromium.org tfiga@chromium.org

Comment 3 by nxia@chromium.org, Aug 18 2017

https://luci-milo.appspot.com/buildbot/chromeos/veyron_mighty-paladin/6307


08/18 11:08:16.133 DEBUG|             utils:0212| Running 'ssh 100.115.219.136 'curl "http://100.115.219.136:8082/kill_au_proc?pid=21900&host_name=100.115.195.178"''
08/18 11:08:16.936 DEBUG|        dev_server:2153| Exception raised on auto_update attempt #2:
 Traceback (most recent call last):
  File "/home/chromeos-test/chromiumos/src/platform/dev/cros_update.py", line 259, in TriggerAU
    self._RootfsUpdate(chromeos_AU)
  File "/home/chromeos-test/chromiumos/src/platform/dev/cros_update.py", line 170, in _RootfsUpdate
    cros_updater.UpdateRootfs()
  File "/home/chromeos-test/chromiumos/chromite/lib/auto_updater.py", line 721, in UpdateRootfs
    ds.Stop()
  File "/home/chromeos-test/chromiumos/chromite/lib/dev_server_wrapper.py", line 603, in Stop
    raise DevServerStopError('Unable to stop devserver: %s' % e)


Comment 4 by nxia@chromium.org, Aug 18 2017

https://luci-milo.appspot.com/buildbot/chromeos/winky-paladin/2650


08/18 11:14:30.961 DEBUG|             utils:0212| Running 'ssh 100.115.219.136 'curl "http://100.115.219.136:8082/kill_au_proc?pid=23234&host_name=100.115.194.196"''
08/18 11:14:32.039 DEBUG|        dev_server:2153| Exception raised on auto_update attempt #2:
 Traceback (most recent call last):
  File "/home/chromeos-test/chromiumos/src/platform/dev/cros_update.py", line 264, in TriggerAU
    self._WriteAUStatus(cros_update_progress.FINISHED)
  File "/home/chromeos-test/chromiumos/chromite/lib/remote_access.py", line 566, in __exit__
    self.device.Cleanup()
  File "/home/chromeos-test/chromiumos/chromite/lib/remote_access.py", line 695, in Cleanup
    self.BaseRunCommand(cmd, **kwargs)
  File "/home/chromeos-test/chromiumos/chromite/lib/remote_access.py", line 889, in BaseRunCommand
    return self.GetAgent().RemoteSh(cmd, **kwargs)
  File "/home/chromeos-test/chromiumos/chromite/lib/remote_access.py", line 344, in RemoteSh
    raise SSHConnectionError(e.result.error)
SSHConnectionError: ssh: connect to host 100.115.194.196 port 22: Connection timed out
08/18 11:14:32.040 DEBUG|        dev_server:2159| Please see error details in log /usr/local/autotest/results/hosts/chromeos4-row3-rack12-host13/731345-provision/20171808105853/autoupdate_logs/CrOS_update_100.115.194.196_23234.log
08/18 11:14:32.046 DEBUG|              test:0389| Test failed due to CrOS auto-update failed for host chromeos4-row3-rack12-host13: SSHConnectionError: ssh: Could not resolve hostname chromeos4-row3-rack12-host13: Name or service not known

Comment 5 by nxia@chromium.org, Aug 18 2017

https://luci-milo.appspot.com/buildbot/chromeos/kip-paladin/2644


08/18 10:56:05.988 DEBUG|             utils:0212| Running 'ssh 100.115.219.136 'curl "http://100.115.219.136:8082/kill_au_proc?pid=25398&host_name=chromeos4-row1-rack12-host19"''
08/18 10:56:07.137 DEBUG|        dev_server:2153| Exception raised on auto_update attempt #1:
 Traceback (most recent call last):
  File "/home/chromeos-test/chromiumos/src/platform/dev/cros_update.py", line 259, in TriggerAU
    self._RootfsUpdate(chromeos_AU)
  File "/home/chromeos-test/chromiumos/src/platform/dev/cros_update.py", line 170, in _RootfsUpdate
    cros_updater.UpdateRootfs()
  File "/home/chromeos-test/chromiumos/chromite/lib/auto_updater.py", line 714, in UpdateRootfs
    self.RevertBootPartition()
  File "/home/chromeos-test/chromiumos/chromite/lib/auto_updater.py", line 639, in RevertBootPartition
    self.device.RunCommand(['/postinst', part], **self._cmd_kwargs)
  File "/home/chromeos-test/chromiumos/chromite/lib/remote_access.py", line 1098, in RunCommand
    return super(ChromiumOSDevice, self).RunCommand(cmd, **kwargs)
  File "/home/chromeos-test/chromiumos/chromite/lib/remote_access.py", line 942, in RunCommand
    return self.BaseRunCommand(cmd, **kwargs)
  File "/home/chromeos-test/chromiumos/chromite/lib/remote_access.py", line 889, in BaseRunCommand
    return self.GetAgent().RemoteSh(cmd, **kwargs)
  File "/home/chromeos-test/chromiumos/chromite/lib/remote_access.py", line 344, in RemoteSh
    raise SSHConnectionError(e.result.error)
SSHConnectionError: None

Comment 6 by nxia@chromium.org, Aug 18 2017

looks like something wrong with devserver 100.115.219.136 chromeos4-devserver8

Comment 7 by nxia@chromium.org, Aug 18 2017

Cc: pprabhu@chromium.org

Comment 8 Deleted

Comment 9 Deleted

Comment 10 Deleted

Comment 11 by nxia@chromium.org, Aug 28 2017

Status: WontFix (was: Untriaged)
Looks to me this is related to the unstable devserver when the last push-to-devserver happened. 

Sign in to add a comment