Repair can't diagnose an offline Moblab instance
Reported by
jrbarnette@chromium.org,
Dec 6 2016
|
|||||
Issue description
If a moblab instance fails by going offline, Repair tasks
don't provide proper diagnosis in status.log.
Here's what status.log looks like:
START ---- repair timestamp=1480789105 localtime=Dec 03 10:18:25
END FAIL ---- repair timestamp=1480789234 localtime=Dec 03 10:20:34
If you dive into the debug logs, you can find this traceback:
Traceback (most recent call last):
File "/usr/local/autotest/server/control_segments/repair", line 20, in repair
try_servo_repair=True)
File "/usr/local/autotest/server/hosts/factory.py", line 252, in create_target_machine
return create_host(machine, **kwargs)
File "/usr/local/autotest/server/hosts/factory.py", line 188, in create_host
host_instance = host_class(hostname, **args)
File "/usr/local/autotest/server/hosts/base_classes.py", line 56, in __init__
super(Host, self).__init__(*args, **dargs)
File "/usr/local/autotest/client/common_lib/hosts/base_classes.py", line 69, in __init__
self._initialize(*args, **dargs)
File "/usr/local/autotest/server/hosts/moblab_host.py", line 85, in _initialize
self.run('rm -rf %s/*' % MOBLAB_IMAGE_STORAGE)
File "/usr/local/autotest/server/hosts/ssh_host.py", line 187, in run
options, stdin, args, ignore_timeout)
File "/usr/local/autotest/server/hosts/ssh_host.py", line 148, in _run
raise error.AutoservSSHTimeout("ssh timed out", result)
The failure happens in the MoblabHost initialization code:
def _initialize(self, *args, **dargs):
# [ ... ]
# Clear the Moblab Image Storage so that staging an image is properly
# tested.
if dargs.get('retain_image_storage') is not True:
self.run('rm -rf %s/*' % MOBLAB_IMAGE_STORAGE)
# [ ... ]
When the moblab is down, the call to self.run() above fails, meaning
creating the MoblabHost fails, meaning repair can't run (because the
repair operation requires a Host object).
As a rule, Host _initialize() methods aren't allowed to fail, and so
aren't allowed to call self.run().
,
Dec 6 2016
,
Dec 7 2016
,
Feb 3 2017
,
Feb 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/b79379f47294920f8d44227e93a704e12724f05b commit b79379f47294920f8d44227e93a704e12724f05b Author: Simran Basi <sbasi@google.com> Date: Wed Feb 15 21:33:10 2017 [autotest] Move MobLab image storage cleanup and ensure moblab_RunSuite runs a test. Fixing two problems with this CL. 1) Migrates the cleanup of the moblab image storage folder from moblab_host to the moblab_RunSuite test. 2) Updates the moblab_RunSuite test and run_suite to ensure atleast a single test is ran to prevent breakages that create 0 tests. BUG= chromium:671709 ,chromium:642905 TEST=None Change-Id: Ief1567c0c1a692d4ca10735b30f49cc43776e9ed Reviewed-on: https://chromium-review.googlesource.com/437750 Commit-Ready: Simran Basi <sbasi@chromium.org> Tested-by: Simran Basi <sbasi@chromium.org> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/b79379f47294920f8d44227e93a704e12724f05b/site_utils/run_suite.py [modify] https://crrev.com/b79379f47294920f8d44227e93a704e12724f05b/server/site_tests/moblab_RunSuite/moblab_RunSuite.py [modify] https://crrev.com/b79379f47294920f8d44227e93a704e12724f05b/server/hosts/moblab_host.py
,
Feb 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ba90ec86aa31ac93f2eeba1f5407f0426ced508d commit ba90ec86aa31ac93f2eeba1f5407f0426ced508d Author: Simran Basi <sbasi@chromium.org> Date: Fri Feb 17 06:59:08 2017 Revert "[autotest] Move MobLab image storage cleanup and ensure moblab_RunSuite runs a test." This reverts commit b79379f47294920f8d44227e93a704e12724f05b. Reason for revert: <INSERT REASONING HERE> Original change's description: > [autotest] Move MobLab image storage cleanup and ensure moblab_RunSuite runs a test. > > Fixing two problems with this CL. > > 1) Migrates the cleanup of the moblab image storage folder > from moblab_host to the moblab_RunSuite test. > > 2) Updates the moblab_RunSuite test and run_suite to ensure > atleast a single test is ran to prevent breakages that > create 0 tests. > > BUG= chromium:671709 ,chromium:642905 > TEST=None > > Change-Id: Ief1567c0c1a692d4ca10735b30f49cc43776e9ed > Reviewed-on: https://chromium-review.googlesource.com/437750 > Commit-Ready: Simran Basi <sbasi@chromium.org> > Tested-by: Simran Basi <sbasi@chromium.org> > Reviewed-by: Dan Shi <dshi@google.com> > TBR=sbasi@chromium.org,sbasi@google.com,dshi@google.com,dshi@chromium.org,akeshet@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. BUG= chromium:671709 ,chromium:642905 Change-Id: Ie2f7b78fe5fd39b004436b367ab62050ae76dc1b Reviewed-on: https://chromium-review.googlesource.com/444310 Reviewed-by: Simran Basi <sbasi@chromium.org> Tested-by: Simran Basi <sbasi@chromium.org> Commit-Queue: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/ba90ec86aa31ac93f2eeba1f5407f0426ced508d/site_utils/run_suite.py [modify] https://crrev.com/ba90ec86aa31ac93f2eeba1f5407f0426ced508d/server/site_tests/moblab_RunSuite/moblab_RunSuite.py [modify] https://crrev.com/ba90ec86aa31ac93f2eeba1f5407f0426ced508d/server/hosts/moblab_host.py
,
Jul 18 2017
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by jrbarnette@chromium.org
, Dec 6 2016