New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 749325 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Aug 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocking:
issue 747056



Sign in to add a comment

Move DUT discovery and setup from moblab provision --> moblab test

Project Member Reported by pprabhu@chromium.org, Jul 26 2017

Issue description

If there is a bug in the autotest code, it is likely to break AFE / server side code on the moblab.
In this case, if we try to add duts etc on a newly provisioned moblab, it'll fail provision. This is wrong.

Provision should just install the image and copy over required files (like .boto)

The test should setup DUTs on the moblab.

This way, bad autotest changes will fail the test, not provision.

There is a complication associated with this though -- if the sub-DUTs attached to moblab are down for some reason, we may currently detect that when running provision on moblab. Moving the DUT setup to the test means that this will blame the CLs being tested.

I argue that this is the right thing to do.
(1) failing to discover the sub-DUTs can be because the DUTs are down / because autotest code installed on moblab is broken -- and these two are indistinguishable.

If we want to detect sub-DUTs during provision, we should do so explicitly, not depending on the software under test just installed on the DUT.
 
Blocking: 747056
Status: Available (was: Untriaged)
moblab_host's verify_software is the offensive override: https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/server/hosts/moblab_host.py?l=234

Compare this to cros_host's verify_software:
https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/server/hosts/cros_host.py?l=1549

It merely validates that there's enough space in various partitions after the install and that we can at least update away from the image if needed.
That's the right thing to do. All the checks of moblab's services, adding duts etc should move to the test.
Cc: akes...@chromium.org sbasi@chromium.org
The impact of this bug is that many autotest CLs that break moblab keep cycling through the CQ until manually kicked because they fail provision and get marked as infra failures.
Status: Untriaged (was: Available)
Untriaged --> on the triage queue.
I might have time to do some work here - I have an OKR to enable the servod and work on adding servo to the duts in the lab so I might be able to implement this suggestion.  I agree setting up the DUT's should be part of the test not part of the provisioning, I would also like to have the DUT setup go through some of the RPC calls the UI uses to reduce the amount of code and improve the testing.
Owner: haddowk@chromium.org
Thanks, haddowk@
Project Member

Comment 7 by bugdroid1@chromium.org, Aug 1 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/007036ca865ca80ad31988aa02080e6370829244

commit 007036ca865ca80ad31988aa02080e6370829244
Author: Keith Haddow <haddowk@chromium.org>
Date: Tue Aug 01 23:22:06 2017

[moblab] Move the moblab setup from the host provision to the test.

In the aim to have moblab issue show up as test failures vs
provision failures tests should setup the moblab.

BUG= chromium:749325 
TEST=trybot https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/paladin/builds/3451

Change-Id: I75e486d4726a33243e4e3238b216b33df1838ad8
Reviewed-on: https://chromium-review.googlesource.com/594663
Commit-Ready: Keith Haddow <haddowk@chromium.org>
Tested-by: Keith Haddow <haddowk@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Keith Haddow <haddowk@chromium.org>

[modify] https://crrev.com/007036ca865ca80ad31988aa02080e6370829244/server/cros/moblab_test.py
[modify] https://crrev.com/007036ca865ca80ad31988aa02080e6370829244/server/hosts/moblab_host.py

Project Member

Comment 8 by bugdroid1@chromium.org, Aug 3 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/26933a0b0dd9642300ef4e83ca9b31f5ccf82352

commit 26933a0b0dd9642300ef4e83ca9b31f5ccf82352
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Thu Aug 03 01:09:12 2017

Revert "[moblab] Move the moblab setup from the host provision to the test."

This reverts commit 007036ca865ca80ad31988aa02080e6370829244.

Reason for revert: This CL tickled a bug that causes moblab provision to fail. 
This CL is a poor victim of earlier bugs. But, CQ must move on...

Original change's description:
> [moblab] Move the moblab setup from the host provision to the test.
> 
> In the aim to have moblab issue show up as test failures vs
> provision failures tests should setup the moblab.
> 
> BUG= chromium:749325 
> TEST=trybot https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/paladin/builds/3451
> 
> Change-Id: I75e486d4726a33243e4e3238b216b33df1838ad8
> Reviewed-on: https://chromium-review.googlesource.com/594663
> Commit-Ready: Keith Haddow <haddowk@chromium.org>
> Tested-by: Keith Haddow <haddowk@chromium.org>
> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
> Reviewed-by: Keith Haddow <haddowk@chromium.org>

Bug= chromium:749325 
BUG= chromium:751895 

Change-Id: Id2dbfc67a8c108695fb5d42d7bac1ba9e0a9457b
Reviewed-on: https://chromium-review.googlesource.com/599356
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/26933a0b0dd9642300ef4e83ca9b31f5ccf82352/server/cros/moblab_test.py
[modify] https://crrev.com/26933a0b0dd9642300ef4e83ca9b31f5ccf82352/server/hosts/moblab_host.py

Project Member

Comment 9 by bugdroid1@chromium.org, Aug 4 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/d7cf1a006871030854ecbb6c9325b5df9010aaf9

commit d7cf1a006871030854ecbb6c9325b5df9010aaf9
Author: Keith Haddow <haddowk@chromium.org>
Date: Fri Aug 04 22:39:11 2017

[moblab] Move the moblab setup from the host provision to the test.

In the aim to have moblab issue show up as test failures vs
provision failures tests should setup the moblab.

BUG= chromium:749325 
TEST=test_that testing

Change-Id: I6534eae1ae6bb65044c3486aa96f4e5aed6be8e0
Reviewed-on: https://chromium-review.googlesource.com/601587
Commit-Ready: Keith Haddow <haddowk@chromium.org>
Tested-by: Keith Haddow <haddowk@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Keith Haddow <haddowk@chromium.org>

[modify] https://crrev.com/d7cf1a006871030854ecbb6c9325b5df9010aaf9/server/cros/moblab_test.py
[modify] https://crrev.com/d7cf1a006871030854ecbb6c9325b5df9010aaf9/server/hosts/moblab_host.py

Status: Fixed (was: Untriaged)

Sign in to add a comment