New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 696882 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Last visit > 30 days ago
Closed: Mar 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

test_push: Not all DUTs used in the test are reverified

Project Member Reported by pprabhu@chromium.org, Feb 28 2017

Issue description

test_push reverifies DUTs before starting creating suite jobs to ensure that they are in a good state. This is doubly important because we run powerwash test that explicitly leaves them without python.


But,
- we only verify DUTs in pool:bvt
- But we do not require the DUTs used for the test to be from the bvt pool.

In particular, the powerwash test here used a DUT that was not in the bvt pool. So it needed to repair the DUT, leading to timeout of the test.

Aborted test job: http://chromeos-autotest.hot.corp.google.com/afe/#tab_id=view_job&object_id=5929

DUT used: http://chromeos-autotest.hot.corp.google.com/afe/#tab_id=view_host&object_id=31

I don't see any reason we shouldn't just reverify all DUTs on the test master / shards before the test.
 

Comment 1 by xixuan@chromium.org, Feb 28 2017

Cc: xixuan@chromium.org
At least for shard, pool:bvt is required for test_push's tests for every shard DUT. I found the push_shards won't match jobs (whose deps includes 'pool:bvt') to hosts if these hosts are lack of 'pool:bvt' label, which blocks test_push for long long time.

So why don't we add pool:bvt to all DUTs ?




Re #1: Note that the powerwash test is launched individually (not part of a suite) and does not have the pool:bvt dependency.

Agreed: I do not know why we don't have pool:bvt on all DUTs.
There are some cases that developer adds a DUT to push server for testing. In this case, the DUT will not have the pool:bvt label.

Comment 4 by xixuan@chromium.org, Feb 28 2017

I assume we only need to reverify DUTs which is used for testing? If all testing DUT are forced to be in pool:BVT then we're fine.


Furthermore we should at least show errors like 'no host is with pool:bvt' if DUTs doesn't qualify and fail directly (I remember we have this feature before? ), or automatically add pool:bvt to all test_push DUTs.

Now the shard's jobs are just queued there and wait for timed out (several hours). When deputy checks it, it's very hard for deputy to find which element is preventing its running. Deputy need to debug very deep and find it's 'pool:bvt' missing and add it to all quawks DUT, which is exactly my hard life at last night :( 
Project Member

Comment 5 by bugdroid1@chromium.org, Mar 4 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/a6cf66b1511713ba221e1365c968887f12eded6c

commit a6cf66b1511713ba221e1365c968887f12eded6c
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Sat Mar 04 04:56:42 2017

[autotest] check the num of required DUTs in the bvt pool

DUTs in pool:bvt is used in test_push. Currently we check all the
DUTs in the push server to ensure we have the minimum number of DUTs in
the Ready state. However, not all the DUTs are in pool:bvt. Enforce the
check to only check the DUTs in pool:bvt, and improve the error message
to explicitly mention the DUTs in pool:bvt are in shortage.

BUG= chromium:696882 
TEST=unittest and also test in testing server

Change-Id: I34711b3f8315ca2a24d2e9162232b052251e058b
Reviewed-on: https://chromium-review.googlesource.com/449873
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/a6cf66b1511713ba221e1365c968887f12eded6c/site_utils/test_push.py

Labels: -current-issue
Project Member

Comment 7 by bugdroid1@chromium.org, Mar 8 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/06deae044b28742dbd420096961507c1c01c250b

commit 06deae044b28742dbd420096961507c1c01c250b
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Wed Mar 08 02:07:35 2017

[autotest] reverify all the DUTs in push master not only the ones in bvt

Reverify all the DUTs in push master.

BUG= chromium:696882 
TEST=unittest and test in push master

Change-Id: I1b8585158a11dddfc4d37bf163570f43fbe2f87a
Reviewed-on: https://chromium-review.googlesource.com/447856
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/06deae044b28742dbd420096961507c1c01c250b/site_utils/test_push.py

Project Member

Comment 8 by bugdroid1@chromium.org, Mar 8 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/06deae044b28742dbd420096961507c1c01c250b

commit 06deae044b28742dbd420096961507c1c01c250b
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Wed Mar 08 02:07:35 2017

[autotest] reverify all the DUTs in push master not only the ones in bvt

Reverify all the DUTs in push master.

BUG= chromium:696882 
TEST=unittest and test in push master

Change-Id: I1b8585158a11dddfc4d37bf163570f43fbe2f87a
Reviewed-on: https://chromium-review.googlesource.com/447856
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/06deae044b28742dbd420096961507c1c01c250b/site_utils/test_push.py

Status: Fixed (was: Assigned)

Comment 10 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61

Comment 12 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Sign in to add a comment