New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 619107 link

Starred by 2 users

Issue metadata

Status: Archived
Owner:
Last visit > 30 days ago
Closed: Sep 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: ----



Sign in to add a comment

Add powerwash as a test for testing DUT in push-to-prod

Project Member Reported by xixuan@chromium.org, Jun 10 2016

Issue description

In push-to-prod, every first verify will fail due to a "shadow" powerwash, which leads to a confusing work flow:
repair -- completed
verify -- fail, Python is missing; may be caused by powerwash
repair -- completed

The powerwash actually exist in the test_push.py, not show in the testing history of the DUT, which is confused and hard to tell.

Example host: http://chromeos-autotest.hot.corp.google.com/afe/#tab_id=view_host&object_id=52

It's better to add powerwash as a test for DUT, so the debug process is not that confused.
 
Why we need powerwash to make sure repair works?
Why the verify will fail due to the powerwash? Is the reverify designed to fail so that a repair can be triggered?

Comment 2 by dshi@chromium.org, Jun 10 2016

The purpose of powerwash is to force a repair job which will do a cros install.
Then why we need a verify job after the powerwash, if a repair is on its way?

Comment 4 by xixuan@chromium.org, Jun 10 2016

in the current workflow, 

"powerwash" -- completed
verify -- fail, due to powerwash
repair -- completed (do the cros auto-update)
provision -- completed (do the cros auto-update)

auto-update will be executed twice, is it necessary?
Re the questions in comments #1 and #3:  Powerwash wipes out
all of /usr/local, including /usr/local/bin/python.  Verify
includes a check for a working 'python' command, so after
Powerwash, a DUT will fail verification.

Repair includes a rule that says "if the Python verifier failed,
reinstall", so after powerwash, we'll force repair, and the DUT
will get fixed.  *IF* repair is working, that is.

What, if any, was the action decided based on this bug?
There's been no action.

IIUC, I believe the change requested is to move the existing
powerwash from a low-visibility code path in test_push.py into
the more visible code represented by the push_to_prod suite.

Comment 8 by dshi@chromium.org, Aug 23 2016

Meanwhile, we don't want to loose coverage on testing repair workflow. The powerwash test itself doesn't go through repair special task.
Hmmm...  Yes, sort of.  Powerwash doesn't go through repair, but
after powerwash, if you verify a DUT, it will force repair.

That said, if we move the powerwash to the be part of the suite,
we'd lose some control over detecting DUT failures.  But really,
we should make looking at DUT failures part of push to prod.
That is, we should make sure that after push to prod, all DUTs
verify and repair cleanly.

Aside: If we forced verify/repair at the end of push to prod, it
would also tend to clean up the problems we experience with TPM
problems at the start of push to prod testing.

Issue 640234 has been merged into this issue.
Cc: nxia@chromium.org
+ @nxia

This issue hit ningning's test_push again. quawks failed finishing 'repair & provision & a test' in 30 minutes.
Project Member

Comment 12 by bugdroid1@chromium.org, Sep 13 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/327b695ce76a4ca8dbb1d05e36f097da36459936

commit 327b695ce76a4ca8dbb1d05e36f097da36459936
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Mon Sep 12 17:42:03 2016

test_push: replace the unobvious powerwash with obvious powerwash test

In order to test the repairt workflow, there is a powerwash scheduled on
shards before running any tests. However, this powerwash is not a test,
so there is no clues on the afe to tell there is a powerwash on the
host. Replace it with a powerwash test, so we have logs for it.

BUG= chromium:619107 
TEST=Test on testing push master.

Change-Id: I82d238fb992375ab8c9afab2e35ed2bc1853d08d
Reviewed-on: https://chromium-review.googlesource.com/384614
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Kevin Cheng <kevcheng@chromium.org>

[modify] https://crrev.com/327b695ce76a4ca8dbb1d05e36f097da36459936/site_utils/test_push.py
[modify] https://crrev.com/327b695ce76a4ca8dbb1d05e36f097da36459936/site_utils/test_push_unittest.py

Status: Fixed (was: Assigned)
Labels: VerifyIn-55

Comment 15 by dchan@chromium.org, Oct 10 2016

Labels: -VerifyIn-55

Comment 16 by dchan@google.com, Nov 19 2016

Labels: VerifyIn-56

Comment 17 by dchan@google.com, Jan 21 2017

Labels: VerifyIn-57

Comment 18 by dchan@google.com, Mar 4 2017

Labels: VerifyIn-58

Comment 19 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 20 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61

Comment 22 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)

Sign in to add a comment