New issue
Advanced search Search tips

Issue 758315 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Aug 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: ----
Type: ----



Sign in to add a comment

platform_Powerwash: reef-uni-release:186 failed: Powerwash count didn't increase after powerwash cycle.

Project Member Reported by briannorris@chromium.org, Aug 23 2017

Issue description

reef-uni-release:186 failed

Builders failed on: 
- reef-uni-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/reef-uni-release/186


08/23 07:22:01.896 DEBUG|          ssh_host:0296| Running (ssh) 'cat /mnt/stateful_partition/unencrypted/preserve/powerwash_count' from '_call_run_once|run_once|_powerwash_count|run|wrapper|run_very_slowly'
08/23 07:22:54.384 DEBUG|             utils:0280| [stderr] Read from socket failed: Connection reset by peer
08/23 07:22:54.392 DEBUG|platform_Powerwash:0080| Powerwash count is: ''
08/23 07:22:54.393 DEBUG|              test:0389| Test failed due to Powerwash count didn't increase after powerwash cycle.. Exception log follows the after_iteration_hooks.
08/23 07:22:54.394 DEBUG|              test:0392| starting after_iteration_hooks
08/23 07:22:54.394 DEBUG|              test:0395| after_iteration_hooks completed
08/23 07:22:54.395 WARNI|              test:0612| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/test.py", line 606, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 806, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 470, in execute
    dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 347, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 380, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/server/site_tests/platform_Powerwash/platform_Powerwash.py", line 61, in run_once
    raise error.TestFail("Powerwash count didn't increase after "
TestFail: Powerwash count didn't increase after powerwash cycle.



Looks like possibly a network flake (see "Read from socket failed: Connection reset by peer") of some kind. But the test is also pretty poor in handling this, as it ignores the SSH exit status. At a minimum, we could fix this to report the right error.

I'm not sure if this is a device issue we should look into also.
 
Project Member

Comment 2 by bugdroid1@chromium.org, Aug 26 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/660fda154038687080e35a25dfe07fd27b9fca5e

commit 660fda154038687080e35a25dfe07fd27b9fca5e
Author: Brian Norris <briannorris@chromium.org>
Date: Sat Aug 26 04:52:47 2017

platform_Powerwash: make network connection failures more obvious

This test can fail due to network issues (e.g., "Read from socket
failed: Connection reset by peer") but still end up reporting this
result:

TestFail: Powerwash count didn't increase after powerwash cycle.

That's a bit cryptic when reading the top-level builder results. It's
also bad if we just assume that the value was 0 when it was actually
unknown.

This should all be clearer if we just let the SSH command fail, and
explicitly test for the one failure we want to ignore: when the
POWERWASH_COUNT file is missing.

BUG= chromium:758315 
TEST=platform_Powerwash

Change-Id: I3427331d32445f79de8a1223b895ba465d3d2044
Signed-off-by: Brian Norris <briannorris@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/629756
Reviewed-by: Andrey Pronin <apronin@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/660fda154038687080e35a25dfe07fd27b9fca5e/server/site_tests/platform_Powerwash/platform_Powerwash.py

Status: Fixed (was: Assigned)
Test should report the SSH error properly now, AFAICT.

Comment 4 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Sign in to add a comment