New issue
Advanced search Search tips

Issue 883017 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Sep 19
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

VMTest: Retried test succeeds, but VMTest fails

Project Member Reported by pprabhu@chromium.org, Sep 11

Issue description

We now retry tests within VMTest to reduce flake. But it seems like VMTest fails even when a test passes on retry: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8936666828069938640

Fixit.
 
Labels: -Pri-2 Pri-1
Let's fix this in a week or so to get some much needed VMTest stability back.
FTR: test level retries were added in  issue 750918 
Status: Started (was: Untriaged)
The root cause is that generate_test_report does not dedupe multiple retries of the same test and returns an error whenever it encounters a fail in one of the retries: https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/site_utils/generate_test_report?rcl=f23305daccc73abd1bf6b70344832baf96ae4f84&l=853

I am working on the fix now.
Project Member

Comment 5 by bugdroid1@chromium.org, Sep 18

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/2d9f821b7c04763fb3a5e4faa645751c63aa31a6

commit 2d9f821b7c04763fb3a5e4faa645751c63aa31a6
Author: Alex Zamorzaev <zamorzaev@chromium.org>
Date: Tue Sep 18 21:04:01 2018

generate_test_report: dedupe test retries

Before: if a test fails generate_test_report returns a non-zero value
regardless of whether any of the retries succeeded.

After: generate_test_report returns a non-zero value only if *all*
retries failed for one of the tests.

BUG= chromium:883017 
TEST=locally using suite:dummy and flaky tests

Change-Id: Ia03006916221556b190cb50da841fc55a1c72538
Reviewed-on: https://chromium-review.googlesource.com/1226157
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Alex Zamorzaev <zamorzaev@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Jacob Kopczynski <jkop@chromium.org>

[modify] https://crrev.com/2d9f821b7c04763fb3a5e4faa645751c63aa31a6/site_utils/generate_test_report
[modify] https://crrev.com/2d9f821b7c04763fb3a5e4faa645751c63aa31a6/site_utils/test_runner_utils_unittest.py

Status: Fixed (was: Started)
Project Member

Comment 7 by bugdroid1@chromium.org, Sep 21

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/6844cfed6130d70824555327475f61af73078ace

commit 6844cfed6130d70824555327475f61af73078ace
Author: Alex Zamorzaev <zamorzaev@chromium.org>
Date: Fri Sep 21 04:30:09 2018

generate_test_report: unit tests for status code

The new unit tests verify that generate_test_report can correctly dedupe
retries of the same test case.

BUG= chromium:883017 
TEST=run unit test locally

Change-Id: I71948e0d5b3200ba6b3d5a8fc53670b237330eac
Reviewed-on: https://chromium-review.googlesource.com/1229376
Commit-Ready: Alex Zamorzaev <zamorzaev@chromium.org>
Tested-by: Alex Zamorzaev <zamorzaev@chromium.org>
Reviewed-by: Alex Zamorzaev <zamorzaev@chromium.org>
Reviewed-by: Jacob Kopczynski <jkop@chromium.org>

[modify] https://crrev.com/6844cfed6130d70824555327475f61af73078ace/site_utils/test_runner_utils_unittest.py

Sign in to add a comment