VMTest: Retried test succeeds, but VMTest fails |
|||
Issue descriptionWe now retry tests within VMTest to reduce flake. But it seems like VMTest fails even when a test passes on retry: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8936666828069938640 Fixit.
,
Sep 11
FTR: test level retries were added in issue 750918
,
Sep 11
,
Sep 13
The root cause is that generate_test_report does not dedupe multiple retries of the same test and returns an error whenever it encounters a fail in one of the retries: https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/site_utils/generate_test_report?rcl=f23305daccc73abd1bf6b70344832baf96ae4f84&l=853 I am working on the fix now.
,
Sep 18
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/2d9f821b7c04763fb3a5e4faa645751c63aa31a6 commit 2d9f821b7c04763fb3a5e4faa645751c63aa31a6 Author: Alex Zamorzaev <zamorzaev@chromium.org> Date: Tue Sep 18 21:04:01 2018 generate_test_report: dedupe test retries Before: if a test fails generate_test_report returns a non-zero value regardless of whether any of the retries succeeded. After: generate_test_report returns a non-zero value only if *all* retries failed for one of the tests. BUG= chromium:883017 TEST=locally using suite:dummy and flaky tests Change-Id: Ia03006916221556b190cb50da841fc55a1c72538 Reviewed-on: https://chromium-review.googlesource.com/1226157 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Alex Zamorzaev <zamorzaev@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Jacob Kopczynski <jkop@chromium.org> [modify] https://crrev.com/2d9f821b7c04763fb3a5e4faa645751c63aa31a6/site_utils/generate_test_report [modify] https://crrev.com/2d9f821b7c04763fb3a5e4faa645751c63aa31a6/site_utils/test_runner_utils_unittest.py
,
Sep 19
,
Sep 21
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/6844cfed6130d70824555327475f61af73078ace commit 6844cfed6130d70824555327475f61af73078ace Author: Alex Zamorzaev <zamorzaev@chromium.org> Date: Fri Sep 21 04:30:09 2018 generate_test_report: unit tests for status code The new unit tests verify that generate_test_report can correctly dedupe retries of the same test case. BUG= chromium:883017 TEST=run unit test locally Change-Id: I71948e0d5b3200ba6b3d5a8fc53670b237330eac Reviewed-on: https://chromium-review.googlesource.com/1229376 Commit-Ready: Alex Zamorzaev <zamorzaev@chromium.org> Tested-by: Alex Zamorzaev <zamorzaev@chromium.org> Reviewed-by: Alex Zamorzaev <zamorzaev@chromium.org> Reviewed-by: Jacob Kopczynski <jkop@chromium.org> [modify] https://crrev.com/6844cfed6130d70824555327475f61af73078ace/site_utils/test_runner_utils_unittest.py |
|||
►
Sign in to add a comment |
|||
Comment 1 by pprabhu@chromium.org
, Sep 11