New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 910706 link

Starred by 1 user

Issue metadata

Status: Untriaged
Owner:
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

--gtest_repeat is being set without --gtest_filter. This causes test timeouts.

Project Member Reported by erikc...@chromium.org, Nov 30

Issue description

Build: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win10_chromium_x64_rel_ng/149779

Swarming task [shard 6] passes all tests, and emits output.json but counts as a swarming failure because it failed to shut down correctly:
https://chromium-swarm.appspot.com/task?id=417ea8baeaea5a10

This in turn causes all tests to be rerun with --gtest_repeat=10, but no --gtest_filter, causing timeouts everywhere.

More detailed explanation:
The swarming failure causes the SwarmingGtestTest to be considered a failed test:
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/test_utils/api.py?type=cs&q=scripts/slave/recipe_modules/test_utils/api.py&sq=package:chromium&g=0&l=461

However, the test results are valid and there are no failing tests. This causes tests_to_retry to return an empty list:
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/steps.py?type=cs&q=recipe_modules/chromium_tests/steps.py&sq=package:chromium&g=0&l=388

Which causes --gtest_filter to not be set:
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/steps.py?type=cs&q=recipe_modules/chromium_tests/steps.py&sq=package:chromium&g=0&l=1825

However, since the tests_to_retry is [], it fails to be caught in our earlier exit block for _test_options_for_running:
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/steps.py?type=cs&q=recipe_modules/chromium_tests/steps.py&sq=package:chromium&g=0&l=129


There are two problems to fix. 
1) We should update _test_options_for_running to also catch the empty list.
2) If test results are valid, and there are no failures, it's possible we should mark the test suite as a success even if there's an infra failure?
 
Project Member

Comment 1 by bugdroid1@chromium.org, Nov 30

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/7450e9d90eb27398718b9c2f579fbeb82b94e5f9

commit 7450e9d90eb27398718b9c2f579fbeb82b94e5f9
Author: erikchen <erikchen@chromium.org>
Date: Fri Nov 30 22:42:10 2018

Fix edge case where --gtest_repeat is set without --gtest_filter.

In rare circumstances, it's possible for test results to be valid, and all tests
to pass, but for the test suite to be considered a failure due to an infra
issue. This can cause the list of failed tests to be empty. In this case,
retries will rerun all tests. We should avoid setting --gtest_repeat.

Bug: 910706
Change-Id: If81cb64bbb391a9883b57b1a465d0d0b708fc926
Reviewed-on: https://chromium-review.googlesource.com/c/1357281
Commit-Queue: Erik Chen <erikchen@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/7450e9d90eb27398718b9c2f579fbeb82b94e5f9/scripts/slave/recipe_modules/chromium_tests/tests/steps/local_gtest_test.expected/android.json
[modify] https://crrev.com/7450e9d90eb27398718b9c2f579fbeb82b94e5f9/scripts/slave/recipe_modules/chromium_tests/tests/steps/local_gtest_test.expected/basic.json
[modify] https://crrev.com/7450e9d90eb27398718b9c2f579fbeb82b94e5f9/scripts/slave/recipe_modules/chromium_tests/tests/api/run_tests_on_tryserver.expected/nonzero_exit_code_no_gtest_output.json
[modify] https://crrev.com/7450e9d90eb27398718b9c2f579fbeb82b94e5f9/scripts/slave/recipe_modules/chromium_tests/steps.py

Labels: Infra-Platform-Test
Owner: erikc...@chromium.org

Sign in to add a comment