When running tests in 'with patch', we want to ignore failures if there is a single success.
When running tests in 'without patch', we want to ignore successes if there is a single failure.
This works, but only with some very convoluted logic.
For GTests, the relevant logic is at:
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/test_utils/api.py?type=cs&q=scripts/slave/recipe_modules/test_utils/api.py&sq=package:chromium&g=0&l=221
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/steps.py?type=cs&q=scripts/slave/recipe_modules/chromium_tests/steps.py&sq=package:chromium&g=0&l=355
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/steps.py?type=cs&q=scripts/slave/recipe_modules/chromium_tests/steps.py&sq=package:chromium&g=0&l=1572
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/test_utils/util.py?type=cs&q=scripts/slave/recipe_modules/test_utils/util.py&sq=package:chromium&g=0&l=263
The line that causes us to ignore all but the last test in an iteration is:
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/test_utils/util.py?type=cs&q=scripts/slave/recipe_modules/test_utils/util.py&sq=package:chromium&g=0&l=259
This logic happens to work because in 'with patch', retries are reported in a single iteration, but for 'without patch', retries are reported in multiple iterations.
We should make this explicit. self.passes and self.failures should present exactly what they suggest [every pass and failure], and we should add some custom logic to run_tests_with_patch to cause us to look for a single success, and etc. for 'without patch', 'retry with patch'.
Comment 1 by erikc...@chromium.org
, Jan 2