Create findit/chromium/flake recipe rerun a gtest N times |
|||||
Issue description
Context here:
1. The findit/chromium/test.py recipe already runs selected tests in _parallel_ Swarming tasks if the step (like browser_tests) is configured with multiple tasks.
2. For a base/test/launcher-based gtest binary (like browser_tests), we've made it possible to run a test N times in a _single_ Swarming task with these flags:
--gtest_filter=AppMenuControllerTest.RecentTabsFavIcon --gtest_repeat=100 --test-launcher-retry-limit=0 --gtest_also_run_disabled_tests
The specific missing parts here are:
1. How to pass the flags "--gtest_repeat=100 --test-launcher-retry-limit=0 --gtest_also_run_disabled_tests" from the recipe, into recipe_modules/chromium_tests, all the way down to recipe_modules/swarming, and then send them over the wire to Swarming sever ?
2. How to retrieve the number of passed/failed runs among the 100 iteration? We wanted to calculate the pass rate or flakiness.
,
Dec 14 2016
,
Dec 14 2016
,
Dec 22 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/12817dce9c8c4def8f3c0716fda7892bf45a4d28 commit 12817dce9c8c4def8f3c0716fda7892bf45a4d28 Author: Roberto Carrillo <robertocn@google.com> Date: Thu Dec 22 09:58:35 2016 Add support for flakiness testing GTests. Add a new recipe that runs a test a number of times (100) by default for a single revision and reports the number of times the test passes/fails. This change supports GTests both local and swarmed and reuses as much code from test.py as possible. R=stgao@chromium.org, phajdan.jr@chromium.org, iannucci@chromium.org BUG= 672721 Change-Id: I646ec35657b841fdc8ae4270aac6a19ca4b2d84d Reviewed-on: https://chromium-review.googlesource.com/418875 Commit-Queue: Roberto Carrillo <robertocn@chromium.org> Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org> [modify] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipe_modules/chromium_tests/steps.py [modify] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipe_modules/findit/__init__.py [modify] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipe_modules/findit/api.py [add] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipes/findit/chromium/flake.expected/flakiness_non-swarming_tests.json [add] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipes/findit/chromium/flake.expected/flakiness_swarming_tests.json [add] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipes/findit/chromium/flake.expected/record_infra_failure.json [add] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipes/findit/chromium/flake.expected/use_build_parameter_for_tests.json [add] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipes/findit/chromium/flake.py [modify] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipes/findit/chromium/test.py
,
Jan 3 2017
,
Jan 9 2017
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by st...@chromium.org
, Dec 9 2016