New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 672721 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Jan 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocking:
issue 666097
issue 671735



Sign in to add a comment

Create findit/chromium/flake recipe rerun a gtest N times

Project Member Reported by st...@chromium.org, Dec 9 2016

Issue description

Context here:
1. The findit/chromium/test.py recipe already runs selected tests in _parallel_ Swarming tasks if the step (like browser_tests) is configured with multiple tasks.
2. For a base/test/launcher-based gtest binary (like browser_tests), we've made it possible to run a test N times in a _single_ Swarming task with these flags:
    --gtest_filter=AppMenuControllerTest.RecentTabsFavIcon --gtest_repeat=100 --test-launcher-retry-limit=0 --gtest_also_run_disabled_tests

The specific missing parts here are:
1. How to pass the flags  "--gtest_repeat=100 --test-launcher-retry-limit=0 --gtest_also_run_disabled_tests" from the recipe, into recipe_modules/chromium_tests, all the way down to recipe_modules/swarming, and then send them over the wire to Swarming sever ?
2. How to retrieve the number of passed/failed runs among the 100 iteration? We wanted to calculate the pass rate or flakiness.
 

Comment 1 by st...@chromium.org, Dec 9 2016

Blocking: 672724

Comment 2 by st...@chromium.org, Dec 14 2016

Blocking: 671735

Comment 3 by st...@chromium.org, Dec 14 2016

Summary: Create findit/chromium/flake recipe rerun a gtest N times (was: Make findit/chromium/test recipe rerun a gtest N times)
Project Member

Comment 4 by bugdroid1@chromium.org, Dec 22 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/12817dce9c8c4def8f3c0716fda7892bf45a4d28

commit 12817dce9c8c4def8f3c0716fda7892bf45a4d28
Author: Roberto Carrillo <robertocn@google.com>
Date: Thu Dec 22 09:58:35 2016

Add support for flakiness testing GTests.

Add a new recipe that runs a test a number of times (100) by default for a single revision
and reports the number of times the test passes/fails.

This change supports GTests both local and swarmed and reuses as much code from test.py as
possible.

R=stgao@chromium.org, phajdan.jr@chromium.org, iannucci@chromium.org
BUG= 672721 

Change-Id: I646ec35657b841fdc8ae4270aac6a19ca4b2d84d
Reviewed-on: https://chromium-review.googlesource.com/418875
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>

[modify] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipe_modules/chromium_tests/steps.py
[modify] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipe_modules/findit/__init__.py
[modify] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipe_modules/findit/api.py
[add] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipes/findit/chromium/flake.expected/flakiness_non-swarming_tests.json
[add] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipes/findit/chromium/flake.expected/flakiness_swarming_tests.json
[add] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipes/findit/chromium/flake.expected/record_infra_failure.json
[add] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipes/findit/chromium/flake.expected/use_build_parameter_for_tests.json
[add] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipes/findit/chromium/flake.py
[modify] https://crrev.com/12817dce9c8c4def8f3c0716fda7892bf45a4d28/scripts/slave/recipes/findit/chromium/test.py

Status: Fixed (was: Assigned)
Blocking: -672724

Sign in to add a comment