New issue
Advanced search Search tips

Issue 659906 link

Starred by 1 user
Status: Fixed
Owner: ----
Closed: Oct 2016
Cc:
dtu@chromium.org
simonhatch@chromium.org
st...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Infinite* loop in bisect

Project Member Reported by robert...@chromium.org, Oct 27 2016 
Because checking for sufficient sample size to declare that two revisions are not different has been delegated to compare_samples in catapult, the refactored bisect is now breaking when re-testing a revision that produces no comparable values. (As it defaults to NEED_MORE_DATA)

E.g.
https://build.chromium.org/p/tryserver.chromium.perf/builders/staging_mac_10_10_perf_bisect/builds/20

Bisect needs to either:
A) Set an upper limit to the number of times a revision is re-tested regardless of the number of observations yielded by the test, or
B) Give up if either sample is empty after a fixed number of tests.

Comment 1 by dtu@chromium.org, Oct 28 2016 

Hm what is causing it to produce no comparable values?

Comment 2 by robert...@chromium.org, Oct 28 2016 

It could be a missing metric. Or simply test failures.

Comment 3 by dtu@chromium.org, Oct 28 2016 

Ah, yeah, we should set a limit on the number of retests per revision.
Project Member

Comment 4 by bugdroid1@chromium.org, Oct 31 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/d8aa54bdb96167b4fd89ad1c5627fecadd18b21c

commit d8aa54bdb96167b4fd89ad1c5627fecadd18b21c
Author: robertocn <robertocn@chromium.org>
Date: Mon Oct 31 20:39:53 2016

Setting a maximum number of test runs per revision.

The chosen value is 20 runs per revision. The reasoning is that 20 is a
round number that should produce no less than 18 values (one per run, in
a test that is at most 10% flaky) which has in the past seemed
sufficient to decide that the values are unlikely to converge per MWU.

Also, adding test coverage and fixing a small bug highlighted by this
change. (See bisector.py one line change)

R=dtu@chromium.org
BUG= 659906 

Review-Url: https://codereview.chromium.org/2469503002

[modify] https://crrev.com/d8aa54bdb96167b4fd89ad1c5627fecadd18b21c/scripts/slave/recipe_modules/auto_bisect_staging/bisector.py
[modify] https://crrev.com/d8aa54bdb96167b4fd89ad1c5627fecadd18b21c/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/failed_build.json
[modify] https://crrev.com/d8aa54bdb96167b4fd89ad1c5627fecadd18b21c/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/failed_buildbucket_get.json
[modify] https://crrev.com/d8aa54bdb96167b4fd89ad1c5627fecadd18b21c/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/failed_reference_range.json
[modify] https://crrev.com/d8aa54bdb96167b4fd89ad1c5627fecadd18b21c/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/failed_revision.json
[add] https://crrev.com/d8aa54bdb96167b4fd89ad1c5627fecadd18b21c/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/no_values.json
[modify] https://crrev.com/d8aa54bdb96167b4fd89ad1c5627fecadd18b21c/scripts/slave/recipe_modules/auto_bisect_staging/example.py
[modify] https://crrev.com/d8aa54bdb96167b4fd89ad1c5627fecadd18b21c/scripts/slave/recipe_modules/auto_bisect_staging/revision_state.py
[modify] https://crrev.com/d8aa54bdb96167b4fd89ad1c5627fecadd18b21c/scripts/slave/recipe_modules/auto_bisect_staging/test_api.py
[modify] https://crrev.com/d8aa54bdb96167b4fd89ad1c5627fecadd18b21c/scripts/slave/recipes/bisection/android_bisect_staging.expected/failure_in_ref_range.json

Comment 5 by robert...@chromium.org, Oct 31 2016

Status: Fixed (was: Untriaged)

Comment 6 by benhenry@chromium.org, Feb 3 2017

Components: Speed>Bisection
►

Sign in to add a comment