In recipe_modules/auto_bisect/revision_state.py there is now a "minimum repeat count", which has been there since https://codereview.chromium.org/1610203003 ("Iteratively increase sample size for good/bad classification"), which was meant to make testing repeat until there's "enough evidence" to classify a revision as good or bad.

But return code bisects are a special and interesting case, because if the test doesn't fail flakily, then "enough evidence" is on only one run. Maybe sample size should *not* be iteratively increased in the case of return-code bisects?

Roberto, Dave, WDYT?

 Issue 592691  has been merged into this issue.

There's a question of whether or not the increased code complexity of a special case is worth it. The benefit would be reduced latency/runtime, but we've had problems in the past with the differences between the code paths of return code and perf bisect.

I'd also like bisect to work with flaky test failures, in which case you'd still need the statistical classification.

Hm, I could see us wanting it for generalized bisect, so maybe it is worth the complexity of a special case. But maybe not at this point in time?

By "generalized bisect," I mean locally bisecting on arbitrary functional regressions, like  issue 592189  or  issue 593614 .

I am going to make a patch to do the exact repeat count for 'return_code' bisects.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/8e3799f5d53dc21d72d7176cc80ffb35775d5921

commit 8e3799f5d53dc21d72d7176cc80ffb35775d5921
Author: robertocn@chromium.org <robertocn@chromium.org>
Date: Tue Mar 29 23:52:49 2016

Respecting repeat count for non perf bisects.

R=prasadv
BUG= 593055 

Review URL: https://codereview.chromium.org/1843023002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/build@299551 0039d316-1c4b-4281-b951-d872f2087c98

[modify] https://crrev.com/8e3799f5d53dc21d72d7176cc80ffb35775d5921/scripts/slave/recipe_modules/auto_bisect/revision_state.py
[modify] https://crrev.com/8e3799f5d53dc21d72d7176cc80ffb35775d5921/scripts/slave/recipes/bisect.expected/basic_return_code_test.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/8e3799f5d53dc21d72d7176cc80ffb35775d5921

commit 8e3799f5d53dc21d72d7176cc80ffb35775d5921
Author: robertocn@chromium.org <robertocn@chromium.org>
Date: Tue Mar 29 23:52:49 2016

Respecting repeat count for non perf bisects.

R=prasadv
BUG= 593055 

Review URL: https://codereview.chromium.org/1843023002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/build@299551 0039d316-1c4b-4281-b951-d872f2087c98

[modify] https://crrev.com/8e3799f5d53dc21d72d7176cc80ffb35775d5921/scripts/slave/recipe_modules/auto_bisect/revision_state.py
[modify] https://crrev.com/8e3799f5d53dc21d72d7176cc80ffb35775d5921/scripts/slave/recipes/bisect.expected/basic_return_code_test.json