New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 635838 link

Starred by 0 users

Issue metadata

Status: Archived
Owner: ----
Closed: Sep 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocking:
issue 633193
issue 633538



Sign in to add a comment

Bisect ignores repeat_count and max_time_minutes

Project Member Reported by petrcermak@chromium.org, Aug 9 2016

Issue description

I've run two bisects with different parameters (for the same regression,  issue 633193 ):

Bisect 1: https://chromeperf.appspot.com/buildbucket_job_status/9005529157844552592
max_time_minutes: 20
repeat_count: 10

Bisect 2: https://chromeperf.appspot.com/buildbucket_job_status/9004900681642967744
max_time_minutes: 100
repeat_count: 100

Nevertheless, the benchmark was run the same number of times in both cases:

Bisect 1: https://bugs.chromium.org/p/chromium/issues/detail?id=633193#c3
Revision         Mean    Std Dev  N  Good?
chromium@407992  464392  1022.11  8  good
chromium@408012  463840  0.0      8  bad

Bisect 2: https://bugs.chromium.org/p/chromium/issues/detail?id=633193#c7
Revision         Mean    Std Dev  N  Good?
chromium@407954  463840  0.0      8  good
chromium@408012  465062  3456.34  8  bad

Given that I increased max_time_minutes 5 times and repeat_count 10 times, I would expect that the second bisect would run each revision ~40 times (5×8).
 
I've just noticed that the devices for the two bisects were one:

Bisect 1: android_nexus7_perf_bisect
Bisect 2: android_nexus5_perf_bisect

Nevertheless, I wouldn't expect N5 to be 5× slower than N7.
Blocking: 633538
Cc: dtu@chromium.org
+dtu

The bisect dialog wasn't updated with the new classification algorithm and it's really confusing people. This is logged as https://github.com/catapult-project/catapult/issues/2602 on the catapult side.

Dave, Roberto, can you explain how the new classification algorithm works wrt repeats?

Comment 5 by dtu@chromium.org, Sep 13 2017

Status: Archived (was: Untriaged)

Sign in to add a comment