This happens when all intermediate builds fail to compile, how does the following message sound?

"Bisect failed to identify a culprit because all revisions between the last good revision and first bad revision in the table below failed to be built."

Would it also make sense to link to the buildjobs that failed?

The most important thing is to clearly list the range of revisions that cannot be built. Listing the build jobs that failed in addition would be helpful.

I would have thought this would be a failure, is green status intended?

The green status is not intended. It's bad behavior. All non-culprit-found scenarios should be red.

CL ready for review: https://chromium-review.googlesource.com/411502

I think https://bugs.chromium.org/p/chromium/issues/detail?id=664134#c9 is also caused by this?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/5dca59c419492fcdd739d3fe19f7fe955238a2ab

commit 5dca59c419492fcdd739d3fe19f7fe955238a2ab
Author: Roberto Carrillo <robertocn@google.com>
Date: Tue Nov 15 22:29:18 2016

No-culprit bisects due to bad builds explicit message and status.

When a bisect fails to find a culprit because all revisions between the
last known good and first known bad failed to build, provide a
descriptive message and provide details of the failed builds if their
number is manageable (<= 10)

R=sullivan@chromium.org,dtu@chromium.org
BUG= 665113 

Change-Id: Ia399b03b912de0d1098cd8fc03845abc1c49275c
Reviewed-on: https://chromium-review.googlesource.com/411502
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Annie Sullivan <sullivan@chromium.org>

[modify] https://crrev.com/5dca59c419492fcdd739d3fe19f7fe955238a2ab/scripts/slave/recipe_modules/auto_bisect_staging/bisector.py
[modify] https://crrev.com/5dca59c419492fcdd739d3fe19f7fe955238a2ab/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/no_values.json
[modify] https://crrev.com/5dca59c419492fcdd739d3fe19f7fe955238a2ab/scripts/slave/recipe_modules/auto_bisect_staging/local_bisect.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/5dca59c419492fcdd739d3fe19f7fe955238a2ab

commit 5dca59c419492fcdd739d3fe19f7fe955238a2ab
Author: Roberto Carrillo <robertocn@google.com>
Date: Tue Nov 15 22:29:18 2016

No-culprit bisects due to bad builds explicit message and status.

When a bisect fails to find a culprit because all revisions between the
last known good and first known bad failed to build, provide a
descriptive message and provide details of the failed builds if their
number is manageable (<= 10)

R=sullivan@chromium.org,dtu@chromium.org
BUG= 665113 

Change-Id: Ia399b03b912de0d1098cd8fc03845abc1c49275c
Reviewed-on: https://chromium-review.googlesource.com/411502
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Annie Sullivan <sullivan@chromium.org>

[modify] https://crrev.com/5dca59c419492fcdd739d3fe19f7fe955238a2ab/scripts/slave/recipe_modules/auto_bisect_staging/bisector.py
[modify] https://crrev.com/5dca59c419492fcdd739d3fe19f7fe955238a2ab/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/no_values.json
[modify] https://crrev.com/5dca59c419492fcdd739d3fe19f7fe955238a2ab/scripts/slave/recipe_modules/auto_bisect_staging/local_bisect.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/5dca59c419492fcdd739d3fe19f7fe955238a2ab

commit 5dca59c419492fcdd739d3fe19f7fe955238a2ab
Author: Roberto Carrillo <robertocn@google.com>
Date: Tue Nov 15 22:29:18 2016

No-culprit bisects due to bad builds explicit message and status.

When a bisect fails to find a culprit because all revisions between the
last known good and first known bad failed to build, provide a
descriptive message and provide details of the failed builds if their
number is manageable (<= 10)

R=sullivan@chromium.org,dtu@chromium.org
BUG= 665113 

Change-Id: Ia399b03b912de0d1098cd8fc03845abc1c49275c
Reviewed-on: https://chromium-review.googlesource.com/411502
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Annie Sullivan <sullivan@chromium.org>

[modify] https://crrev.com/5dca59c419492fcdd739d3fe19f7fe955238a2ab/scripts/slave/recipe_modules/auto_bisect_staging/bisector.py
[modify] https://crrev.com/5dca59c419492fcdd739d3fe19f7fe955238a2ab/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/no_values.json
[modify] https://crrev.com/5dca59c419492fcdd739d3fe19f7fe955238a2ab/scripts/slave/recipe_modules/auto_bisect_staging/local_bisect.py

Roberto, looks like all the bugs mentioned here had successful bisects. Is there any way you can think of to verify this?

I am not sure. I was going over this other issue:  665439  and it seems this goes through the same codepath, because unclassifyiable revisions are still considered failed (so that bisect skips over them correctly)

I launched https://chromeperf.appspot.com/buildbucket_job_status/8995884977197255328 in staging which should hit the codepath (Even if the messaging still needs work before pushing to production)

Issue 664811 has been merged into this issue.

I am seeing a problem on downstream bisect. Let me know if I should open up a separate bug, or if this hasn't rolled downstream yet.

https://bugs.chromium.org/p/chromium/issues/detail?id=660168#c33
https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/clankium-phone-perf-bisect/builds/76

Gathering reference values succeeded, but then EVERY RUN of the test failed and did not produce the expected metrics in between the good and bad revision. build is green, recipe_result is null, and bug comment is super confusing:


===== BISECT JOB RESULTS =====
Status: completed


===== TESTED REVISIONS =====
Revision                   Mean      Std Dev  N   Good?
android-chrome@8a3cecfd18  81392299  799555   18  good
android-chrome@82803cf5a7  N/A       N/A      0   unknown
android-chrome@cdd988e5f7  N/A       N/A      0   unknown
android-chrome@02bdd6f1f3  N/A       N/A      0   unknown
android-chrome@50a211f88c  N/A       N/A      0   unknown
android-chrome@2cf23089d5  N/A       N/A      0   unknown
android-chrome@0b36a2b2f0  N/A       N/A      0   unknown
android-chrome@3320385cb7  N/A       N/A      0   unknown
android-chrome@9ed97d3eca  N/A       N/A      0   unknown
android-chrome@ae75e54e48  N/A       N/A      0   unknown
android-chrome@80e696ae65  N/A       N/A      0   unknown
android-chrome@962711aa5e  N/A       N/A      0   unknown
android-chrome@2dfcb6d821  N/A       N/A      0   unknown
android-chrome@5a274e5edd  99881188  837839   18  bad

Seeing another problem in https://bugs.chromium.org/p/chromium/issues/detail?id=663705#c13:
https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/webview-phone-perf-bisect/builds/54

===== BISECT JOB RESULTS =====
Status: completed


===== TESTED REVISIONS =====
Revision                                   Mean      Std Dev   N   Good?
android-chrome@1858ef09a5                  73807189  37668106  18  good
android-chrome@1858ef09a5,chromium@429351  82001920  9107205   5   good
android-chrome@1858ef09a5,chromium@429353  63748779  660799    12  bad
android-chrome@1858ef09a5,chromium@429356  63803392  454047    18  bad
android-chrome@611b8fea8a                  63694165  644345    12  bad
android-chrome@3b9eecd152                  63825237  880884    12  bad

Looks like the build failed for 429352: https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/webview-phone-perf-bisect/builds/54/steps/Working%20on%20revision%20android-chrome%401858ef09a5%2Cchromium%40429352.Waiting%20for%20build.buildbucket.get%20%2813%29/logs/json.output

But the bug update is very unclear.

Is there any progress to report on this? Is it a common kind of failure, or are these failures relatively rare? 

+simonhatch who has been digging into bisect failures.

Should this be its own bug? Or should we just work on it in more specific bugs? It looks like a lot of problems in this area were addressed in  bug 666447 ?

I'm fine with closing this and opening specific bugs as they're encountered. There's already a few bugs detailing some specific improvements to output:

Bisect - Failed run output contains no mention of failed build
https://bugs.chromium.org/p/chromium/issues/detail?id=668540

Bisect - Output unclear when expanding catapult
https://bugs.chromium.org/p/chromium/issues/detail?id=670363

Bisect - Make output on rollbacks more obvious.
https://bugs.chromium.org/p/chromium/issues/detail?id=670062


Roberto, did you look into the bisects in #c12 and #c14? Do you know what's happening there?

I'm closing this one since we've done a lot of fixes since these came in, and there are more specific bugs.