If you look at comments 4 and 5, it looks like Chrome doesn't build between the revisions that regressed. Not sure how bisect could narrow this further?

Yeah, I just wonder how did the perf bots then come up with a result? Aren't they using the same build infra?

Also, #9 didn't find a regression while I am pretty sure that I manually selected sub graphs where one could easily see the regression.

So many things going wrong here :(

For comments 4 and 5, the perf bots know there is a regression because they did a build at r444017 and another at r444070 and more memory was consumed in the latter. But then when we ran a bisect to see which commit in that range caused the regression, those revisions build and reproduce the regression, but then nothing from r444018-r444069 builds, so we can't narrow further.

For comment 9, looks like you re-ran on Windows 8, which was a great idea because maybe it builds on windows when it fails on Android. But if you click through to the buildbucket job details: https://chromeperf.appspot.com/buildbucket_job_status/8989671091441468352 and through to the bisect job: https://build.chromium.org/p/tryserver.chromium.perf/builders/win_8_perf_bisect/builds/2337 and expand "Nested step(s) for: Gathering reference values" you'll see that the test failed for the good and bad revision, even though it did pass when it ran on the perf bots. I'm not sure what happened because the logs don't appear to be available anymore (+dnj is that right? I'm looking at the "Failure Output" log, it doesn't appear to be available in logdog?)

I kicked off another bisect on a different Windows builder to see if maybe the tests will pass and builds will build on it, and if not maybe this time we'll get a log.

Simon, looks like the bisect comment could be clearer for https://bugs.chromium.org/p/chromium/issues/detail?id=683030#c9 ?

As Annie mentioned, the perf test suites tend to take 4-6 hours to run, and we don't have enough hardware deployed to do that for every chromium commit (let alone depots we pull in!). So instead after running the perf tests, it grabs the latest build (which may contain X commits), and runs the perf tests for that range of commits. If there's a regression detected, we know it was in that range of commits and the bisect bots try to narrow it further. In this case, the build was broken so it's not actually possible to proceed.

As for the broken windows bisect:

Blocking on:
  crbug.com/683999  - Bisect - Tee'ing output seems to break Windows
  crbug.com/683998  - Bisect - Output incorrect when test fails to run in gathering reference values.

"Failure reason" seems to be working here: https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Ftryserver.chromium.perf%2Fwin_8_perf_bisect%2F2337%2F%2B%2Frecipes%2Fsteps%2FFailure_reason%2F0%2Flogs%2Freason%2F0