All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=727476

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg2tWLnQsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg-ua_rAoM


Bot(s) for this bug's original alert(s):

android-webview-nexus5X

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8978200197409668048

=== BISECT JOB RESULTS ===
NO Perf regression found, tests failed to produce values

Bisect Details
  Configuration: android_webview_arm64_aosp_perf_bisect
  Benchmark    : system_health.common_mobile
  Metric       : timeToFirstMeaningfulPaint_avg/load_media/load_media_imgur


To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.media.imgur system_health.common_mobile

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8978200197409668048

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6734311434223616


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8978178317215375232

=== BISECT JOB RESULTS ===
NO Perf regression found, tests failed to produce values

Bisect Details
  Configuration: android_webview_arm64_aosp_perf_bisect
  Benchmark    : system_health.common_mobile
  Metric       : timeToFirstMeaningfulPaint_avg/load_media/load_media_imgur


To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.media.imgur system_health.common_mobile

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8978178317215375232

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6734311434223616


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8977937942020780544

Bisects were failing due to  crbug.com/728594 , restarting this.

=== BISECT JOB RESULTS ===
NO Perf regression found, tests failed to produce values

Bisect Details
  Configuration: android_webview_arm64_aosp_perf_bisect
  Benchmark    : system_health.common_mobile
  Metric       : timeToFirstMeaningfulPaint_avg/load_media/load_media_imgur


To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.media.imgur system_health.common_mobile

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8977937942020780544

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5262999318691840


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8977778193199652144

=== BISECT JOB RESULTS ===
NO Perf regression found, tests failed to produce values

Bisect Details
  Configuration: android_webview_arm64_aosp_perf_bisect
  Benchmark    : system_health.common_mobile
  Metric       : timeToFirstMeaningfulPaint_avg/load_media/load_media_imgur


To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.media.imgur system_health.common_mobile

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8977778193199652144

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5262999318691840


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

"No Perf regression found" is a little confusing, but this seems to just fail.

> 2017-06-04 03:09:00 Failed to run bisect 2 times.Stopping automatic restart for this job.
> 2017-06-01 03:09:14 Failed to run bisect 2 times.Stopping automatic restart for this job.
TRIAGE

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8971004375732069472

I kicked off one more bisect, but it looks like BattOR was failing in this range:

https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Ftryserver.chromium.perf%2Fandroid_webview_arm64_aosp_perf_bisect%2F1306%2F%2B%2Frecipes%2Fsteps%2FGathering_reference_values%2F0%2Fsteps%2FPerformance_Test_1_of_6%2F0%2Flogs%2FFailure_Output%2F0
ERROR) 2017-06-03 19:08:27,087 battor_wrapper._FlashBattOr:166  Git hash returned from BattOr was not as expected: [0603/190827.087135:ERROR:serial_io_handler.cc(152)] Failed to open serial port: FILE_ERROR_ACCESS_DENIED
Traceback (most recent call last):
  File "/b/c/b/android_webview_arm64_aosp_perf_bisect/src/third_party/catapult/common/battor/battor/battor_wrapper.py", line 153, in _FlashBattOr
    device_git_hash = self.GetFirmwareGitHash()
  File "/b/c/b/android_webview_arm64_aosp_perf_bisect/src/third_party/catapult/common/battor/battor/battor_wrapper.py", line 381, in GetFirmwareGitHash
    int(self._git_hash, 16)
ValueError: invalid literal for int() with base 16: '[0603/190827.087135:ERROR:serial_io_handler.cc(152)] Failed to open serial port: FILE_ERROR_ACCESS_DENIED'


charliea: Is there any way to get a bisect like this to succeed? Or was the test just broken in the range?

Unfortunately, I don't know of any way to make this bisect succeed without changing Telemetry. I filed a Catapult bug discussing the solution that Ned and I think will work here: https://github.com/catapult-project/catapult/issues/3798.

There's also a bug tracking this specific error (FILE_ERROR_ACCESS_DENIED):  bug 759004 

And a bug tracking the restart of this specific BattOr, which should hopefully allow us to run this bisect to completion:  bug 759008 

Once labs restarts that BattOr, I'll kick off this bisect again.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8969906694136404864

Restarted the bisect. Based on  bug 759004 , it's very likely that this isn't actually a BattOr bug but rather an infrastructure bug in which docker isn't properly allocating giving us the serial port access that we need in order to reach the BattOr. bpastene@ has a fix that he just sent out that thinks might affect this, so we're going to wait and see if that has an impact.

However, I'm also going to add a third tracking bug pertaining to another problem: not having the BattOr data shouldn't make Telemetry automatically fail. Rather, we should rely on the bisect to fail because the metric we're bisecting on isn't present. Here's a tracking bug for that: https://github.com/catapult-project/catapult/issues/3798

=== BISECT JOB RESULTS ===
NO Perf regression found, tests failed to produce values

Bisect Details
  Configuration: android_webview_arm64_aosp_perf_bisect
  Benchmark    : system_health.common_mobile
  Metric       : timeToFirstMeaningfulPaint_avg/load_media/load_media_imgur


To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.media.imgur system_health.common_mobile

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8969906694136404864


For feedback, file a bug with component Speed>Bisection

Closing this as WontFix. I think this is a manifestation of https://github.com/catapult-project/catapult/issues/1574