Didn't mention: it seems that the failing story is reliably video.html?src=crowd.ogg&type=audio

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8965364701067099232

The problem is with battOr: 

https://chromium-swarm.appspot.com/task?id=3947927206c4f210&refresh=10&show_raw=1

Traceback (most recent call last):
  RunBenchmark at c:\b\s\w\ir\third_party\catapult\telemetry\telemetry\internal\story_runner.py:334
    max_num_values=benchmark.MAX_NUM_VALUES)
  Run at c:\b\s\w\ir\third_party\catapult\telemetry\telemetry\internal\story_runner.py:202
    _RunStoryAndProcessErrorIfNeeded(story, results, state, test)
  _RunStoryAndProcessErrorIfNeeded at c:\b\s\w\ir\third_party\catapult\telemetry\telemetry\internal\story_runner.py:96
    test.WillRunStory(state.platform)
  WillRunStory at c:\b\s\w\ir\third_party\catapult\telemetry\telemetry\web_perf\timeline_based_measurement.py:272
    platform.tracing_controller.StartTracing(self._tbm_options.config)
  StartTracing at c:\b\s\w\ir\third_party\catapult\telemetry\telemetry\core\tracing_controller.py:43
    self._tracing_controller_backend.StartTracing(tracing_config, timeout)
  StartTracing at c:\b\s\w\ir\third_party\catapult\telemetry\telemetry\internal\platform\tracing_controller_backend.py:91
    started = agent.StartAgentTracing(config, timeout)
  StartAgentTracing at c:\b\s\w\ir\third_party\catapult\telemetry\telemetry\internal\platform\tracing_agent\battor_tracing_agent.py:73
    self._battor.StartTracing()
  StartTracing at c:\b\s\w\ir\third_party\catapult\common\battor\battor\battor_wrapper.py:235
    self._SendBattOrCommand(self._START_TRACING_CMD)
  _SendBattOrCommand at c:\b\s\w\ir\third_party\catapult\common\battor\battor\battor_wrapper.py:359
    'Outputted: %s' % (cmd, status))
BattOrError: BattOr did not complete command 'StartTracing' correctly.
Outputted: [1018/023722.617:FATAL:battor_agent_bin.cc(100)] Fatal error when communicating with the BattOr: TOO MANY COMMAND RETRIES

Not sure why disabling our benchmark when BattOr is broken is considered the right course of action. It would make much more sense to have failures in the battOr code not fail our whole benchmark (since the other data collected would still be useful). This would be similar to how it media.desktop works on devices that don't have BattOrs attached to them.

Sorry for the churn - I saw other stories failing on media.desktop earlier, then saw this story failing, so assumed that there were other stories in the benchmark that were passing and that this was the only failing story. Disabling this definitely wasn't the correct course of action in this case.

I'm working on making Telemetry work in the way that you described (the entire story doesn't fail if the BattOr fails, just some metrics aren't reported). You're definitely right - that's the way it should work.

Meanwhile, I'm going to kick this over to Infra labs. 

Infra>Labs, could you please restart the BattOr attached to build180-b4?

=== BISECT JOB RESULTS ===
NO Test failure found

Bisect Details
  Configuration: winx64_high_dpi_perf_bisect
  Benchmark    : media.desktop
  Metric       : cpu_time_percentage_avg/video.html?src_crowd.ogg_type_audio

Revision             Exit Code      N
chromium@508967      0 +- N/A       2      good
chromium@508991      0 +- N/A       2      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=video.html.src.crowd.ogg.type.audio media.desktop

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8965364701067099232


For feedback, file a bug with component Speed>Bisection

Battor restarted and indication light look normal again.

Thanks John!

The NextAction date has arrived: 2017-10-19