📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/1074d3a2840000

Actually bisecting benchmark_duration since there's a clear drop:
https://chromeperf.appspot.com/report?sid=7da0597fd3d59a86fa389c3f4037f1c8ee97eccd210a8e60e803e06fa1190ec8

📍 Couldn't reproduce a difference.
https://pinpoint-dot-chromeperf.appspot.com/job/1074d3a2840000

+dtu, simonhatch: do you know why benchmark_duration is 0 here?

It looks like benchmark_duration is only available in chartjson, not histograms.

It is, and that's a bug. I noticed this while doing some test path parity investigation earlier with simonhatch@ (we're getting ready to launch the histogram pipeline).

Filed https://github.com/catapult-project/catapult/issues/4207 on myself to sort this out.

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/169236da840000

Thanks, kicking off a functional bisect.

📍 Couldn't reproduce a difference.
https://pinpoint-dot-chromeperf.appspot.com/job/169236da840000

This test fails on the waterfall but passes on pinpoint.

Charlie, I see some BattOr errors in the log, could they be related?

https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FMac_Air_10.11_Perf%2F1832%2F%2B%2Frecipes%2Fsteps%2Fmedia.desktop_on_Intel_GPU_on_Mac_on_Mac-10.11%2F0%2Fstdout

(ERROR) 2018-01-28 23:20:21,521 battor_wrapper._FlashBattOr:174  Git hash returned from BattOr was not as expected: [0128/232021.520446:FATAL:battor_agent_bin.cc(100)] Fatal error when communicating with the BattOr: TOO MANY COMMAND RETRIES
Traceback (most recent call last):
  File "/b/s/w/ir/third_party/catapult/common/battor/battor/battor_wrapper.py", line 161, in _FlashBattOr
    device_git_hash = self.GetFirmwareGitHash()
  File "/b/s/w/ir/third_party/catapult/common/battor/battor/battor_wrapper.py", line 397, in GetFirmwareGitHash
    int(self._git_hash, 16)
ValueError: invalid literal for int() with base 16: '[0128/232021.520446:FATAL:battor_agent_bin.cc(100)] Fatal error when communicating with the BattOr: TOO MANY COMMAND RETRIES'
(CRITICAL) 2018-01-28 23:20:21,523 battor_wrapper.KillBattOrShell:190  BattOr shell was not properly closed. Killing now.

/facepalm

Yea, this seems to be related to BattOr failures. 

As a short term, assigning to Infra > Labs. Would you folks mind resetting the BattOr attached to build127-b1?

There are two problems here: 

1) BattOr errors shouldn't cause the a benchmark to fail entirely (https://github.com/catapult-project/catapult/issues/3798). I was making an effort to fix this (which is mostly complete), but that effort got interrupted my the maintenance mutex work and I never returned to it. Given that I'm fairly confident I could finish it up in the afternoon, I'll try and knock it out today.

2) The fact that the BattOr is failing at all is problematic. Aaron has some work in a branch that greatly increases the reliability of the BattOr, but it's still going through code review and progress has been slow recently for work-unrelated reasons.

/bump

Could someone on Infra>Labs take a look at this? 

(unassigning myself for now to redirect to Infra>Labs, and I'll probably dupe this into https://github.com/catapult-project/catapult/issues/3798 when they're done.

Setting OS to help get into infra labs triage queue.

Orange led on the battor was cycling on 3 fast blinks.

Reset it and now looks normal (led wise)