The bot has a huge number of failures still.

This seems to be an infra issue rather than a bad test. The test passes twice and then the failure happens before the test is even run.

From the second link in the original post:

[ RUN      ] text_constant_full_page_raster_90000_pixels_per_second
...
(INFO) 2017-06-26 15:11:36,979 timeout_retry.WaitFor:107  condition 'battery_updates_enabled' met (4.9s)
...
(INFO) 2017-06-26 15:11:37,095 timeout_retry.WaitFor:107  condition 'cool_device' met
...
(INFO) 2017-06-26 15:11:38,220 timeout_retry.WaitFor:107  condition 'device_online_with_root' met (0.2s)
...
(CRITICAL) 2017-06-26 15:11:38,543 timeout_retry.Run:174  (TimeoutThread-1-for-MainThread) Exception on RunShellCommand(ZX1G225JCX, ['ndc', 'resolver', 'flushdefaultif'], as_root=True, check_return=True, timeout=30, retries=3), attempt 1 of 4: AdbShellCommandFailedError('(device: ZX1G225JCX) shell command run via adb failed on the device:\n  command: ndc resolver flushdefaultif\n  exit status: 4\n  output:\n  - Error connecting (Connection refused)\n',)

+vmiura@

"'cool_device' met" ... is that related to thermal throttling we are seeing in other places? (e.g.  issue 811584 )

> "'cool_device' met" ... is that related to thermal throttling we are seeing in other places?

Yes, I think this is related to waiting for the device to be cool between runs.

+Ned, can you help triage?

The log is:
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/core/platform.py", line 193, in FlushDnsCache
    return self._platform_backend.FlushDnsCache()
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/android_platform_backend.py", line 330, in FlushDnsCache
    ['ndc', 'resolver', 'flushdefaultif'], as_root=True, check_return=True)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 159, in Run
    error_log_func=error_log_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 158, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 152, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1025, in RunShellCommand
    output = handle_large_output(cmd, large_output)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 994, in handle_large_output
    return handle_large_command(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 976, in handle_large_command
    return handle_check_return(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 967, in handle_check_return
    return run(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 963, in run
    return self.adb.Shell(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 505, in Shell
    command, output, status=status, device_serial=self._device_serial)
AdbShellCommandFailedError: (device: ZX1G225JCX) shell command run via adb failed on the device:
  command: ndc resolver flushdefaultif
  exit status: 4
  output:
  - Error connecting (Connection refused)

(https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus6_Perf%2F197%2F%2B%2Frecipes%2Fsteps%2Fthread_times.tough_scrolling_cases_on_Android%2F0%2Fstdout)

So basically the adb command to run "ndc resolver flushdefaultif" is flaky.

I suspect this is just another case of adb flakiness. 

John/Juan: thoughts?

Downgrade priority since this seems to no longer happen.

"cool_device" is indeed us waiting for the device to cool down.

I don't think this is specifically adb flakiness, at least flakiness outside of ndc. (If ndc is turning around and doing something back over adb, then maybe.)

It looks like at least thread_times.tough_scrolling_cases is not failing as often on that bot right now. At least some of the recent failures seem to be caused by some error in a script? Filed  issue 827025  for those.

Assigning this to Ned to decide what the next step should be, since this seems related to infra.

This doesn't happen again. Will create a tracking bug if I see this often enough.