Looks like the device went offline after device_status ran. Yay :(

And it's flakily doing this.

Not sure if trooper can do anything about this; 

Relevant output and very ungraceful failure. This looks like a problem with swarming, or a problem of one bot not responding to adb commands from the host. CC'ed John. Let's add stip, too.

Traceback (most recent call last):
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/android/tools/device_recovery.py", line 65, in RecoverDevice
    device.WaitUntilFullyBooted(retries=0)
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 159, in Run
    error_log_func=error_log_func)
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 151, in _JoinAll
    (len(alive_threads), len(self._threads)))
CommandTimeoutError: Timed out waiting for 1 of 1 threads.
I 4473.049s TimeoutThread-1-for-device_shard_helper(2)  [host]> /b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s ZX1G22KB64 reboot
E 4473.114s device_shard_helper(2)  Failed to reboot ZX1G22KB64.
Traceback (most recent call last):
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/android/tools/device_recovery.py", line 72, in RecoverDevice
    device.Reboot(block=False, timeout=5, retries=0)
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 159, in Run
    error_log_func=error_log_func)
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 158, in _JoinAll
    thread.ReraiseIfException()
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 152, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/android/device_utils.py", line 626, in Reboot
    self.adb.Reboot()
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 854, in Reboot
    self._RunDeviceAdbCmd(cmd, timeout, retries)
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 282, in _RunDeviceAdbCmd
    check_error=check_error)
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/rr/tmpsZ4z0q/w/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 258, in _RunAdbCmd
    args, output, status, device_serial)
AdbCommandFailedError: (device: ZX1G22KB64) adb reboot: failed with exit status 1 and output:
- error: device 'ZX1G22KB64' not found

This bot isn't on swarming.

this bot is also still using adb 1.0.32, fwiw.

Renaming this bug to follow the perf bot sheriff guideline.

This bot is still behaving badly. It has been alternating between a lovely shade of purple and an attractive red as far back as I can see (December 8th). It is currently continuing a four day run of solid purple.

Well I fixed up the offline/missing device side of things.

List of devices attached
ZX1G22KB64	device
ZX1G22KB72	device
ZX1G22K6GS	device
ZX1G22K86J	device
ZX1G22KBZ8	device
ZX1G22KFD7	device
ZX1G22KG9L	device

This will be effective starting with https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus6%20Perf%20%282%29/builds/4684

This looks good.

Closing.