Peter,

Do we have any spare N9s?  It's possible that this 3 year old device is dead.

There are 3 N9's attached to this host. Looks like the ac battery charging support on the hub went away as two off the N9's were close to having depleted batteries while the third was powered off.

I've swapped the hub and will reattach the N9's once they have been fast charged.

All devices charged and attached.  Effective starting with https://build.chromium.org/p/client.webrtc.perf/builders/Android64%20Tests%20%28L%20Nexus9%29/builds/4245

Checking HT4ADJT03192...
  AC powered: true
  level: 100
Checking HT4AEJT01652...
  AC powered: true
  level: 100
Checking HT593JT00040...
  AC powered: true
  level: 100

Hi, it seems like this is happening again:

I    0.003s TimeoutThread-1-for-MainThread  [host]> /b/c/b/Android64_Tests__L_Nexus9_/src/third_party/android_tools/sdk/platform-tools/adb devices
E    0.011s Main  No available devices to provision.
E    0.011s Main  Local device blacklist: {u'HT593JT00040': {u'timestamp': 1507634572.155261, u'reason': u'reboot_timeout'}, u'HT4ADJT03192': {u'timestamp': 1507634572.153724, u'reason': u'reboot_timeout'}, u'HT4AEJT01652': {u'timestamp': 1507634572.156283, u'reason': u'reboot_timeout'}}
E    0.021s Main  Unable to provision local devices.
Traceback (most recent call last):
  File "/b/c/b/Android64_Tests__L_Nexus9_/src/third_party/catapult/devil/devil/android/tools/provision_devices.py", line 640, in main
    wipe=not args.skip_wipe and not args.emulators)
  File "/b/c/b/Android64_Tests__L_Nexus9_/src/third_party/catapult/devil/devil/android/tools/provision_devices.py", line 95, in ProvisionDevices
    devices = script_common.GetDevices(devices, blacklist)
  File "/b/c/b/Android64_Tests__L_Nexus9_/src/third_party/catapult/devil/devil/android/tools/script_common.py", line 47, in GetDevices
    raise device_errors.NoDevicesError()
NoDevicesError: No devices attached.
step returned non-zero exit code: 87

Full logs at https://build.chromium.org/p/client.webrtc.perf/builders/Android64%20Tests%20%28L%20Nexus9%29/builds/4509/steps/provision_devices/logs/stdio.

Thanks for your support.

Looks like these have stabilized by themselves since https://build.chromium.org/p/client.webrtc.perf/builders/Android64%20Tests%20%28L%20Nexus9%29/builds/4517

Well, not really. :)

https://build.chromium.org/p/client.webrtc.perf/builders/Android64%20Tests%20%28L%20Nexus9%29

Yeah.  The devices would not become adb aware until I power cycled them.

For got measure I re-flashed them.  

Effective starting with https://build.chromium.org/p/client.webrtc.perf/builders/Android64%20Tests%20%28L%20Nexus9%29/builds/4565

Looks like the devices have been pretty stable.

The exact same problem again.

E    0.006s Main  Local device blacklist: {u'HT593JT00040': {u'timestamp': 1517994798.340946, u'reason': u'reboot_timeout'}, u'HT4ADJT03192': {u'timestamp': 1517994798.345469, u'reason': u'reboot_timeout'}, u'HT4AEJT01652': {u'timestamp': 1517994798.339919, u'reason': u'reboot_timeout'}}

https://build.chromium.org/p/client.webrtc.perf/builders/Android64%20Tests%20%28L%20Nexus9%29

One device is back online, two are still "USB status: offline"

Had to powercycle the 2 "missing" N9's before they became adb aware.

Effective starting with https://build.chromium.org/p/client.webrtc.perf/builders/Android64%20Tests%20%28L%20Nexus9%29/builds/5698