Device 00ce6aad89eb30f7 blacklisted

Looking at the history on this slave it looks like devices drop off randomly.  I suspect that a usb hub power cycle is in order.  Will do this after the current job completes.

usb hub power cycled.  Effective starting with https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Nexus5X%20WebView%20Perf%20%283%29/builds/1718

Well power cycling the hub did not change the random device offline behavior.  I will switch it out.

New hub in place starting with https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Nexus5X%20WebView%20Perf%20%283%29/builds/1720

Adding a couple of clank infra folks in case they have some ideas or suggestions.

New hub made no difference.

I checked Android Nexus5X WebView Perf (1)  & Android Nexus5X WebView Perf (2)  and the offline randomness appears there as well.

As a test on build77-b1 I rebooted the slave 5 times without the slave connecting to the master and in all cases adb sees all the devices and can interact with them.  Something in the build is making various devices go offline?

I found the following after the last build:

chrome-+ 13988  2.0  0.0 150376 11600 ?        Sl   15:23   0:31 adb -P 5037 fork-server server
chrome-+ 26763  0.0  0.0  67424  1892 ?        Ssl  15:27   0:00 /b/rr/tmpD5c8uS/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/forwarder_host --adb=/b/rr/tmpD5c8uS/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb --serial-id=00eca87b89a352ea --map 0 33594

lsof -p 13988 shows:

adb     13988 chrome-bot  txt    REG                8,2  1221540  7872906 /b/rr/tmpD5c8uS/w/src/third_party/android_tools/sdk/platform-tools/adb (deleted)

There are competing adb processes on the slave?

#8: that's one adb process and one forwarder that appears to be attempting to use a different adb binary (which still isn't right).