Actually looking closer at the benchmark log, I find this is a device issue. Device was unreachable after sometime, which makes the rest of the benchmark run completely fails. 

DeviceUnreachableError: ZX1G22KGFM
(https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus6_WebView_Perf%2F2049%2F%2B%2Floading.mobile_625ea9fb-a9c4-43f2-8b73-465a22dce528)


Lab: can you inspect device build203-b7--device2? Possibly just take it off completely.

Is it possible that something in the shard is killing the device mid-run?

#3: John: why would the shard kill the device mid-run? Is that because of timedout?

"something in the shard" -> e.g. the browser causes a device crash, etc

I think that is very unlikely, given how this is the only bot that have such problem among other 14 shards.

For the first step, I would like labs to help with turning off this device, so soft device affinity will kick in and replace it with another healthy device.

If the problem persist, we can know that this is software issue.

Would it not be similarly unlikely that the device would suddenly and consistently develop issues at the same time that we switched to OBBS? https://chrome-swarming.appspot.com/bot?id=build203-b7--device2

OBBS does transfer the load on the device to one test at a time to triggering a job once that runs all the tests.  This means that the duration of the task is much longer (ie 4-5 hours vs 2-3 minutes).  This could be a large load difference for this bot and it could be having trouble with such a long running task.

I think we should remove this device to see if another device is better able to handle the load.

Ping

This bot just died again in https://chrome-swarming.appspot.com/task?id=3e9f6b7fa715d510&refresh=10&show_raw=1

John, can you take a look at build203-b7--device2?  thanks

Replaced build203-b7--device2 with a spare I happened to find (last of it's kind, in all probability).

Hopefully this can answer the question of whether it was a s/w problem or not.

Fixing the device make all the tests passing now, amazing!

https://ci.chromium.org/buildbot/chromium.perf/Android%20Nexus6%20WebView%20Perf/2060

Thanks John!

The NextAction date has arrived: 2018-07-12