All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=737193

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgjqihpQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgtv_V7AkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg9o74owoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgtsOknwgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg9p2OnAkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgjorHugoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgjsr-vgoM


Bot(s) for this bug's original alert(s):

webrtc-android-tests-nexus5-kitkat

You're right that the number of devices most likely affect the results with variations in the scores, but running at one device also comes with a price since when the device is offline: these are consumer devices, they always break sooner or later, especially if they run tests non-stop. One device will also take a lot longer to run all the perf tests, so blame lists will be longer for regressions, which is tough since we don't have auto-bisection.

It's a tough call to make: have one device to push hard, with longer blamelists on regressions, or several identical devices for better reliability, faster executed tests and more time to cool down in between. I think the latter is better, but I'm open to discuss other strategies.

Either way I don't have any actionable item in this bug, but feel free ot start a thread at g/webrtc-perf-sheriffs to discuss this further.