All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=846271

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=8dc65a204ecd8c87e70a7f22a1d44eb2c1313317dd77aaa576896f74a5d0f31f


Bot(s) for this bug's original alert(s):

webrtc-android-tests-nexus4-lollipop
webrtc-android-tests-nexus5-kitkat

I think these android perf alerts are due to measurement noise. I see no obvious suspects among nearby cls.

Do you agree? 

Can we do something to either reduce noise or increase thresholds?

I am okay with increasing thresholds for these tests but I don't own them. ssilkin, you seem to have worked on these tests recently, what do you think?

I think these devices are too weak for tests which try to encode/decode HD+VGA+CIF in realtime with software codecs.
We should consider these tests as stress tests in such case and ignore fluctuations in performance metrics.

Are the alarm thresholds defined per platform? If not, then I suggest to ignore this regression and close the ticket.

Patrik, is there any easy way to increase thresholds for certain {test, device} combinations which are known to be particularly noisy?

Otherwise, I guess we'll just have to ignore alerts like these.

Yes: https://sites.google.com/a/google.com/rtc-platform/test-engineering/performance-tests#TOC-Silencing-Alerts-in-Noisy-Graphs-Admin-

Please give me a complete list of the tests you would like to silence for these device models, .e.g.

WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests / cpu_usage / simulcast_vp8_3sl_low
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests / cpu_usage / simulcast_vp8_3sl_high
...

for devices

webrtc-android-tests-nexus4-lollipop
webrtc-android-tests-nexus5-kitkat?

Should I silence the ones in this bug or others too?

The ones in this list would be a good start, but maybe Sergey can make up an extended list?

I suggest to ignore the following configs:

WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/cpu_usage/simulcast_vp8_3sl_low
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/cpu_usage/simulcast_vp8_3sl_medium
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/cpu_usage/simulcast_vp8_3sl_high
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/encode_time/simulcast_vp8_3sl_low
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/encode_time/simulcast_vp8_3sl_medium
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/encode_time/simulcast_vp8_3sl_high
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/encode_frame_rate/simulcast_vp8_3sl_low
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/encode_frame_rate/simulcast_vp8_3sl_medium
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/encode_frame_rate/simulcast_vp8_3sl_high
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/time_between_rendered_frames//simulcast_vp8_3sl_low
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/time_between_rendered_frames//simulcast_vp8_3sl_medium
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/time_between_rendered_frames//simulcast_vp8_3sl_high
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/time_between_rendered_frames//simulcast_vp8_3sl_low
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/time_between_rendered_frames//simulcast_vp8_3sl_medium
WebRTCPerf/webrtc-android-tests-nexus4-lollipop/webrtc_perf_tests/time_between_rendered_frames//simulcast_vp8_3sl_high

WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/cpu_usage/simulcast_vp8_3sl_low
WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/cpu_usage/simulcast_vp8_3sl_medium
WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/cpu_usage/simulcast_vp8_3sl_high
WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/encode_time/simulcast_vp8_3sl_low
WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/encode_time/simulcast_vp8_3sl_medium
WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/encode_time/simulcast_vp8_3sl_high
WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/encode_frame_rate/simulcast_vp8_3sl_low
WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/encode_frame_rate/simulcast_vp8_3sl_medium
WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/encode_frame_rate/simulcast_vp8_3sl_high
WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/time_between_rendered_frames//simulcast_vp8_3sl_low
WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/time_between_rendered_frames//simulcast_vp8_3sl_medium
WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/time_between_rendered_frames//simulcast_vp8_3sl_high
WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/time_between_rendered_frames//simulcast_vp8_3sl_low
WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/time_between_rendered_frames//simulcast_vp8_3sl_medium
WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/time_between_rendered_frames//simulcast_vp8_3sl_high

Ok, I've added everything in #9 to the "Disable Alerting" anomaly config in the dashboard; there should be no new alerts generated for those now.