All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=699072

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDglOevoQkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDglJPNpgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg5JfNrgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDglLzBoAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDglOnstQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDglNe0owsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDglPHbvwsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDglIGRvgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDglILPoQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg1LC4vQoM


Bot(s) for this bug's original alert(s):

webrtc-android-tests-nexus5-kitkat
webrtc-android-tests-nexus9

Looks like changing the order in which tests run cause values from different tests to be used to track the same metric.

For example: WebRTCPerf/webrtc-android-tests-nexus5-kitkat/webrtc_perf_tests/ramp_up_down_up_1stream_rtx/first_rampup

Build 1271: https://build.chromium.org/p/client.webrtc.perf/builders/Android32%20Tests%20%28K%20Nexus5%29/builds/1271
The tests which produce an output for the metric "ramp_up_down_up_1stream_rtx" is reported in this order:
[ RUN      ] RampUpTest.UpDownUpAudioTransportSequenceNumberRtx
RESULT ramp_up_down_up_1stream_rtx: first_rampup= 13 ms
[ RUN      ] RampUpTest.UpDownUpTransportSequenceNumberPacketLoss
RESULT ramp_up_down_up_1stream_rtx: first_rampup= 94 ms

Build 1272: https://build.chromium.org/p/client.webrtc.perf/builders/Android32%20Tests%20%28K%20Nexus5%29/builds/1272
The tests run in this order instead:
[ RUN      ] RampUpTest.UpDownUpTransportSequenceNumberPacketLoss
RESULT ramp_up_down_up_1stream_rtx: first_rampup= 106 ms
[ RUN      ] RampUpTest.UpDownUpAudioTransportSequenceNumberRtx
RESULT ramp_up_down_up_1stream_rtx: first_rampup= 13 ms

Is this working as intended?

I'm not sure I'm following what's right/wrong here. The perf dashboard doesn't know anything about the test in which the perf metric is reported (and it shouldn't), since it's perfectly fine if a perf test doesn't even use gtest. It only categorizes the data points using the test executable name and the identifiers printed after the RESULT line.

So it seems like the problem here is to make sure each entry printed has a unique name, to avoid mixing them together

Removing R-V-G label (was set upon bug filing because the bot was incorrectly configured as internal)

Stefan, looks like the test UpDownUpTransportSequenceNumberPacketLoss (added in https://codereview.webrtc.org/2705603002) report the same metric as UpDownUpAudioTransportSequenceNumberRtx.

Now the order of those two tests have changed so now certain metrics use the result from one test instead of the other.

The following revision refers to this bug:
  https://chromium.googlesource.com/external/webrtc.git/+/45b5fe549f9bea7a4abc9932937b6c35ad5adc54

commit 45b5fe549f9bea7a4abc9932937b6c35ad5adc54
Author: stefan <stefan@webrtc.org>
Date: Thu Mar 09 14:27:02 2017

Don't report perf metrics for packet loss ramp-up tests.

BUG= chromium:699072 

Review-Url: https://codereview.webrtc.org/2744603002
Cr-Commit-Position: refs/heads/master@{#17145}

[modify] https://crrev.com/45b5fe549f9bea7a4abc9932937b6c35ad5adc54/webrtc/call/rampup_tests.cc
[modify] https://crrev.com/45b5fe549f9bea7a4abc9932937b6c35ad5adc54/webrtc/call/rampup_tests.h

Metrics back down -- added to the report at the top of this bug.