Quoting hlundin@:
"We have a number of performance tests that try to catch regressions in complexity (measured as run time for a fixed test case) for some components. Some of these tests are quite noisy, notably the APM component tests.

This [1] is a typical example, where a regression happened (turned out the system failed to flag it) that could perhaps be explained by a new Clang version. However, the subsequent recovery (flagged) is quite unexplainable. This behavior is typical for these tests, where the graphs jump from one steady state to another without reason. We are not talking about uncorrelated noise from one run to the next, but some kind of shifts in the bot behavior."

Proposed solution:
We could introduce the use of reference builds, similar to how Chromium runs with a "stable" reference build in addition to the newly built Chrome. This takes place only in the Chromium Perf waterfall [2]. Example graph with both a metric + ref: [3].

The configuration of which reference Chrome executable to use is stored in [4] and it seems doable to refactor this to allow reuse and have us maintain a similar .json file with WebRTC reference builds of the webrtc_perf_tests binary (and possibly other binaries as well). 
The actual test configuration is located in this file, living in the Chromium repo: [5] (warning: huge JSON file)

[1]: https://chromeperf.appspot.com/report?sid=6c45d38f7179bab5d1e579c0fc441696ea311d90dddf4a3a4627a9e7f41e3c70&start_rev=17971&end_rev=18296
[2]: https://build.chromium.org/p/chromium.perf/waterfall
[3]: https://chromeperf.appspot.com/report?sid=88999edf2eee4f4cffa814d1f566df729574447fc8c1038b06691b7b446ce004&rev=475286
[4]: https://cs.chromium.org/chromium/src/third_party/catapult/common/py_utils/py_utils/chrome_binaries.json
[5]: https://cs.chromium.org/chromium/src/testing/buildbot/chromium.perf.json?rcl=f05d0f177c30e5ad1b0309b46b69a288336012a8&l=88892