All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=669934

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgh9WE_wgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgx-GysQoM


Bot(s) for this bug's original alert(s):

webrtc-mac-large-tests

Patrik, could it be your CL (https://codereview.webrtc.org/2529393006) that is causing these changes to the level controller perf stats? Please have a look

I guess it was. I really don't understand why though; I'm calling the code that generated the metric  level_controller_call_durations_48000_48000_44100_44100Hz_2_channels_render in exactly the same way I used to. The only difference is there will be a lot less calls to the production code since my CL drastically reduces the amount of tested variants.

peah@, that's also a theory why the tests are unstable: maybe different runs interfere with each other? Is there any global state in any of the production code you're testing here?

There are also alerts from the echo detector perf tests firing on this CL. Could this be due to an infra change?

There are no infra changes I know of that happened around yesterday. Edward, can you confirm? That would be both linux-large-tests and mac-large-tests; sounds unlikely.

Also this regression was flagged 15326 - 15330, and I think you should probably look at 15324 - 15332 or so to be sure. Could it be one of those CLs? My CL should really not affect echo detector perf tests unless 1) the tests interact via global state or 2) the tests affect global state on the machine, such as the sound card.

See #6

Not that I know of. Henrik?

No infra changes AFAIK.

Regarding #3:
There is no global state in the production code that is tested so the tests should not be affected by that.

What I do in the tests is that I benchmark the call duration for 1 api call.
My guess of what is happening is that the complexity of the call is too low to allow that to be done reliably.

Furthermore, this could have gotten worse when the DCHECKS were disabled in the performance tests as that even further reduced the call duration.

Does that make sense?

So why is that problematic? Is it because the benchmark is too short to be accurately measured? If that is the problem, it should be easy to fix by increasing the number of times the function is called, right?

That is not problematic and I'll create a CL for that. But since it worked before I saw no reason for changing that and it is a bit strange that it worked before, but not now. The DCHECKS could be a reason for it though.