All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=637568

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgtrvZ4goM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgtrvZ4gkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg9ob6vAkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgtu2cswsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg9qGIuwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgtpuKsgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg9trAsQsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgtuvmuAkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg9uDvpQkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg9on3owsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg9v6FugkM


Bot(s) for this bug's original alert(s):

chromium-webrtc-rel-7

Comment #2 deleted since it related to the wrong issue.

I can see that Edward's https://codereview.chromium.org/2233943003 is in the blame list . It was later reverted (https://codereview.chromium.org/2237313003/) since a new test started breaking on Win7, so if that was the culprit I believe the graphs should have started to recover by then (or maybe these perf numbers were produced only when that test was enabled, so that now there's no new data being added).

Edward can you investigate a few of the metrics to add more information?

Yes, those metrics are printed in the webrtc_audio_quality_browsertest that was briefly re-enabled.

https://cs.chromium.org/chromium/src/chrome/browser/media/webrtc_audio_quality_browsertest.cc?rcl=0&l=583
https://cs.chromium.org/chromium/src/chrome/browser/media/webrtc_audio_quality_browsertest.cc?rcl=0&l=603

Also, the last measurement before the regression was on June 28th, so I'd agree with Henrik, that those perf numbers were produced only when the test was re-enabled and no new data has been produced since.

Great! So that what you are basically is that it was caused by a CL that was reverted but that the effect of the revert is not yet visible in the graphs?!

Then I guess it would make sense for me to keep this open until I've seen the effect in the grahps, right?

Right now we can only see the Win7 regression (ChromiumWebRTC, chromium-webrtc-rel-7) for this bug, and that machine was involved in a reimage + swap in bug 611083.

We checked builds and the time the test was disabled matches well with the perf data.
I suggest closing as WontFix even though the perf data is slightly more noisy now.

or do you think it's worth having an audio engineer with more hands-on experience of this test look at why it went from +3dB to -5dB?

The measure is still quite strange though. Looking at the code for the metric, a negative value indicates that the output has a lower level than the input, which is the opposite what I would expect of such a test. 
Furthermore, before this issue it has been positive for over a year. 
Or am I missing something?

Ah, missed reading comment #10 before posting #11.

Yes, these values look strange to me. Is there any guide for running the test locally?

This possibly looks related to 
https://bugs.chromium.org/p/webrtc/issues/detail?id=6206

The regression was affecting only the Win7 bot. 

Do you have a Windows machine handy?
Or we could help so you can run the tests on the bot.

I have no windows machine, so it would be great if I could run the test on the bot and collect the .wav files that are generated there.

Is there a guide for how to do that?

The instructions are at:
https://chrome-internal.googlesource.com/infra/infra_internal/+/master/doc/ssh.md

You should request access to chrome-build-access:
https://ganpati.corp.google.com/#Create_Membership?name=chrome-build-access&member=

And once you have it, you can let us now :)

I have the access now. But I'm not fully sure of how to apply it. How do I know what windows machine to log into to verify the regression?

Note that the graphs have no values between June 28 and August 11. Anything could have happened in between.

The reason for the gap is that the tests were disabled. 
Then the machines were changed, a new sound card was installed and when we re-enabled the tests, this is what happened.

There recently was a further regression in pesq that cannot easily be explained by the surrounding CLs, so I added it to this issue. It looks like the regression happened in 3 steps where the first one was the biggest. 

You're correct ivoc, those values are likely explained by hardware changes/configuration of the sound card. I spun off  bug 642819  for that to avoid confusing this bug.

 Issue 642819  has been merged into this issue.

The problems we're seeing with this bug are related to that the machine was replaced and got a new sound card installed (bug 625808 covers all of that). Then tests were re-enabled, reverted and then re-enabled again:

Like ivoc@ commented, the regressions happened in three steps:

1. Large drop from 3.5 to 2.3 at 2016-08-11 (around #411297). This matches with Edward's first re-enabling of the tests in https://codereview.chromium.org/2233943003

2. Small drop to 2.1 at 2016-08-12 (around #411576). The above enabling was reverted in https://codereview.chromium.org/2237313003 at this day.

3. Small drop to 1.9 at 2016-08-29 (around #415034). Don't know what caused this, Edwards final reland in https://codereview.chromium.org/2241223002 happened at Aug 17th with no noticeable change.

Edward: do you think manual configuration of the output level on the machine could be related to the different values here? Maybe worth taking a look at the machine once and then monitor the next build?

The metrics are back up again; I've added the new alerts to this bug, mainly for posterity.

Yep, the audio quality tests are fixed.

Added some more alerts (caused by metrics returning to their old values) to this bug.