This is flaky. 4 tests:

4 tests failed:
    WebRtcAudioQualityBrowserTest.MANUAL_TestAutoGainControlOnLowAudio (e:\b\c\b\win_builder\src\chrome\browser\media\webrtc\webrtc_audio_quality_browsertest.cc:803)
    WebRtcAudioQualityBrowserTest.MANUAL_TestAutoGainIsOffWithAudioProcessingOff (e:\b\c\b\win_builder\src\chrome\browser\media\webrtc\webrtc_audio_quality_browsertest.cc:810)
    WebRtcAudioQualityBrowserTest.MANUAL_TestCallQualityWithAudioFromFakeDevice (e:\b\c\b\win_builder\src\chrome\browser\media\webrtc\webrtc_audio_quality_browsertest.cc:683)
    WebRtcAudioQualityBrowserTest.MANUAL_TestCallQualityWithAudioFromWebAudio (e:\b\c\b\win_builder\src\chrome\browser\media\webrtc\webrtc_audio_quality_browsertest.cc:688)

Maybe this is know - I think I've talked to phoglund@ about this before. It coudl have been something similar too.

We don't know what's causing these things, it might be bad luck or Chromium changes.
Luckily it's back to a solid green again.

This issue is still happening.

https://build.chromium.org/p/chromium.webrtc/builders/Win7%20Tester/builds/33648

[2936:4648:0301/105813.229:ERROR:webrtc_audio_quality_browsertest.cc(473)] PESQ was not able to compute a score; we probably recorded only silence. Please check the output/input volume levels.
e:\b\c\b\win_builder\src\chrome\browser\media\webrtc\webrtc_audio_quality_browsertest.cc(602): error: Value of: succeeded
  Actual: false
Expected: true
Failed to run PESQ.

Can this be related to these CLs ehmaldonado@?

https://codereview.chromium.org/2319033004
https://codereview.chromium.org/2313373002

This seems to be coming back over and over again - the same pattern repeats: it goes red for 5-10 builds or so and then recovers. I'm very close to suggest that we disable this test for good by now. I don't think we have proof for them actually catching any perf regressions. Does anyone else know?

This is happening again:
https://build.chromium.org/p/chromium.webrtc/builders/Win7%20Tester/builds/33901

What about using MediaStream Recording API [1] instead of using the stereo mix device, at least on Windows?
I think the main downside would be that it wouldn't exercise the audio rendering in chrome, but the test as it is is too flaky as to be useful.

That's a great idea to try out (assuming it can also record audio, not just video) if we suspect it's the OS recording tools that are causing the problems.

It could also be that the audio code in Chrome is actually having problems when we see these failures, causing us to actually record just silence. What goes against that is that we've never seen a CL (AFAIK) that could explain either why it went red or why it went green again, so I think we have a problem in our test.

Yeah, we can also record audio. I think it is the sound card that we need to use to record the audio, it is quite hard to configure correctly.

Would using the MediaStream Recording API catch the problems that the audio code in Chrome has, if that were the case?

Re #10: we probably wouldn't cover exactly as much code, but that's rather a good thing than a bad thing. We want our tests to be focused. What this test is supposed to do is to measure how much audio quality we lose when doing a video+audio call. Preferably we trust other tests to cover other problems.

If we can use MediaStream Recording API to get rid of a lot of this (also platform dependent) tooling for this test, we should definitely create a feature bug for that and do it in Q2. Can you file one?

Filed: https://bugs.chromium.org/p/chromium/issues/detail?id=701383

This failed 7 out of 9 runs, see 33901-33909 on https://build.chromium.org/p/chromium.webrtc/builders/Win7%20Tester/. Can we disable this test for now? ehmaldonado@, kjellander@ how long do you think it would take for the fix?

I think you can disable it. The new test should be ready by next week.

They have been green since 33912. I will wait for the next flake to disable as kjellander described the pattern on #6.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a0a47bfebb016ec468bc6d1b4be9b1acb22a4075

commit a0a47bfebb016ec468bc6d1b4be9b1acb22a4075
Author: ehmaldonado <ehmaldonado@chromium.org>
Date: Thu Mar 23 11:54:56 2017

WebRTC: Use the MediaStream Recording API for the audio_quality_browsertest.

This CL uses the MediaStream Recording API to record the audio received
by the right tag. It saves it as a webm file that is later converted to a
wav file using ffmpeg.

This might cause a performance regression because we're using webm+ffmpeg
to generate the wav file. I set a, hopefully high enough, bitrate to
mitigate this

BUG= chromium:681577 
NOTRY=True

Review-Url: https://codereview.chromium.org/2753543010
Cr-Commit-Position: refs/heads/master@{#459050}

[modify] https://crrev.com/a0a47bfebb016ec468bc6d1b4be9b1acb22a4075/chrome/browser/media/webrtc/webrtc_audio_quality_browsertest.cc
[add] https://crrev.com/a0a47bfebb016ec468bc6d1b4be9b1acb22a4075/chrome/test/data/webrtc/audio_extraction.js
[delete] https://crrev.com/5ea4533d1c12a7cbf21c6f4471946765f73b7270/chrome/test/data/webrtc/resources/force_mic_volume_max.exe.sha1
[modify] https://crrev.com/a0a47bfebb016ec468bc6d1b4be9b1acb22a4075/chrome/test/data/webrtc/webrtc_audio_quality_test.html

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e512d162ed486a0a9f71a1e20e3cde8fb20255e1

commit e512d162ed486a0a9f71a1e20e3cde8fb20255e1
Author: ehmaldonado <ehmaldonado@chromium.org>
Date: Thu Mar 23 21:39:13 2017

Revert of WebRTC: Use the MediaStream Recording API for the audio_quality_browsertest. (patchset #8 id:140001 of https://codereview.chromium.org/2753543010/ )

Reason for revert:
This might be causing failures in Win7 bots:

https://build.chromium.org/p/chromium.webrtc/builders/Win7%20Tester/builds/34106

Original issue's description:
> WebRTC: Use the MediaStream Recording API for the audio_quality_browsertest.
>
> This CL uses the MediaStream Recording API to record the audio received
> by the right tag. It saves it as a webm file that is later converted to a
> wav file using ffmpeg.
>
> This might cause a performance regression because we're using webm+ffmpeg
> to generate the wav file. I set a, hopefully high enough, bitrate to
> mitigate this
>
> BUG= chromium:681577 
> NOTRY=True
>
> Review-Url: https://codereview.chromium.org/2753543010
> Cr-Commit-Position: refs/heads/master@{#459050}
> Committed: https://chromium.googlesource.com/chromium/src/+/a0a47bfebb016ec468bc6d1b4be9b1acb22a4075

TBR=kjellander@chromium.org,hbos@chromium.org,mcasas@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= chromium:681577 

Review-Url: https://codereview.chromium.org/2773803002
Cr-Commit-Position: refs/heads/master@{#459234}

[modify] https://crrev.com/e512d162ed486a0a9f71a1e20e3cde8fb20255e1/chrome/browser/media/webrtc/webrtc_audio_quality_browsertest.cc
[delete] https://crrev.com/f1cf4b8eff3ccb7b8d880d82fa1806e9b2d3adde/chrome/test/data/webrtc/audio_extraction.js
[add] https://crrev.com/e512d162ed486a0a9f71a1e20e3cde8fb20255e1/chrome/test/data/webrtc/resources/force_mic_volume_max.exe.sha1
[modify] https://crrev.com/e512d162ed486a0a9f71a1e20e3cde8fb20255e1/chrome/test/data/webrtc/webrtc_audio_quality_test.html

Assigning to ehmaldonado@ as this is being actively worked on with the MediaStream recording stuff.

https://build.chromium.org/p/chromium.webrtc/builders/Win7%20Tester
[2892:4144:0914/031905.719:ERROR:webrtc_audio_quality_browsertest.cc(474)] PESQ was not able to compute a score; we probably recorded only silence. Please check the output/input volume levels.
../../chrome/browser/media/webrtc/webrtc_audio_quality_browsertest.cc(603): error: Value of: succeeded
  Actual: false
Expected: true
Failed to run PESQ.

These errors are coming up consistently now. Can you take another look? 

This machine was most likely reimaged in  bug 764660  yesterday, which would explain this. I forgot to check if it ran fine since I've been OOO.

Patrik/Edward: can you run the manual audio setup on it? It's very cumbersome to RDP to these hosts if you're not on the corp network...

Looks like Patrik took care of it.

Yes, I see >10 green builds in a row now.