New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 0 users

Issue metadata

Status: Fixed
Owner:
Closed: Sep 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug



Sign in to add a comment
link

Issue 681577: Flaky audio quality test failures on WebRTC Win 7 Tester

Reported by grunell@chromium.org, Jan 16 2017 Project Member

Issue description

I suspect this could be an infra issue.

Last successful:
https://build.chromium.org/p/chromium.webrtc/builders/Win7%20Tester/builds/32670

First failure:
https://build.chromium.org/p/chromium.webrtc/builders/Win7%20Tester/builds/32671

One piece of output:

-----
Reading reference file C:\Users\chrome-bot.LABS\AppData\Local\Temp\B705.tmp.wav...done.
Reading degraded file C:\Users\chrome-bot.LABS\AppData\Local\Temp\B706.tmp.wav...done.
An error of type 2  (Reference or Degraded below 1/4 second - processing stopped ) occurred during processing.

[5624:5252:0116/070818.243:ERROR:webrtc_audio_quality_browsertest.cc(473)] PESQ was not able to compute a score; we probably recorded only silence. Please check the output/input volume levels.
e:\b\c\b\win_builder\src\chrome\browser\media\webrtc\webrtc_audio_quality_browsertest.cc(602): error: Value of: succeeded
  Actual: false
Expected: true
Failed to run PESQ.
-----

I can't find any suspicious CL.

kjellander@: does it make sense to try to restart/clobber/something the bot?
 

Comment 1 by grunell@chromium.org, Jan 16 2017

Summary: Flaky audio quality test failures on WebRTC Win 7 Tester (was: Audio quality test failures on WebRTC Win 7 Tester)
This is flaky. 4 tests:

4 tests failed:
    WebRtcAudioQualityBrowserTest.MANUAL_TestAutoGainControlOnLowAudio (e:\b\c\b\win_builder\src\chrome\browser\media\webrtc\webrtc_audio_quality_browsertest.cc:803)
    WebRtcAudioQualityBrowserTest.MANUAL_TestAutoGainIsOffWithAudioProcessingOff (e:\b\c\b\win_builder\src\chrome\browser\media\webrtc\webrtc_audio_quality_browsertest.cc:810)
    WebRtcAudioQualityBrowserTest.MANUAL_TestCallQualityWithAudioFromFakeDevice (e:\b\c\b\win_builder\src\chrome\browser\media\webrtc\webrtc_audio_quality_browsertest.cc:683)
    WebRtcAudioQualityBrowserTest.MANUAL_TestCallQualityWithAudioFromWebAudio (e:\b\c\b\win_builder\src\chrome\browser\media\webrtc\webrtc_audio_quality_browsertest.cc:688)

Maybe this is know - I think I've talked to phoglund@ about this before. It coudl have been something similar too.

Comment 2 by kjellander@chromium.org, Jan 16 2017

Components: Infra>Client>WebRTC

Comment 3 by kjellander@chromium.org, Jan 19 2017

Status: WontFix (was: Untriaged)
We don't know what's causing these things, it might be bad luck or Chromium changes.
Luckily it's back to a solid green again.

Comment 4 by emir...@chromium.org, Mar 1 2017

Status: Available (was: WontFix)
This issue is still happening.

https://build.chromium.org/p/chromium.webrtc/builders/Win7%20Tester/builds/33648

[2936:4648:0301/105813.229:ERROR:webrtc_audio_quality_browsertest.cc(473)] PESQ was not able to compute a score; we probably recorded only silence. Please check the output/input volume levels.
e:\b\c\b\win_builder\src\chrome\browser\media\webrtc\webrtc_audio_quality_browsertest.cc(602): error: Value of: succeeded
  Actual: false
Expected: true
Failed to run PESQ.

Comment 5 by emir...@chromium.org, Mar 1 2017

Comment 6 by kjellander@chromium.org, Mar 1 2017

Cc: ehmaldonado@chromium.org
This seems to be coming back over and over again - the same pattern repeats: it goes red for 5-10 builds or so and then recovers. I'm very close to suggest that we disable this test for good by now. I don't think we have proof for them actually catching any perf regressions. Does anyone else know?

Comment 8 by ehmaldonado@chromium.org, Mar 14 2017

What about using MediaStream Recording API [1] instead of using the stereo mix device, at least on Windows?
I think the main downside would be that it wouldn't exercise the audio rendering in chrome, but the test as it is is too flaky as to be useful.

Comment 9 by kjellander@chromium.org, Mar 14 2017

That's a great idea to try out (assuming it can also record audio, not just video) if we suspect it's the OS recording tools that are causing the problems.

It could also be that the audio code in Chrome is actually having problems when we see these failures, causing us to actually record just silence. What goes against that is that we've never seen a CL (AFAIK) that could explain either why it went red or why it went green again, so I think we have a problem in our test.

Comment 10 by ehmaldonado@chromium.org, Mar 14 2017

Yeah, we can also record audio. I think it is the sound card that we need to use to record the audio, it is quite hard to configure correctly.

Would using the MediaStream Recording API catch the problems that the audio code in Chrome has, if that were the case?

Comment 11 by kjellander@chromium.org, Mar 14 2017

Re #10: we probably wouldn't cover exactly as much code, but that's rather a good thing than a bad thing. We want our tests to be focused. What this test is supposed to do is to measure how much audio quality we lose when doing a video+audio call. Preferably we trust other tests to cover other problems.

If we can use MediaStream Recording API to get rid of a lot of this (also platform dependent) tooling for this test, we should definitely create a feature bug for that and do it in Q2. Can you file one?

Comment 13 by emir...@chromium.org, Mar 14 2017

This failed 7 out of 9 runs, see 33901-33909 on https://build.chromium.org/p/chromium.webrtc/builders/Win7%20Tester/. Can we disable this test for now? ehmaldonado@, kjellander@ how long do you think it would take for the fix?

Comment 14 by ehmaldonado@chromium.org, Mar 14 2017

I think you can disable it. The new test should be ready by next week.

Comment 15 by emir...@chromium.org, Mar 15 2017

They have been green since 33912. I will wait for the next flake to disable as kjellander described the pattern on #6.

Comment 16 by bugdroid1@chromium.org, Mar 23 2017

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a0a47bfebb016ec468bc6d1b4be9b1acb22a4075

commit a0a47bfebb016ec468bc6d1b4be9b1acb22a4075
Author: ehmaldonado <ehmaldonado@chromium.org>
Date: Thu Mar 23 11:54:56 2017

WebRTC: Use the MediaStream Recording API for the audio_quality_browsertest.

This CL uses the MediaStream Recording API to record the audio received
by the right tag. It saves it as a webm file that is later converted to a
wav file using ffmpeg.

This might cause a performance regression because we're using webm+ffmpeg
to generate the wav file. I set a, hopefully high enough, bitrate to
mitigate this

BUG= chromium:681577 
NOTRY=True

Review-Url: https://codereview.chromium.org/2753543010
Cr-Commit-Position: refs/heads/master@{#459050}

[modify] https://crrev.com/a0a47bfebb016ec468bc6d1b4be9b1acb22a4075/chrome/browser/media/webrtc/webrtc_audio_quality_browsertest.cc
[add] https://crrev.com/a0a47bfebb016ec468bc6d1b4be9b1acb22a4075/chrome/test/data/webrtc/audio_extraction.js
[delete] https://crrev.com/5ea4533d1c12a7cbf21c6f4471946765f73b7270/chrome/test/data/webrtc/resources/force_mic_volume_max.exe.sha1
[modify] https://crrev.com/a0a47bfebb016ec468bc6d1b4be9b1acb22a4075/chrome/test/data/webrtc/webrtc_audio_quality_test.html

Comment 17 by bugdroid1@chromium.org, Mar 23 2017

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e512d162ed486a0a9f71a1e20e3cde8fb20255e1

commit e512d162ed486a0a9f71a1e20e3cde8fb20255e1
Author: ehmaldonado <ehmaldonado@chromium.org>
Date: Thu Mar 23 21:39:13 2017

Revert of WebRTC: Use the MediaStream Recording API for the audio_quality_browsertest. (patchset #8 id:140001 of https://codereview.chromium.org/2753543010/ )

Reason for revert:
This might be causing failures in Win7 bots:

https://build.chromium.org/p/chromium.webrtc/builders/Win7%20Tester/builds/34106

Original issue's description:
> WebRTC: Use the MediaStream Recording API for the audio_quality_browsertest.
>
> This CL uses the MediaStream Recording API to record the audio received
> by the right tag. It saves it as a webm file that is later converted to a
> wav file using ffmpeg.
>
> This might cause a performance regression because we're using webm+ffmpeg
> to generate the wav file. I set a, hopefully high enough, bitrate to
> mitigate this
>
> BUG= chromium:681577 
> NOTRY=True
>
> Review-Url: https://codereview.chromium.org/2753543010
> Cr-Commit-Position: refs/heads/master@{#459050}
> Committed: https://chromium.googlesource.com/chromium/src/+/a0a47bfebb016ec468bc6d1b4be9b1acb22a4075

TBR=kjellander@chromium.org,hbos@chromium.org,mcasas@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= chromium:681577 

Review-Url: https://codereview.chromium.org/2773803002
Cr-Commit-Position: refs/heads/master@{#459234}

[modify] https://crrev.com/e512d162ed486a0a9f71a1e20e3cde8fb20255e1/chrome/browser/media/webrtc/webrtc_audio_quality_browsertest.cc
[delete] https://crrev.com/f1cf4b8eff3ccb7b8d880d82fa1806e9b2d3adde/chrome/test/data/webrtc/audio_extraction.js
[add] https://crrev.com/e512d162ed486a0a9f71a1e20e3cde8fb20255e1/chrome/test/data/webrtc/resources/force_mic_volume_max.exe.sha1
[modify] https://crrev.com/e512d162ed486a0a9f71a1e20e3cde8fb20255e1/chrome/test/data/webrtc/webrtc_audio_quality_test.html

Comment 18 by kjellander@chromium.org, Mar 27 2017

Cc: -ehmaldonado@chromium.org
Components: -Infra>Client>WebRTC Blink>WebRTC>Audio
Owner: ehmaldonado@chromium.org
Status: Assigned (was: Available)
Assigning to ehmaldonado@ as this is being actively worked on with the MediaStream recording stuff.

Comment 19 by emir...@chromium.org, Sep 14 2017

Labels: -Pri-2 Pri-1
https://build.chromium.org/p/chromium.webrtc/builders/Win7%20Tester
[2892:4144:0914/031905.719:ERROR:webrtc_audio_quality_browsertest.cc(474)] PESQ was not able to compute a score; we probably recorded only silence. Please check the output/input volume levels.
../../chrome/browser/media/webrtc/webrtc_audio_quality_browsertest.cc(603): error: Value of: succeeded
  Actual: false
Expected: true
Failed to run PESQ.

These errors are coming up consistently now. Can you take another look?

Comment 20 by kjellander@chromium.org, Sep 14 2017

This machine was most likely reimaged in  bug 764660  yesterday, which would explain this. I forgot to check if it ran fine since I've been OOO.

Patrik/Edward: can you run the manual audio setup on it? It's very cumbersome to RDP to these hosts if you're not on the corp network...

Comment 21 by ehmaldonado@chromium.org, Sep 14 2017

Looks like Patrik took care of it.

Comment 22 by kjellander@chromium.org, Sep 15 2017

Status: Fixed (was: Assigned)
Yes, I see >10 green builds in a row now.

Sign in to add a comment