Archive audio files from WebRtcAudioQualityBrowserTest |
|||||
Issue descriptionThe WebRtcAudioQualityBrowserTest tests (part of browser_tests) sometimes have issues with flakiness (example: bug 681577 ) which is time-consuming to troubleshoot. It's surprisingly hard to log in to the machine to pull the output file in order to listen to it. This bug tracks implementing an archiving step to execute after browser_tests that will upload audio output files to Google storage and print links to them in the waterfall UI, so it's easy to download and listen to when failures occur. gsutil upload step example: https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/webrtc/api.py?rcl=7ce0eb3b9636e8b74c2ef74446ceb3af92fb8dff&l=244 One problem is that the test simply writes the files as temporary files into the system's temp folder, e.g. [4584:4528:0920/202102.188:ERROR:webrtc_audio_quality_browsertest.cc(474)] PESQ was not able to compute a score; we probably recorded only silence. Please check the output/input volume levels. ../../chrome/browser/media/webrtc/webrtc_audio_quality_browsertest.cc(603): error: Value of: succeeded Actual: false Expected: true Failed to run PESQ. [4584:2572:0920/202102.201:WARNING:audio_sync_reader.cc(172)] ASR: No room in socket buffer.: The pipe is being closed. (0xE8) [4584:4528:0920/202102.224:INFO:user_input_monitor_win.cc(157)] RegisterRawInputDevices() failed for RIDEV_REMOVE: The parameter is incorrect. (0x57) [4584:4528:0920/202102.232:INFO:chrome_cryptauth_service.cc(222)] Profile is not authenticated yet; waiting before starting CryptAuth managers. Test failed; keeping recording(s) at C:\Users\chrome-bot.LABS\AppData\Local\Temp\fcefa3ec-743e-4a99-944e-df574d9f8dba.tmp.wav. Test failed; keeping recording(s) at C:\Users\chrome-bot.LABS\AppData\Local\Temp\879785e0-38d3-48bb-8551-8549a944280b.tmp.wav. Test failed; keeping recording(s) at C:\Users\chrome-bot.LABS\AppData\Local\Temp\962363b2-fcbd-4365-9d62-737f4a18996c.tmp.wav. In order for the recipe to know what files are written, it needs to either pass a flag telling the test where (a directory) to put such files (needs landing a separate CL) or parse stdout from the test (fragile and poor).
,
Oct 17 2017
,
Oct 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a1d732ff7d1c47f37ecbe5c33f859d4c4c30fa87 commit a1d732ff7d1c47f37ecbe5c33f859d4c4c30fa87 Author: Patrik Höglund <phoglund@chromium.org> Date: Fri Oct 20 15:00:37 2017 Add a flag telling WebRTC browser tests where to put recordings. This flag will be used by the WebRTC recipes later. The recipe will be able to tell the test where to put files, and the recipe will then simply upload the files. I highly recommend diffing patch set 1 and 2; patch set 1 just moves a bunch of functions into the test class (because they need to access wav_dump_path_). The diff between 1 and 2 shows what I really did. Bug: 767857 Change-Id: I3af4b849daa817bd03a1308aa785539d3920b951 Reviewed-on: https://chromium-review.googlesource.com/728242 Reviewed-by: Henrik Kjellander <kjellander@chromium.org> Commit-Queue: Patrik Höglund <phoglund@chromium.org> Cr-Commit-Position: refs/heads/master@{#510434} [modify] https://crrev.com/a1d732ff7d1c47f37ecbe5c33f859d4c4c30fa87/chrome/browser/media/webrtc/webrtc_audio_quality_browsertest.cc
,
Oct 27 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/11f06c50f9319346b609e7e4aa8421a97a24ddb4 commit 11f06c50f9319346b609e7e4aa8421a97a24ddb4 Author: Patrik Höglund <phoglund@chromium.org> Date: Fri Oct 27 07:44:19 2017 Make WebRTC bots upload wav files from failed AQ tests. This will make it a lot easier to work out what's going on on the bots when audio quality tests fail. We will be able to listen to recordings by clicking a link in the failed job and unzipping the gathered wav files. This change makes the test receive a new flag --webrtc_save_audio_recordings_in=/some/tmpdir. The temp dir will disappear from the bot after the build is complete. https://chromium-review.googlesource.com/c/chromium/src/+/728242 implements the flag and should land before this CL. Bug:767857 Change-Id: I93bc2b358fd75c509fe125a83616a33c2f9c6bbd Reviewed-on: https://chromium-review.googlesource.com/725292 Commit-Queue: Patrik Höglund <phoglund@chromium.org> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org> Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org> [modify] https://crrev.com/11f06c50f9319346b609e7e4aa8421a97a24ddb4/scripts/slave/recipe_modules/chromium_tests/__init__.py [modify] https://crrev.com/11f06c50f9319346b609e7e4aa8421a97a24ddb4/scripts/slave/recipe_modules/chromium_tests/chromium_webrtc.py [modify] https://crrev.com/11f06c50f9319346b609e7e4aa8421a97a24ddb4/scripts/slave/recipe_modules/chromium_tests/tests/steps/webrtc_perf_test.expected/webrtc_tester.json [modify] https://crrev.com/11f06c50f9319346b609e7e4aa8421a97a24ddb4/scripts/slave/recipe_modules/chromium_tests/tests/steps/webrtc_perf_test.expected/webrtc_fyi_tester.json [modify] https://crrev.com/11f06c50f9319346b609e7e4aa8421a97a24ddb4/scripts/slave/README.recipes.md [add] https://crrev.com/11f06c50f9319346b609e7e4aa8421a97a24ddb4/scripts/slave/recipe_modules/chromium_tests/tests/steps/webrtc_perf_test.expected/no_upload_if_no_wav_files.json [add] https://crrev.com/11f06c50f9319346b609e7e4aa8421a97a24ddb4/scripts/slave/recipe_modules/chromium_tests/tests/steps/webrtc_perf_test.expected/upload_any_wav_files_from_audio_quality_test.json [modify] https://crrev.com/11f06c50f9319346b609e7e4aa8421a97a24ddb4/scripts/slave/recipe_modules/chromium_tests/steps.py [modify] https://crrev.com/11f06c50f9319346b609e7e4aa8421a97a24ddb4/scripts/slave/recipe_modules/chromium_tests/tests/steps/webrtc_perf_test.py
,
Oct 31 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9a3203bde873491510ec470437fc58e74ebb3434 commit 9a3203bde873491510ec470437fc58e74ebb3434 Author: Patrik Höglund <phoglund@chromium.org> Date: Tue Oct 31 17:24:54 2017 Make audio quality test upload if PESQ score is sufficiently bad. R=guidou Bug: 767857 Change-Id: I1a3aa68c3a6996c17bc9d0b96595736e4fb453ab Reviewed-on: https://chromium-review.googlesource.com/741598 Commit-Queue: Patrik Höglund <phoglund@chromium.org> Reviewed-by: Guido Urdaneta <guidou@chromium.org> Cr-Commit-Position: refs/heads/master@{#512860} [modify] https://crrev.com/9a3203bde873491510ec470437fc58e74ebb3434/chrome/browser/media/webrtc/webrtc_audio_quality_browsertest.cc
,
Nov 14 2017
,
Nov 14 2017
Two issues remaining: 1) The looking for wav files step flakily fails on Windows 2) The test fails to delete one file.
,
Nov 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a06cdd8debb95c60fb9dc5cd531b9be7d8d215b1 commit a06cdd8debb95c60fb9dc5cd531b9be7d8d215b1 Author: Patrik Höglund <phoglund@chromium.org> Date: Tue Nov 14 11:12:19 2017 Actually delete the recording from the WebRTC web-audio test. Found this bug when I saw files were getting uploaded (after landing chromium:767857), even when the test was succeeding. This is not WAI, it should only upload when the test fails. Turns out it's a bug in the test where we missed to delete one file. Bug: chromium:767857 Change-Id: I750173bc479bd9d084defb5e7cfbd63ef2d4c2ca Reviewed-on: https://chromium-review.googlesource.com/768667 Reviewed-by: Guido Urdaneta <guidou@chromium.org> Commit-Queue: Patrik Höglund <phoglund@chromium.org> Cr-Commit-Position: refs/heads/master@{#516264} [modify] https://crrev.com/a06cdd8debb95c60fb9dc5cd531b9be7d8d215b1/chrome/browser/media/webrtc/webrtc_audio_quality_browsertest.cc
,
Nov 15 2017
Also turns out uploading doesn't work if the test fails; fixing that too... 2) is fixed, still need to look into 1)
,
Nov 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/ebbd5c7b595d085c7ad74ad90d306e81a0000877 commit ebbd5c7b595d085c7ad74ad90d306e81a0000877 Author: Patrik Höglund <phoglund@chromium.org> Date: Mon Nov 20 14:47:10 2017 Make sure uploads happen even when the test step fails. Turns out failed steps throw exceptions rather than returning something. Now uploading will work when the test fails, which was the primary case I aimed for. Bug: chromium:767857 Change-Id: I1a4667bd5ee73ccdf762015a23e3dbf866a1d9c9 Reviewed-on: https://chromium-review.googlesource.com/771310 Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org> Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org> Commit-Queue: Patrik Höglund <phoglund@chromium.org> [add] https://crrev.com/ebbd5c7b595d085c7ad74ad90d306e81a0000877/scripts/slave/recipe_modules/chromium_tests/tests/steps/webrtc_perf_test.expected/upload_any_wav_files_even_if_test_fails.json [modify] https://crrev.com/ebbd5c7b595d085c7ad74ad90d306e81a0000877/scripts/slave/recipe_modules/chromium_tests/tests/steps/webrtc_perf_test.py [modify] https://crrev.com/ebbd5c7b595d085c7ad74ad90d306e81a0000877/scripts/slave/recipe_modules/chromium_tests/steps.py
,
Nov 20 2017
,
Nov 27 2017
The step "look for wav files" failed on Windows and Linux bots: https://build.chromium.org/p/chromium.webrtc/builders/Win10%20Tester/builds/22547 https://build.chromium.org/p/chromium.webrtc/builders/Win8%20Tester/builds/38661 https://build.chromium.org/p/chromium.webrtc/builders/Linux%20Tester/builds/33321 On the Win 10 tester the next build passed. (The other are currently running the next build.) Win 10: --- look for wav files look for wav files [Error 3] The system cannot find the path specified: 'C:\\b\\build\\slave\\Win10_Tester\\build.dead\\recordings_dir/*.*' ( 0 secs ) --- And stdout: --- 'C:\b\depot_tools\win_tools-2_7_6_bin\python\bin\python.exe' -u 'C:\b\rr\tmpvfdj87\rw\checkout\scripts\slave\.recipe_deps\recipe_engine\recipe_modules\file\resources\fileutil.py' --json-output 'c:\users\chrome~2\appdata\local\temp\tmpmgqx0u.json' listdir 'C:\b\build\slave\Win10_Tester\build.dead\recordings_dir' in dir C:\b\c\b\Win10_Tester: allow_subannotations: False base_name: 'look for wav files' cmd: ['C:\\b\\depot_tools\\win_tools-2_7_6_bin\\python\\bin\\python.exe', '-u', 'C:\\b\\rr\\tmpvfdj87\\rw\\checkout\\scripts\\slave\\.recipe_deps\\recipe_engine\\recipe_modules\\file\\resources\\fileutil.py', '--json-output', 'c:\\users\\chrome~2\\appdata\\local\\temp\\tmpmgqx0u.json', 'listdir', 'C:\\b\\build\\slave\\Win10_Tester\\build.dead\\recordings_dir'] cwd: 'C:\\b\\c\\b\\Win10_Tester' env: {'CHROME_HEADLESS': '1', 'DEPOT_TOOLS_WIN_TOOLCHAIN_ROOT': 'C:\\b\\c\\win_toolchain'} env_prefixes: {} infra_step: True name: 'look for wav files' nest_level: 0 ok_ret: frozenset([0]) stdout: 'c:\\users\\chrome~2\\appdata\\local\\temp\\tmpvvxgwr' step_test_data: <lambda>(...) trigger_specs: () full environment: APPDATA: C:\Users\chrome-bot\AppData\Roaming AWS_CREDENTIAL_FILE: C:\b\build\site_config\.boto BOTO_CONFIG: C:\b\build\site_config\.boto BUILDBOT_BLAMELIST: [u'blakeo@chromium.org', u'ericwilligers@chromium.org', u'falken@chromium.org', u'shanmuga.m@samsung.com'] BUILDBOT_BRANCH: master BUILDBOT_BUILDBOTURL: http://build.chromium.org/p/chromium.webrtc/ BUILDBOT_BUILDERNAME: Win10 Tester BUILDBOT_BUILDNUMBER: 22547 BUILDBOT_CLOBBER: BUILDBOT_GOT_REVISION: None BUILDBOT_MASTERNAME: chromium.webrtc BUILDBOT_REVISION: 44542fba7e29f76987cec86748f48b512b3f97bf BUILDBOT_SCHEDULER: None BUILDBOT_SLAVENAME: build95-b1 CHROME_HEADLESS: 1 CIPD_CACHE_DIR: C:\b\c\cipd COMMONPROGRAMFILES: C:\Program Files (x86)\Common Files COMMONPROGRAMFILES(X86): C:\Program Files (x86)\Common Files COMMONPROGRAMW6432: C:\Program Files\Common Files COMPUTERNAME: BUILD95-B1 COMSPEC: C:\Windows\system32\cmd.exe DEPOT_TOOLS_WIN_TOOLCHAIN_ROOT: C:\b\c\win_toolchain GIT_USER_AGENT: git/2.15.0.windows.1 win32 BUILD95-B1.labs.chromium.org HOME: c:\Users\chrome-bot HOMEDRIVE: C: HOMEPATH: \Users\chrome-bot INFRA_BUILDBOT_MASTER_CLASS_NAME: ChromiumWebRTC INFRA_BUILDBOT_SLAVE_ACTIVE_SUBDIR: INFRA_BUILDBOT_SLAVE_NAME: build95-b1 LOCALAPPDATA: C:\Users\chrome-bot\AppData\Local LOGDOG_COORDINATOR_HOST: logs.chromium.org LOGDOG_STREAM_PREFIX: bb/chromium.webrtc/Win10_Tester/22547 LOGDOG_STREAM_PROJECT: chromium LOGDOG_STREAM_SERVER_PATH: net.pipe:LUCILogDogButler LOGNAME: chrome-bot NUMBER_OF_PROCESSORS: 8 OS: Windows_NT PATH: C:\b\cipd_path_tools;C:\b\cipd_path_tools\bin;C:\b\cipd_client;C:\b\depot_tools\win_tools-2_7_6_bin\python\bin;C:\b\depot_tools\win_tools-2_7_6_bin\python\bin\Scripts;C:\b\depot_tools;C:\b\depot_tools\win_tools-2_7_6_bin\python\bin;C:\Windows\system32;C:\Windows\system32\WBEM;C:\Program Files\7-Zip;C:\cmake\bin;c:\windows\system32\windowspowershell\v1.0;C:\b\build_internal\tools PATHEXT: .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC PROCESSOR_ARCHITECTURE: x86 PROCESSOR_ARCHITEW6432: AMD64 PROCESSOR_IDENTIFIER: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel PROGRAMFILES: C:\Program Files (x86) PROGRAMFILES(X86): C:\Program Files (x86) PROGRAMW6432: C:\Program Files PROMPT: $P$G PWD: C:\b\build\slave\Win10_Tester\build PYTHONIOENCODING: UTF-8 PYTHONPATH: C:\b\build\site_config;C:\b\build\scripts;C:\b\build\scripts\release;C:\b\build\third_party;C:\b\build\third_party\google_api_python_client;C:\b\build\third_party\httplib2\python2;C:\b\build\third_party\infra_libs;C:\b\build\third_party\oauth2client;C:\b\build\third_party\pyasn1;C:\b\build\third_party\pyasn1-modules;C:\b\build\third_party\python-rsa;C:\b\build\third_party\requests_2_10_0;C:\b\build\third_party\setuptools-0.6c11;C:\b\build\third_party\site-packages;C:\b\build\third_party\uritemplate;C:\b\build_internal\site_config;C:\b\build_internal\symsrc;C:\b\build\slave;C:\b\build\third_party\buildbot_slave_8_4;C:\b\build\third_party\twisted_10_2; PYTHONUNBUFFERED: 1 PYTHON_BAT_RUNNER: 1 PYTHON_BIN_RELDIR: win_tools-2_7_6_bin\python\bin SYSTEMDRIVE: C: SYSTEMROOT: C:\Windows TEMP: C:\Users\CHROME~2\AppData\Local\Temp TMP: C:\Users\CHROME~2\AppData\Local\Temp USERDOMAIN: LABS USERNAME: chrome-bot USERPROFILE: C:\Users\chrome-bot VPYTHON_VIRTUALENV_ROOT: C:\b\c\vpython WINDIR: C:\Windows ---
,
Dec 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/90216e0ea943e9c3eddbcddb119f5ec70402070b commit 90216e0ea943e9c3eddbcddb119f5ec70402070b Author: Patrik Höglund <phoglund@chromium.org> Date: Wed Dec 13 09:19:52 2017 Try to use tmp_base instead of cleanup for recordings. It appears api.path['cleanup'] isn't fully working: sometimes the dir it points at goes away before the recipe has finished executing. It does point to build.dead somewhere. Bug: 767857 Change-Id: I9b783c2edf634c8481f2a29cd570e61ab0810c79 Reviewed-on: https://chromium-review.googlesource.com/822470 Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org> Commit-Queue: Patrik Höglund <phoglund@chromium.org> [modify] https://crrev.com/90216e0ea943e9c3eddbcddb119f5ec70402070b/scripts/slave/recipe_modules/chromium_tests/tests/steps/webrtc_perf_test.expected/upload_any_wav_files_even_if_test_fails.json [modify] https://crrev.com/90216e0ea943e9c3eddbcddb119f5ec70402070b/scripts/slave/recipe_modules/chromium_tests/tests/steps/webrtc_perf_test.expected/no_upload_if_no_wav_files.json [modify] https://crrev.com/90216e0ea943e9c3eddbcddb119f5ec70402070b/scripts/slave/recipe_modules/chromium_tests/tests/steps/webrtc_perf_test.expected/webrtc_fyi_tester.json [modify] https://crrev.com/90216e0ea943e9c3eddbcddb119f5ec70402070b/scripts/slave/recipe_modules/chromium_tests/tests/steps/webrtc_perf_test.expected/upload_any_wav_files_from_audio_quality_test.json [modify] https://crrev.com/90216e0ea943e9c3eddbcddb119f5ec70402070b/scripts/slave/recipe_modules/chromium_tests/steps.py [modify] https://crrev.com/90216e0ea943e9c3eddbcddb119f5ec70402070b/scripts/slave/recipe_modules/chromium_tests/tests/steps/webrtc_perf_test.expected/webrtc_tester.json
,
Dec 15 2017
This appears to work now! |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by ehmaldonado@chromium.org
, Sep 22 2017Owner: phoglund@chromium.org