Flaky WebRtcMediaRecorderTest.AddingTrackToMediaStreamFiresErrorEvent on chromium.webrtc.MacTester |
|||||||||||
Issue descriptionThis tests has been flaky for a while: - First instance Feb 24, see https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.webrtc%2FMac_Tester%2F65757%2F%2B%2Frecipes%2Fsteps%2Fcontent_browsertests%2F0%2Fstdout. - Last insatnce Mar 13, see https://build.chromium.org/p/chromium.webrtc/builders/Mac%20Tester/builds/65924/steps/content_browsertests/logs/stdio. [ RUN ] WebRtcMediaRecorderTest.AddingTrackToMediaStreamFiresErrorEvent [4991:779:0313/110304.171259:247244577176:INFO:CONSOLE(104)] "Recorder object created.", source: http://127.0.0.1:50026/media/mediarecorder_test.html (104) [4991:779:0313/110304.172206:247245517717:INFO:CONSOLE(22)] "Waiting for", source: http://127.0.0.1:50026/media/mediarecorder_test_utils.js (22) [4994:779:0313/110304.228255:247301571730:ERROR:MediaRecorder.cpp(353)] Amount of tracks in MediaStream has changed. Received signal 11 SEGV_MAPERR 000000000000 [0x0001045e982c] [0x0001045e96e1] [0x7fff91595f1a] [0x7f9d9b63c228] [0x00010699d05a] [0x000102e2d2af] [0x000102e2d037] [0x000102e2cccf] [0x000102ccf6db] [0x000102ccf635] [0x0001045ea195] [0x0001046263a9] [0x00010462676c] [0x000104626c73] [0x00010462baa0] [0x00010460f91a] [0x00010462b424] [0x7fff875aba01] [0x7fff8759db8d] [0x7fff8759d1bf] [0x7fff8759cbd8] [0x00010462bfaf] [0x00010462b897] [0x0001046260d1] [0x000104659c47] [0x00010469e6de] [0x00010469ec8c] [0x00010469475f] [0x7fff9538b05a] [0x7fff9538afd7] [0x7fff953883ed] [end of stack trace] [4991:779:0313/110305.107267:248180611551:ERROR:browser_test_utils.cc(148)] Cannot communicate with DOMMessageQueue. ../../content/browser/webrtc/webrtc_content_browsertest_base.cc:66: Failure Value of: ExecuteScriptAndExtractString(shell(), javascript, &result) Actual: false Expected: true Failed to execute javascript testAddingTrackToMediaStreamFiresErrorEvent();. From javascript: (nothing) When executing 'testAddingTrackToMediaStreamFiresErrorEvent();' ../../content/browser/webrtc/webrtc_content_browsertest_base.cc:90: Failure Failed [ FAILED ] WebRtcMediaRecorderTest.AddingTrackToMediaStreamFiresErrorEvent, where TypeParam = and GetParam() = (1400 ms) [143/143] WebRtcMediaRecorderTest.AddingTrackToMediaStreamFiresErrorEvent (1585 ms) Retrying 1 test (retry #3) [0313/110306.882455:ERROR:kill_posix.cc(84)] Unable to terminate process group 4995: No such process [ RUN ] WebRtcMediaRecorderTest.AddingTrackToMediaStreamFiresErrorEvent [4995:779:0313/110305.768324:248841672581:INFO:CONSOLE(104)] "Recorder object created.", source: http://127.0.0.1:50031/media/mediarecorder_test.html (104) [4995:779:0313/110305.769268:248842611665:INFO:CONSOLE(22)] "Waiting for", source: http://127.0.0.1:50031/media/mediarecorder_test_utils.js (22) [4998:779:0313/110305.824606:248897955079:ERROR:MediaRecorder.cpp(353)] Amount of tracks in MediaStream has changed. Received signal 11 SEGV_MAPERR 000000000000 [0x00010dcd182c] [0x00010dcd16e1] [0x7fff91595f1a] [0x7f89bb52ad18] [0x00011008505a] [0x00010c5152af] [0x00010c515037] [0x00010c514ccf] [0x00010c3b76db] [0x00010c3b7635] [0x00010dcd2195] [0x00010dd0e3a9] [0x00010dd0e76c] [0x00010dd0ec73] [0x00010dd13aa0] [0x00010dcf791a] [0x00010dd13424] [0x7fff875aba01] [0x7fff8759db8d] [0x7fff8759d1bf] [0x7fff8759cbd8] [0x00010dd13faf] [0x00010dd13897] [0x00010dd0e0d1] [0x00010dd41c47] [0x00010dd866de] [0x00010dd86c8c] [0x00010dd7c75f] [0x7fff9538b05a] [0x7fff9538afd7] [0x7fff953883ed] [end of stack trace] [4995:779:0313/110306.708774:249782150992:ERROR:browser_test_utils.cc(148)] Cannot communicate with DOMMessageQueue. ../../content/browser/webrtc/webrtc_content_browsertest_base.cc:66: Failure Value of: ExecuteScriptAndExtractString(shell(), javascript, &result) Actual: false Expected: true Failed to execute javascript testAddingTrackToMediaStreamFiresErrorEvent();. From javascript: (nothing) When executing 'testAddingTrackToMediaStreamFiresErrorEvent();' ../../content/browser/webrtc/webrtc_content_browsertest_base.cc:90: Failure Failed [ FAILED ] WebRtcMediaRecorderTest.AddingTrackToMediaStreamFiresErrorEvent, where TypeParam = and GetParam() = (1412 ms) [144/144] WebRtcMediaRecorderTest.AddingTrackToMediaStreamFiresErrorEvent (1586 ms) 1 test failed: WebRtcMediaRecorderTest.AddingTrackToMediaStreamFiresErrorEvent (../../content/browser/webrtc/webrtc_media_recorder_browsertest.cc:168)
,
Mar 14 2017
Seems like VEA related, I will take a look. [71983:775:0314/152212.752097:170477661688920:ERROR:video_track_recorder.cc(1205)] InitializeEncoder640x480 [71983:29443:0314/152212.752803:170477662393985:ERROR:video_track_recorder.cc(761)] ConfigureEncoderOnEncodingTaskRunner [71981:775:0314/152212.753971:170477663561326:ERROR:vt_video_encode_accelerator_mac.cc(129)] Initialize: input_format=PIXEL_FORMAT_I420, input_visible_size=640x480, output_profile=h264 baseline, initial_bitrate=614400 [71983:775:0314/152212.757916:170477667507923:ERROR:MediaRecorder.cpp(353)] Amount of tracks in MediaStream has changed. [71981:775:0314/152212.772253:170477681844362:ERROR:vt_video_encode_accelerator_mac.cc(528)] VTCompressionSession created with HW encode: 0, input size=640x480 [71981:775:0314/152212.772442:170477682033460:ERROR:vt_video_encode_accelerator_mac.cc(214)] RequestEncodingParametersChange: bitrate=614400: framerate=30 [71983:29443:0314/152212.773211:170477682800848:ERROR:video_track_recorder.cc(593)] ~VEAEncoder [71983:29443:0314/152212.773982:170477683571188:ERROR:video_track_recorder.cc(602)] RequireBitstreamBuffers [71983:29443:0314/152212.774891:170477684481857:ERROR:video_track_recorder.cc(651)] UseOutputBitstreamBufferId Received signal 11 SEGV_MAPERR 000000000000 [0x00010e5f97be] [0x00010e5f985d] [0x00010e5f7cbc] [0x00010e5f96a7] [0x7fffc653ebba]
,
Mar 14 2017
[73505:29187:0314/154838.891434:172063724556782:ERROR:gpu_video_encode_accelerator_host.cc(254)] OnRequireBitstreamBuffers [73505:29187:0314/154838.891478:172063724598872:ERROR:gpu_video_encode_accelerator_host.cc(256)] OnRequireBitstreamBuffers input_count=3, input_coded_size=640x480, output_buffer_size=307200 [73505:29187:0314/154838.891516:172063724636489:ERROR:video_track_recorder.cc(602)] RequireBitstreamBuffers [73505:29187:0314/154838.892395:172063725518656:ERROR:video_track_recorder.cc(656)] UseOutputBitstreamBufferId [73505:29187:0314/154838.893013:172063726135046:ERROR:video_track_recorder.cc(656)] UseOutputBitstreamBufferId [73505:775:0314/154838.893024:172063726144941:ERROR:video_track_recorder.cc(593)] ~VEAEncoder [73505:29187:0314/154838.893105:172063726226538:ERROR:video_track_recorder.cc(656)] UseOutputBitstreamBufferId Received signal 11 SEGV_MAPERR 000000000000 [0x00010e97e7be] [0x00010e97e85d] [0x00010e97ccbc] The issue happens because: - GpuVideoEncodeAcceleratorHost::Initialize() takes a naked ptr to |client| which is VEAEncoder. - GpuVideoEncodeAcceleratorHost::OnRequireBitstreamBuffers() calls client->OnRequireBitstreamBuffers(). - While VEAEncoder::RequireBitstreamBuffers() is running on |encoding_task_runner_|, ~VEAEncoder() gets called on main render thread through ~VideoTrackRecorder() after the error event. ~ VEAEncoder() needs a mechanism similar to H264Encoder::ShutdownEncoder() to make sure that |encoding_task_runner_| is completed before moving on.
,
Mar 14 2017
,
Mar 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/caa1c995a9a8dc99a4d6999d78f796aa4dbdae0f commit caa1c995a9a8dc99a4d6999d78f796aa4dbdae0f Author: emircan <emircan@chromium.org> Date: Wed Mar 15 16:30:03 2017 Move destruction of VEAEncoder to encoding task runner We were seeing random crashes in WebRtcMediaRecorderTest.*FiresErrorEvent tests because: - GpuVideoEncodeAcceleratorHost::Initialize() takes a naked ptr to |client| which is VEAEncoder. - GpuVideoEncodeAcceleratorHost::OnRequireBitstreamBuffers() calls client->OnRequireBitstreamBuffers(). - While VEAEncoder::RequireBitstreamBuffers() is running on |encoding_task_runner_|, ~VEAEncoder() gets called on main render thread through ~VideoTrackRecorder() after the error event. ~ VEAEncoder() needs a mechanism similar to H264Encoder::ShutdownEncoder() to make sure that shutdown tasks on |encoding_task_runner_| are completed before moving on. This CL solves the issues by making sure that dtor completes tasks on encoding task runner via DestroyOnEncodingTaskRunner(). BUG= 701030 TEST=WebRtcMediaRecorderTest.*FiresErrorEvent tests consistently pass on local Mac builds. Review-Url: https://codereview.chromium.org/2750993002 Cr-Commit-Position: refs/heads/master@{#457101} [modify] https://crrev.com/caa1c995a9a8dc99a4d6999d78f796aa4dbdae0f/content/renderer/media_recorder/video_track_recorder.cc
,
Mar 15 2017
,
Mar 16 2017
Your change meets the bar and is auto-approved for M58. Please go ahead and merge the CL to branch 3029 manually. Please contact milestone owner if you have questions. Owners: amineer@(Android), cmasso@(iOS), bhthompson@(ChromeOS), govind@(Desktop) For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Mar 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d23165856a4a8c63c88b0585757e5249da66ba95 commit d23165856a4a8c63c88b0585757e5249da66ba95 Author: emircan <emircan@chromium.org> Date: Thu Mar 16 17:30:49 2017 Merge 58: Move destruction of VEAEncoder to encoding task runner We were seeing random crashes in WebRtcMediaRecorderTest.*FiresErrorEvent tests because: - GpuVideoEncodeAcceleratorHost::Initialize() takes a naked ptr to |client| which is VEAEncoder. - GpuVideoEncodeAcceleratorHost::OnRequireBitstreamBuffers() calls client->OnRequireBitstreamBuffers(). - While VEAEncoder::RequireBitstreamBuffers() is running on |encoding_task_runner_|, ~VEAEncoder() gets called on main render thread through ~VideoTrackRecorder() after the error event. ~ VEAEncoder() needs a mechanism similar to H264Encoder::ShutdownEncoder() to make sure that shutdown tasks on |encoding_task_runner_| are completed before moving on. This CL solves the issues by making sure that dtor completes tasks on encoding task runner via DestroyOnEncodingTaskRunner(). BUG= 701030 TEST=WebRtcMediaRecorderTest.*FiresErrorEvent tests consistently pass on local Mac builds. Review-Url: https://codereview.chromium.org/2750993002 Cr-Commit-Position: refs/heads/master@{#457101} (cherry picked from commit caa1c995a9a8dc99a4d6999d78f796aa4dbdae0f) NOTRY=true NOPRESUBMIT=true TBR=mcasas@chromium.org Review-Url: https://codereview.chromium.org/2753073002 Cr-Commit-Position: refs/branch-heads/3029@{#243} Cr-Branched-From: 939b32ee5ba05c396eef3fd992822fcca9a2e262-refs/heads/master@{#454471} [modify] https://crrev.com/d23165856a4a8c63c88b0585757e5249da66ba95/content/renderer/media_recorder/video_track_recorder.cc
,
Mar 16 2017
,
Mar 21 2017
Issue 703631 has been merged into this issue.
,
Mar 21 2017
I believe the duplicated issue is related and below list of crashes are related to this bug. https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20AND%20custom_data.ChromeCrashProto.channel%3D%27%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27media%3A%3AGpuVideoEncodeAcceleratorHost%3A%3AOnRequireBitstreamBuffers%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D Adding merge request for 57. There are 58 crashes on M57 so far but I believe that would increase with M57 usage. The fix has a very specific purpose and touches only MediaRecorder.
,
Mar 21 2017
This crash is also present on Windows too, Please find the query below, There are total 62 crashes on Windows and 15 on Mac on Current stable i.e., 57.0.2987.110 : https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27media%3A%3AGpuVideoEncodeAcceleratorHost%3A%3AOnRequireBitstreamBuffers%27%20AND%20product.name!%3D%27Chrome_Android%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion:1000
,
Mar 21 2017
emircan@, could you please confirm change is well baked/verified in Canary and extremely safe to merge to M57?
,
Mar 21 2017
It has been baked in canary. There are no crashes after 59.0.3043.0. It has also been merged to beta on 58.0.3029.25 and there are no crashes, but AFAICT that beta hasn't been rolled yet. The fix has a very specific purpose and touches only MediaRecorder.
,
Mar 21 2017
Approving merge to M57 branch 2987 based on comment #11, #12 and #13. Please merge ASAP. FYI: New M58 beta version will go out tomorrow so please verify this fix on beta on Thursday.
,
Mar 21 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/cbd3849cd2254532cc0351c2c9211d343bd53d7c commit cbd3849cd2254532cc0351c2c9211d343bd53d7c Author: emircan <emircan@chromium.org> Date: Tue Mar 21 23:32:04 2017 Merge 57: Move destruction of VEAEncoder to encoding task runner We were seeing random crashes in WebRtcMediaRecorderTest.*FiresErrorEvent tests because: - GpuVideoEncodeAcceleratorHost::Initialize() takes a naked ptr to |client| which is VEAEncoder. - GpuVideoEncodeAcceleratorHost::OnRequireBitstreamBuffers() calls client->OnRequireBitstreamBuffers(). - While VEAEncoder::RequireBitstreamBuffers() is running on |encoding_task_runner_|, ~VEAEncoder() gets called on main render thread through ~VideoTrackRecorder() after the error event. ~ VEAEncoder() needs a mechanism similar to H264Encoder::ShutdownEncoder() to make sure that shutdown tasks on |encoding_task_runner_| are completed before moving on. This CL solves the issues by making sure that dtor completes tasks on encoding task runner via DestroyOnEncodingTaskRunner(). BUG= 701030 TEST=WebRtcMediaRecorderTest.*FiresErrorEvent tests consistently pass on local Mac builds. Review-Url: https://codereview.chromium.org/2750993002 Cr-Commit-Position: refs/heads/master@{#457101} (cherry picked from commit caa1c995a9a8dc99a4d6999d78f796aa4dbdae0f) NOTRY=true NOPRESUBMIT=true TBR=mcasas@chromium.org Review-Url: https://codereview.chromium.org/2766753003 Cr-Commit-Position: refs/branch-heads/2987@{#857} Cr-Branched-From: ad51088c0e8776e8dcd963dbe752c4035ba6dab6-refs/heads/master@{#444943} [modify] https://crrev.com/cbd3849cd2254532cc0351c2c9211d343bd53d7c/content/renderer/media/video_track_recorder.cc
,
Mar 24 2017
There are no crashes after 58.0.3029.25 reported. https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20AND%20custom_data.ChromeCrashProto.channel%3D%27%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27media%3A%3AGpuVideoEncodeAcceleratorHost%3A%3AOnRequireBitstreamBuffers%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D |
|||||||||||
►
Sign in to add a comment |
|||||||||||
Comment 1 by emir...@chromium.org
, Mar 14 2017