WebRtcVideoQualityBrowserTest.MANUAL_TestVideoQualityH264 crash |
|||||||||||||
Issue descriptionOn https://build.chromium.org/p/chromium.webrtc/builders/Mac%20Tester and https://build.chromium.org/p/chromium.webrtc.fyi/builders/Mac%20Tester since yesterday it seems. Example https://build.chromium.org/p/chromium.webrtc/builders/Mac%20Tester/builds/53179/steps/browser_tests/logs/stdio Seeing ---------------- [10733:1299:0405/020948:FATAL:vt_video_decode_accelerator_mac.cc(863)] Check failed: CFGetRetainCount(picture_info->cv_image) == 1 (2 vs. 1) 0 Chromium Framework 0x000000010573d9d3 _ZN4base5debug10StackTraceC1Ev + 19 1 Chromium Framework 0x000000010575ae99 _ZN7logging10LogMessageD2Ev + 73 2 Chromium Framework 0x000000010649ac2a _ZN7content24VTVideoDecodeAccelerator18ReusePictureBufferEi + 378 3 Chromium Framework 0x000000010646ce7c _ZN7content25GpuVideoDecodeAccelerator20OnReusePictureBufferEi + 108 4 Chromium Framework 0x000000010646cd10 _ZN3IPC8MessageTI50AcceleratedVideoDecoderMsg_ReusePictureBuffer_MetaNSt3__15tupleIJiEEEvE8DispatchIN7content25GpuVideoDecodeAcceleratorES8_vMS8_FviEEEbPKNS_7MessageEPT_PT0_PT1_T2_ + 112 5 Chromium Framework 0x000000010646be71 _ZN7content25GpuVideoDecodeAccelerator17OnMessageReceivedERKN3IPC7MessageE + 529 6 Chromium Framework 0x00000001063c655c _ZN3IPC13MessageRouter12RouteMessageERKNS_7MessageE + 140 7 Chromium Framework 0x0000000106456227 _ZN7content10GpuChannel19HandleMessageHelperERKN3IPC7MessageE + 151 8 Chromium Framework 0x0000000106456164 _ZN7content10GpuChannel13HandleMessageERK13scoped_refptrINS_22GpuChannelMessageQueueEE + 564 9 Chromium Framework 0x00000001064585e2 _ZN4base8internal7InvokerINS_13IndexSequenceIJLm0ELm1EEEENS0_9BindStateINS0_15RunnableAdapterIMN7content10GpuChannelEFvRK13scoped_refptrINS6_22GpuChannelMessageQueueEEEEEFvPS7_SC_EJNS_7WeakPtrIS7_EESC_EEENS0_12InvokeHelperILb1EvSF_EEFvvEE3RunEPNS0_13BindStateBaseE + 114 10 Chromium Framework 0x000000010573df2b _ZN4base5debug13TaskAnnotator7RunTaskEPKcRKNS_11PendingTaskE + 187 11 Chromium Framework 0x000000010576b743 _ZN4base11MessageLoop7RunTaskERKNS_11PendingTaskE + 579 12 Chromium Framework 0x000000010576ba5c _ZN4base11MessageLoop21DeferOrRunPendingTaskERKNS_11PendingTaskE + 44 13 Chromium Framework 0x000000010576bc4b _ZN4base11MessageLoop6DoWorkEv + 299 14 Chromium Framework 0x000000010572fbd7 _ZN4base24MessagePumpCFRunLoopBase7RunWorkEv + 55 15 Chromium Framework 0x000000010575c0fa _ZN4base3mac15CallWithEHFrameEU13block_pointerFvvE + 10 16 Chromium Framework 0x000000010572f584 _ZN4base24MessagePumpCFRunLoopBase13RunWorkSourceEPv + 68 17 CoreFoundation 0x00007fff81d09a01 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17 18 CoreFoundation 0x00007fff81cfbb8d __CFRunLoopDoSources0 + 269 19 CoreFoundation 0x00007fff81cfb1bf __CFRunLoopRun + 927 20 CoreFoundation 0x00007fff81cfabd8 CFRunLoopRunSpecific + 296 21 Chromium Framework 0x000000010572ffff _ZN4base20MessagePumpCFRunLoop5DoRunEPNS_11MessagePump8DelegateE + 79 22 Chromium Framework 0x000000010572fa0f _ZN4base24MessagePumpCFRunLoopBase3RunEPNS_11MessagePump8DelegateE + 127 23 Chromium Framework 0x000000010576b2a2 _ZN4base11MessageLoop10RunHandlerEv + 210 24 Chromium Framework 0x0000000105787c87 _ZN4base7RunLoop3RunEv + 183 25 Chromium Framework 0x000000010576a547 _ZN4base11MessageLoop3RunEv + 103 26 Chromium Framework 0x00000001056e0395 _ZN7content7GpuMainERKNS_18MainFunctionParamsE + 3301 27 Chromium Framework 0x00000001056edebd _ZN7content21ContentMainRunnerImpl3RunEv + 253 28 Chromium Framework 0x00000001056ed0f6 _ZN7content11ContentMainERKNS_17ContentMainParamsE + 54 29 Chromium Framework 0x00000001050f06d2 ChromeMain + 66 30 Chromium Helper 0x0000000105089d62 main + 530 31 Chromium Helper 0x0000000105089b44 start + 52 32 ??? 0x0000000000000012 0x0 + 18 [10739:32771:0405/020948:ERROR:rtc_video_decoder.cc(488)] VDA Error:4 [10730:1299:0405/020948:ERROR:gpu_process_transport_factory.cc(663)] Lost UI shared context. [10739:37899:0405/020948:ERROR:rtc_video_decoder.cc(164)] Decoding error occurred. [10739:37899:0405/020948:WARNING:generic_decoder.cc(158)] Failed to decode frame with timestamp 636982447, error code: -1 [10737:31747:0405/020948:FATAL:gpu_video_decode_accelerator_host.cc(123)] Check failed: false. Failed to duplicate buffer handler 0 Chromium Framework 0x0000000100d3c9d3 _ZN4base5debug10StackTraceC1Ev + 19 1 Chromium Framework 0x0000000100d59e99 _ZN7logging10LogMessageD2Ev + 73 2 Chromium Framework 0x0000000101a49545 _ZN7content29GpuVideoDecodeAcceleratorHost6DecodeERKN5media15BitstreamBufferE + 325 3 Chromium Framework 0x0000000105a8b420 _ZN7content15RTCVideoDecoder19RequestBufferDecodeEv + 1040 4 Chromium Framework 0x0000000105a8dacb _ZN4base8internal7InvokerINS_13IndexSequenceIJLm0EEEENS0_9BindStateINS0_15RunnableAdapterIMN7content15RTCVideoDecoderEFvvEEEFvPS7_EJNS_7WeakPtrIS7_EEEEENS0_12InvokeHelperILb1EvSA_EEFvvEE3RunEPNS0_13BindStateBaseE + 107 5 Chromium Framework 0x0000000100d3cf2b _ZN4base5debug13TaskAnnotator7RunTaskEPKcRKNS_11PendingTaskE + 187 6 Chromium Framework 0x0000000100d6a743 _ZN4base11MessageLoop7RunTaskERKNS_11PendingTaskE + 579 7 Chromium Framework 0x0000000100d6aa5c _ZN4base11MessageLoop21DeferOrRunPendingTaskERKNS_11PendingTaskE + 44 8 Chromium Framework 0x0000000100d6ac4b _ZN4base11MessageLoop6DoWorkEv + 299 9 Chromium Framework 0x0000000100d2ebd7 _ZN4base24MessagePumpCFRunLoopBase7RunWorkEv + 55 10 Chromium Framework 0x0000000100d5b0fa _ZN4base3mac15CallWithEHFrameEU13block_pointerFvvE + 10 11 Chromium Framework 0x0000000100d2e584 _ZN4base24MessagePumpCFRunLoopBase13RunWorkSourceEPv + 68 12 CoreFoundation 0x00007fff81d09a01 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17 13 CoreFoundation 0x00007fff81cfbb8d __CFRunLoopDoSources0 + 269 14 CoreFoundation 0x00007fff81cfb1bf __CFRunLoopRun + 927 15 CoreFoundation 0x00007fff81cfabd8 CFRunLoopRunSpecific + 296 16 Chromium Framework 0x0000000100d2efff _ZN4base20MessagePumpCFRunLoop5DoRunEPNS_11MessagePump8DelegateE + 79 17 Chromium Framework 0x0000000100d2ea0f _ZN4base24MessagePumpCFRunLoopBase3RunEPNS_11MessagePump8DelegateE + 127 18 Chromium Framework 0x0000000100d6a2a2 _ZN4base11MessageLoop10RunHandlerEv + 210 19 Chromium Framework 0x0000000100d86c87 _ZN4base7RunLoop3RunEv + 183 20 Chromium Framework 0x0000000100d69547 _ZN4base11MessageLoop3RunEv + 103 21 Chromium Framework 0x0000000100dae02e _ZN4base6Thread10ThreadMainEv + 430 22 Chromium Framework 0x0000000100da9038 _ZN4base12_GLOBAL__N_110ThreadFuncEPv + 104 23 libsystem_pthread.dylib 0x00007fff8fae905a _pthread_body + 131 24 libsystem_pthread.dylib 0x00007fff8fae8fd7 _pthread_body + 0 25 libsystem_pthread.dylib 0x00007fff8fae63ed thread_start + 13 -------------------------- and -------------------------- BrowserTestBase signal handler received SIGTERM. Backtrace: 0 browser_tests 0x00000001019a1773 _ZN4base5debug10StackTraceC1Ev + 19 1 browser_tests 0x0000000101a760e1 _ZN7content12_GLOBAL__N_1L27DumpStackTraceSignalHandlerEi + 65 2 libsystem_platform.dylib 0x00007fff8bcf3f1a _sigtramp + 26 3 AppKit 0x00007fff70f07570 __block_descriptor_tmp218 + 304 4 CoreFoundation 0x00007fff81d40217 __CFRunLoopDoTimer + 1735 5 CoreFoundation 0x00007fff81db353d __CFRunLoopDoTimers + 301 6 CoreFoundation 0x00007fff81cfb608 __CFRunLoopRun + 2024 7 CoreFoundation 0x00007fff81cfabd8 CFRunLoopRunSpecific + 296 8 HIToolbox 0x00007fff8773656f RunCurrentEventLoopInMode + 235 9 HIToolbox 0x00007fff877362ea ReceiveNextEventCommon + 431 10 HIToolbox 0x00007fff8773612b _BlockUntilNextEventMatchingListInModeWithFilter + 71 11 AppKit 0x00007fff846e48ab _DPSNextEvent + 978 12 AppKit 0x00007fff846e3e58 -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 346 13 AppKit 0x00007fff846d9af3 -[NSApplication run] + 594 14 browser_tests 0x000000010199454f _ZN4base24MessagePumpNSApplication5DoRunEPNS_11MessagePump8DelegateE + 351 15 browser_tests 0x0000000101993abf _ZN4base24MessagePumpCFRunLoopBase3RunEPNS_11MessagePump8DelegateE + 127 16 browser_tests 0x00000001019cd952 _ZN4base11MessageLoop10RunHandlerEv + 210 17 browser_tests 0x00000001019ea6e7 _ZN4base7RunLoop3RunEv + 183 18 browser_tests 0x0000000106eac7df _ZN7content14RunThisRunLoopEPN4base7RunLoopE + 79 19 browser_tests 0x0000000101a77e0b _ZN7content12_GLOBAL__N_119ExecuteScriptHelperEPNS_15RenderFrameHostERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEEPNS3_10unique_ptrIN4base5ValueENS3_14default_deleteISE_EEEE + 267 20 browser_tests 0x0000000101a78447 _ZN7content29ExecuteScriptAndExtractStringERKNS_17ToRenderFrameHostERKNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEEPS9_ + 119 21 browser_tests 0x0000000100d5c5d7 _ZNK14WebRtcTestBase17ExecuteJavascriptERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEEPN7content11WebContentsE + 71 22 browser_tests 0x0000000100d5d4e9 _ZNK14WebRtcTestBase6HangUpEPN7content11WebContentsE + 73 23 browser_tests 0x0000000100d6dd64 _ZN29WebRtcVideoQualityBrowserTest16TestVideoQualityERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE + 1044 24 browser_tests 0x0000000100d6e733 _ZN62WebRtcVideoQualityBrowserTest_MANUAL_TestVideoQualityH264_Test19RunTestOnMainThreadEv + 83 25 browser_tests 0x000000010198bea9 _ZN20InProcessBrowserTest23RunTestOnMainThreadLoopEv + 169 26 browser_tests 0x0000000101a75ecb _ZN7content15BrowserTestBase28ProxyRunTestOnMainThreadLoopEv + 235 27 browser_tests 0x0000000101441e85 _ZN22ChromeBrowserMainParts25PreMainMessageLoopRunImplEv + 4597 28 browser_tests 0x0000000101440b5a _ZN22ChromeBrowserMainParts21PreMainMessageLoopRunEv + 122 29 browser_tests 0x0000000105706096 _ZN7content15BrowserMainLoop21PreMainMessageLoopRunEv + 182 30 browser_tests 0x0000000105a7e9e0 _ZN7content17StartupTaskRunner14RunAllTasksNowEv + 48 31 browser_tests 0x0000000105704604 _ZN7content15BrowserMainLoop18CreateStartupTasksEv + 644 32 browser_tests 0x0000000105708a72 _ZN7content21BrowserMainRunnerImpl10InitializeERKNS_18MainFunctionParamsE + 690 33 browser_tests 0x0000000105701c45 _ZN7content11BrowserMainERKNS_18MainFunctionParamsE + 149 34 browser_tests 0x0000000106e7979d _ZN7content21ContentMainRunnerImpl3RunEv + 253 35 browser_tests 0x0000000106e789d6 _ZN7content11ContentMainERKNS_17ContentMainParamsE + 54 36 browser_tests 0x0000000101a75c52 _ZN7content15BrowserTestBase5SetUpEv + 914 37 browser_tests 0x000000010198b1f4 _ZN20InProcessBrowserTest5SetUpEv + 308 38 browser_tests 0x0000000101f2ddfb _ZN7testing4Test3RunEv + 299 39 browser_tests 0x0000000101f2ea3a _ZN7testing8TestInfo3RunEv + 410 40 browser_tests 0x0000000101f2f003 _ZN7testing8TestCase3RunEv + 451 41 browser_tests 0x0000000101f35609 _ZN7testing8internal12UnitTestImpl11RunAllTestsEv + 841 42 browser_tests 0x0000000101f3528d _ZN7testing8UnitTest3RunEv + 269 43 browser_tests 0x0000000101a56886 _ZN4base9TestSuite3RunEv + 150 44 browser_tests 0x000000010197165f _ZN21ChromeTestSuiteRunner12RunTestSuiteEiPPc + 31 45 browser_tests 0x0000000106ea9a2d _ZN7content11LaunchTestsEPNS_20TestLauncherDelegateEiiPPc + 397 46 browser_tests 0x0000000101275a6a main + 90 47 browser_tests 0x000000010074b934 start + 52 48 ??? 0x000000000000000b 0x0 + 11 [95/95] WebRtcVideoQualityBrowserTests/WebRtcVideoQualityBrowserTest.MANUAL_TestVideoQualityH264/1 (TIMED OUT) 1 test timed out: WebRtcVideoQualityBrowserTests/WebRtcVideoQualityBrowserTest.MANUAL_TestVideoQualityH264/1 (../../chrome/browser/media/webrtc_video_quality_browsertest.cc:359) --------------------------
,
Apr 5 2016
Hmm, maybe this is hbos@ stuff? (Cc him)
,
Apr 5 2016
or maybe emircan? Assigned to hbos, cc emircan.
,
Apr 5 2016
emircan@ can you take a look at this. hbos@ is ooo today and maybe tomorrow.
,
Apr 5 2016
,
Apr 6 2016
The M51 cut is this week, this must be sorted out asap. hbos@: if you work today, please have a look at this.
,
Apr 6 2016
Sorry, I'm still sick. Might be able to take a look tomorrow if I feel better, though I'm not familiar with the code that is crashing (HW stuff). I know the test and the SW encoder/decoder the test was originally meant to test, but it will choose HW over SW when available. +jbauman who did changes to related files recently (https://codereview.chromium.org/1751323002). Could your CL have caused this (haven't looked closely) or do you know people who might know this code better? (If you want to run this test you have to build with proprietary_codecs=1 and ffmpeg_branding=Chrome, and the feature WebRTC-H264WithOpenH264FFmpeg enabled.)
,
Apr 6 2016
I think ccameron's recent changes to video decoding are the most likely suspects, though I can't be sure.
,
Apr 6 2016
Yeah, this is probably me. Do we have the particular version? Most of the related work has been reverted and is being added back in.
,
Apr 7 2016
See the two builders on top (in the initial report). They have failed flakily since April 4, and still do.
,
Apr 7 2016
Actually, that assert is going to be not only invalid, but unpredictably invalid (we'll be sending the CVPixelBuffer to an AVSampleBufferDisplayLayer). IMO we should just delete it.
,
Apr 7 2016
If the sample buffer is in use by the AVSampleBufferDisplayLayer, is it safe to reuse it in the decoder? It could have a new image decoded into it while it's being displayed, right?
,
Apr 7 2016
We don't actually end up reusing the CVPixelBuffer -- we release it back into the CVPixelBufferPool. If we (or the AVSampleBufferDisplayLayer) have any extra retains out on it, the CVPixelBufferPool will take them into account. The name of the function is something of a misnomer here.
,
Apr 8 2016
It looks like the bots now fail consistently since https://build.chromium.org/p/chromium.webrtc/builders/Mac%20Tester/builds/53264 CC some people FYI.
,
Apr 8 2016
The CHECK is one problem, but the problem in #14 is probably a different problem. That one is a timeout and not a crash. We get tons of this: [8219:37667:0407/153659:ERROR:video_sender.cc(313)] Failed to encode frame. Error code: -1 [8217:37407:0407/153659:ERROR:video_sender.cc(313)] Failed to encode frame. Error code: -1 ...
,
Apr 8 2016
I would strongly suspect Change #184312 Changed by emircan@chromium.org Changed at Thu 07 Apr 2016 14:29:21 Repository https://chromium.googlesource.com/chromium/src Project chromium Branch master Revision e123143c71aae717dca25f3e2767994b3098ccc9 Comments Tie kWebRtcH264WithOpenH264FFmpeg and kEnableWebRtcHWH264Encoding flags on Mac This CL enables using VideoToolbox encoder when kWebRtcH264WithOpenH264FFmpeg flag is on. BUG= 597334 Review URL: https://codereview.chromium.org/1852523002 Cr-Commit-Position: refs/heads/master@{#385860} Changed files chrome/app/generated_resources.grd content/renderer/media/rtc_video_encoder_factory.cc causes #16. I'll go ahead and make a speculative revert for now.
,
Apr 8 2016
,
Apr 8 2016
"Tie kWebRtcH264WithOpenH264FFmpeg and kEnableWebRtcHWH264Encoding flags on Mac" was reverted in https://codereview.chromium.org/1868133002.
,
Apr 8 2016
Seems like the revert made the bot go green! https://build.chromium.org/p/chromium.webrtc/builders/Mac%20Tester/builds/53274
,
Apr 8 2016
Wo-hoo! Note this does not solve the problem in this bug, but now you can carry on and try to solve that flake.
,
Apr 8 2016
Nice! Maybe it goes back to being flaky?
,
Apr 8 2016
CC guidou, next week's WebRTC-in-Chrome sheriff.
,
Apr 8 2016
A thought: someone should fix the test running in the CQ / public waterfalls. We do have a CQ test that runs h264 but video detection is turned off for Mac: https://cs.corp.google.com/clankium/src/chrome/browser/media/webrtc_browsertest.cc?q=webrtc_browsertest.cc&dr&l=65 hbos, could you do it when you get back? It's quite possible you can just remove the #ifdefs for Mac, I don't see why it shouldn't work.
,
Apr 11 2016
Still flaky for sure, just occurred on the mac tester: https://build.chromium.org/p/chromium.webrtc/builders/Mac%20Tester/builds/53357
,
Apr 12 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/2d474e5aa58b32a51f7e4e3ec91e915c6657cfd1 commit 2d474e5aa58b32a51f7e4e3ec91e915c6657cfd1 Author: ccameron <ccameron@chromium.org> Date: Tue Apr 12 03:40:08 2016 Mac h264: Remove CFRetainCount assertion The CVPixelBuffers have an arbitrarily long lifetime now (they are to be handed to an AVSampleBufferDisplayLayer). It will not guaranteed that they are will be released by this call. BUG= 600685 Review URL: https://codereview.chromium.org/1881783002 Cr-Commit-Position: refs/heads/master@{#386572} [modify] https://crrev.com/2d474e5aa58b32a51f7e4e3ec91e915c6657cfd1/content/common/gpu/media/vt_video_decode_accelerator_mac.cc
,
Apr 12 2016
Looks like the flakiness is gone, but let's wait a little longer to be sure.
,
Apr 20 2016
Fixed?
,
Apr 20 2016
Looks fixed to me.
,
Apr 20 2016
Yes! :) Long history of green bots verifies it. |
|||||||||||||
►
Sign in to add a comment |
|||||||||||||
Comment 1 by grunell@chromium.org
, Apr 5 2016