WebRtcCaptureFromElementBrowserTest.VerifyCanvasCaptureOffscreenCanvasCommitFrames is flaking on Linux ChromiumOS Tests (dbg) |
|||||
Issue descriptionChrome Version: 60.0.3101.0 The WebRtcCaptureFromElementBrowserTest.VerifyCanvasCaptureOffscreenCanvasCommitFrames test seems to be flaking (timing out) more than half the time on the "Linux ChromiumOS Tests (dbg)" builder (it seems fine on the release builder): https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Tests%20%28dbg%29%281%29?numbuilds=100 More often than not, the builder passes on retry, but has gone red twice today because of this. Here's one: https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Tests%20%28dbg%29%281%29/builds/26147 As far as I could find, the first instance of this flake was in https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Tests%20%28dbg%29%281%29/builds/26139 which was earlier today (it has appeared in about 2/3 of the builds since then). I don't know this part of the code well enough to locate an likely culprit CL, but it's probably been in the last 24 hours or so. Additionally, while I couldn't reproduce this locally with yesterday's code, I could with today's code. Here's the log output from content_browsertests / stdout : [0517/102517.115801:ERROR:kill_posix.cc(84)] Unable to terminate process group 30244: No such process [ RUN ] WebRtcCaptureFromElementBrowserTest.VerifyCanvasCaptureOffscreenCanvasCommitFrames Xlib: extension "RANDR" missing on display ":99". [30244:30244:0517/102433.495007:10934818611:ERROR:proxy_service.cc(1534)] ProxyConfigService for ChromeOS should be created in profile_io_data.cc::CreateProxyConfigService and this should be used only for examples. BrowserTestBase received signal: Terminated. Backtrace: #0 0x7efff70721bb base::debug::StackTrace::StackTrace() #1 0x7efff7070efc base::debug::StackTrace::StackTrace() #2 0x0000013d3237 content::(anonymous namespace)::DumpStackTraceSignalHandler() #3 0x7effeabfdcb0 <unknown> #4 0x7effeacb784d __poll #5 0x7effe93f9fe4 <unknown> #6 0x7effe93fa0ec g_main_context_iteration #7 0x7efff71132af base::MessagePumpGlib::Run() #8 0x7efff710184d base::MessageLoop::RunHandler() #9 0x7efff719fd44 base::RunLoop::Run() #10 0x00000142c896 content::RunThisRunLoop() #11 0x00000142d4f1 content::MessageLoopRunner::Run() #12 0x0000013daa3a content::DOMMessageQueue::WaitForMessage() #13 0x0000013d8c90 content::(anonymous namespace)::ExecuteScriptHelper() #14 0x0000013d8a70 content::ExecuteScriptAndExtractString() #15 0x000000af9105 content::WebRtcContentBrowserTestBase::ExecuteJavascriptAndReturnResult() #16 0x000000af93d4 content::WebRtcContentBrowserTestBase::ExecuteJavascriptAndWaitForOk() #17 0x000000af9391 content::WebRtcContentBrowserTestBase::MakeTypicalCall() #18 0x000000aec2c2 content::WebRtcCaptureFromElementBrowserTest_VerifyCanvasCaptureOffscreenCanvasCommitFrames_Test::RunTestOnMainThread() #19 0x0000013d2f27 content::BrowserTestBase::ProxyRunTestOnMainThreadLoop() #20 0x000000621b95 _ZN4base8internal13FunctorTraitsIMN7content21CrossSiteTransferTestEFvvEvE6InvokeIPS3_JEEEvS5_OT_DpOT0_ #21 0x000000621ab1 _ZN4base8internal12InvokeHelperILb0EvE8MakeItSoIRKMN7content21CrossSiteTransferTestEFvvEJPS5_EEEvOT_DpOT0_ #22 0x0000013d4b07 _ZN4base8internal7InvokerINS0_9BindStateIMN7content15BrowserTestBaseEFvvEJNS0_17UnretainedWrapperIS4_EEEEEFvvEE7RunImplIRKS6_RKSt5tupleIJS8_EEJLm0EEEEvOT_OT0_NS_13IndexSequenceIJXspT1_EEEE #23 0x0000013d4a4c _ZN4base8internal7InvokerINS0_9BindStateIMN7content15BrowserTestBaseEFvvEJNS0_17UnretainedWrapperIS4_EEEEEFvvEE3RunEPNS0_13BindStateBaseE #24 0x000000603add _ZNKR4base8CallbackIFvvELNS_8internal8CopyModeE1ELNS2_10RepeatModeE1EE3RunEv #25 0x00000152d770 content::ShellBrowserMainParts::PreMainMessageLoopRun() #26 0x7efff8d6ecd1 content::BrowserMainLoop::PreMainMessageLoopRun() #27 0x7efff80529a5 _ZN4base8internal13FunctorTraitsIMN7content13URLLoaderImplEFvvEvE6InvokeIPS3_JEEEvS5_OT_DpOT0_ #28 0x7efff8d77e81 _ZN4base8internal12InvokeHelperILb0EiE8MakeItSoIRKMN7content15BrowserMainLoopEFivEJPS5_EEEiOT_DpOT0_ #29 0x7efff8d77e27 _ZN4base8internal7InvokerINS0_9BindStateIMN7content15BrowserMainLoopEFivEJNS0_17UnretainedWrapperIS4_EEEEEFivEE7RunImplIRKS6_RKSt5tupleIJS8_EEJLm0EEEEiOT_OT0_NS_13IndexSequenceIJXspT1_EEEE #30 0x7efff8d77d6c _ZN4base8internal7InvokerINS0_9BindStateIMN7content15BrowserMainLoopEFivEJNS0_17UnretainedWrapperIS4_EEEEEFivEE3RunEPNS0_13BindStateBaseE #31 0x7efff80268cd _ZNKR4base8CallbackIFvvELNS_8internal8CopyModeE1ELNS2_10RepeatModeE1EE3RunEv #32 0x7efff98e784b content::StartupTaskRunner::RunAllTasksNow() #33 0x7efff8d6c8f0 content::BrowserMainLoop::CreateStartupTasks() #34 0x7efff8d7b0d7 content::BrowserMainRunnerImpl::Initialize() #35 0x0000014de18b ShellBrowserMain() #36 0x0000014c41d5 content::ShellMainDelegate::RunProcess() #37 0x7efffa5ce9cd content::RunNamedProcessTypeMain() #38 0x7efffa5d0aef content::ContentMainRunnerImpl::Run() #39 0x7efffa5cd69a content::ContentServiceManagerMainDelegate::RunEmbedderProcess() #40 0x7effe7675ef7 service_manager::Main() #41 0x7efffa5ce52b content::ContentMain() #42 0x0000013d2c4f content::BrowserTestBase::SetUp() #43 0x0000013c02d8 content::ContentBrowserTest::SetUp() #44 0x000000af9020 content::WebRtcContentBrowserTestBase::SetUp() #45 0x000000d6ad0e testing::internal::HandleSehExceptionsInMethodIfSupported<>() #46 0x000000d5ccb2 testing::internal::HandleExceptionsInMethodIfSupported<>() #47 0x000000d51b84 testing::Test::Run() #48 0x000000d5230d testing::TestInfo::Run() #49 0x000000d528af testing::TestCase::Run() #50 0x000000d57c5c testing::internal::UnitTestImpl::RunAllTests() #51 0x000000d6e88e testing::internal::HandleSehExceptionsInMethodIfSupported<>() #52 0x000000d5e452 testing::internal::HandleExceptionsInMethodIfSupported<>() #53 0x000000d578ff testing::UnitTest::Run() #54 0x00000147c6f1 RUN_ALL_TESTS() #55 0x00000147b472 base::TestSuite::Run() #56 0x0000013cbb62 content::ContentTestLauncherDelegate::RunTestSuite() #57 0x0000014237b2 content::LaunchTests() #58 0x0000013cb9b8 main #59 0x7effeabe8f45 __libc_start_main #60 0x00000059af64 <unknown> [806/983] WebRtcCaptureFromElementBrowserTest.VerifyCanvasCaptureOffscreenCanvasCommitFrames (TIMED OUT) emircan@ - could you take a look at this or triage it to the appropriate person? Thanks!
,
May 18 2017
https://codereview.chromium.org/2810423003/ causes the compositor task runner to run with a higher priority, which means that timer tasks are run less frequently. This caused several of the WebRtcMediaRecorderTest.RecordWithTransparency tests to fail when run with tsan since that makes the test run 10-15 times slower, which in combination with a bug in the test and my change made the tests timeout. The fix was https://codereview.chromium.org/2868933003/ This sort of fits with the debug/release mismatch here, in that the timeouts may be more likely to trigger on a debug build. I'm not sure how to test directly on chromiumos, but I'll create a debug linux build and extract traces from the test to see if there's anything significant there.
,
May 18 2017
I can repro the failure with a debug build on Linux. From 10 runs, it failed 7 times. It looks like the failure is not a timeout due to the test not having time to complete but an actual hang. I say this because when it passes, it takes less than 5 seconds but takes 45 seconds when it fails. If I comment out https://cs.chromium.org/chromium/src/third_party/WebKit/Source/platform/scheduler/renderer/renderer_scheduler_impl.cc?sq=package:chromium&dr=CSs&l=1092 then the test passes 100%, so it's definitely my change that caused this. Test is https://cs.chromium.org/chromium/src/content/test/data/media/canvas_capture.html?l=54&rcl=ca8295d2e5835178425dcdbe99d9868198aaeb2a with drawFunction = drawOffscreenCanvasCommit. My guess is that the requestAnimation callback is getting called more frequently than the recorder.ondataavailable callback and then when it's run NUMBER_OF_EVENTS_TO_RECORD times, it stops as expected. What could be happening is that ondataavailable is now called less often (perhaps it coalesces calls), which means there aren't NUMBER_OF_EVENTS_TO_RECORD events that meet the threshold and so the test never finishes.
,
May 18 2017
Per flakiness dashboard, the test is also flaky on Windows (debug only, release seems fine): https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=content_browsertests&tests=VerifyCanvasCaptureOffscreenCanvasCommitFrames
,
May 18 2017
I'm going to rollback the compositor prioritisation part of this change now.
,
May 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/fa4bddcb8a6f72ff36d86c84dd8b9f2805bc147d commit fa4bddcb8a6f72ff36d86c84dd8b9f2805bc147d Author: Dan Elphick <delphick@chromium.org> Date: Thu May 18 21:25:03 2017 Revert high priority of compositor in NONE UseCase Fixes WebRtc tests timing out when run on debug builds. Bug: 723805 Change-Id: Ie4de21a6160b2bf649c2e2ee4c8f78fdea585dac Reviewed-on: https://chromium-review.googlesource.com/508792 Reviewed-by: Sami Kyöstilä <skyostil@chromium.org> Commit-Queue: Sami Kyöstilä <skyostil@chromium.org> Cr-Commit-Position: refs/heads/master@{#472935} [modify] https://crrev.com/fa4bddcb8a6f72ff36d86c84dd8b9f2805bc147d/third_party/WebKit/Source/platform/scheduler/renderer/renderer_scheduler_impl.cc [modify] https://crrev.com/fa4bddcb8a6f72ff36d86c84dd8b9f2805bc147d/third_party/WebKit/Source/platform/scheduler/renderer/renderer_scheduler_impl_unittest.cc
,
May 18 2017
,
May 22 2017
,
Jul 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e6347ad5f6a3675f095c277f5cb5794fddedd154 commit e6347ad5f6a3675f095c277f5cb5794fddedd154 Author: Dan Elphick <delphick@chromium.org> Date: Tue Jul 04 17:39:59 2017 Move OffscreenCanvas tasks onto compositor TQ By moving OffscreenCanvas tasks to the compositor TQ, the priority of that TQ can be raised without changing the ordering of OffscreenCanvas tasks and BeginMainFrame tasks. In the short-term this likely has limited effect although it could prevent flakiness when using OffscreenCanvases during scrolling. In future, it will allow the compositor TQ to have a higher priority without making WebRtcCaptureFromElementBrowserTest.VerifyCanvasCaptureOffscreenCanvasCommitFrames flaky. Bug: 723805 Change-Id: I61277fd6c23f2c06c41f86e8b76118811417c150 Reviewed-on: https://chromium-review.googlesource.com/538583 Reviewed-by: Justin Novosad <junov@chromium.org> Commit-Queue: Dan Elphick <delphick@chromium.org> Cr-Commit-Position: refs/heads/master@{#484137} [modify] https://crrev.com/e6347ad5f6a3675f095c277f5cb5794fddedd154/third_party/WebKit/Source/platform/graphics/OffscreenCanvasFrameDispatcherImpl.cpp [modify] https://crrev.com/e6347ad5f6a3675f095c277f5cb5794fddedd154/third_party/WebKit/Source/platform/scheduler/child/web_scheduler_impl.cc |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by emir...@chromium.org
, May 17 2017Owner: delph...@chromium.org