Content Shell is intermittently crashing during try bot runs |
||||||
Issue descriptionOS: Android See flakiness dashboard for ImeTest: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=content_shell_test_apk%20%28with%20patch%29&builder=tryserver.chromium.android%3Alinux_android_rel_ng Not sure how to reproduce locally. Note: many of the failures seem to link to the wrong build log. Attaching a symbolized version of the crash log. Stack trace excerpt: Stack Trace: RELADDR FUNCTION FILE:LINE 00555c11 logging::LogMessage::~LogMessage() /b/c/b/linux_android_rel_ng/src/base/logging.cc:581 025e8f3b WTF::ThreadCheckingCallbackWrapper<base::RepeatingCallback<void (blink::WebLayerTreeView::SwapResult, double)>, void (blink::WebLayerTreeView::SwapResult, double)>::~ThreadCheckingCallbackWrapper() /b/c/b/linux_android_rel_ng/src/third_party/WebKit/Source/platform/wtf/Functional.h:228 ... cc::LayerTreeImpl::ClearSwapPromises() /b/c/b/linux_android_rel_ng/src/cc/trees/layer_tree_impl.cc:1573 00ab1c87 cc::LayerTreeHostImpl::DrawLayers(cc::LayerTreeHostImpl::FrameData*) /b/c/b/linux_android_rel_ng/src/cc/trees/layer_tree_host_impl.cc:1858 00ad3f5d cc::ProxyImpl::DrawInternal(bool) /b/c/b/linux_android_rel_ng/src/cc/trees/proxy_impl.cc:683 00ad3dd9 cc::ProxyImpl::ScheduledActionDrawIfPossible() /b/c/b/linux_android_rel_ng/src/cc/trees/proxy_impl.cc:546 00a7726d cc::Scheduler::DrawIfPossible() /b/c/b/linux_android_rel_ng/src/cc/scheduler/scheduler.cc:590 00a75155 cc::Scheduler::ProcessScheduledActions() /b/c/b/linux_android_rel_ng/src/cc/scheduler/scheduler.cc:688 00a74eaf cc::Scheduler::OnBeginImplFrameDeadline() /b/c/b/linux_android_rel_ng/src/cc/scheduler/scheduler.cc:580
,
Nov 17 2017
This is https://chromium-review.googlesource.com/765749. One of its linux_android_rel_ng tryjobs failed two tests on the DCHECK it added :/
,
Nov 17 2017
Bumping to P1 because this is causing *a lot* of failures in content_shell_test_apk on the trybots. Reverting would be preferable.
,
Nov 17 2017
Seen on Windows as well: https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/46141 WebglConformance_conformance_glsl_misc_shaders_with_uniform_structs and WebglConformance_conformance_more_functions_bindBufferBadArgs failed with crashes: https://chromium-swarm.appspot.com/task?id=39e00292c98fc810&refresh=10&show_raw=1 Stack trace below. I'm reverting the patch in https://chromium-review.googlesource.com/c/chromium/src/+/775542 . In the future, there should be no hesitation to revert patches that introduce flaky failures. The Chromium project's testing philosophy is that the tests must be green, and to revert rather than attempt to patch things live on the waterfall. Stack Trace: ******************************************************************************** Last event: 550.eec: Break instruction exception - code 80000003 (first/second chance not available) debugger time: Thu Nov 16 15:41:49.631 2017 (UTC - 8:00) ChildEBP RetAddr Args to Child 04bae00c 67071f65 68dc9e56 000000e4 05f8d2dc chrome_child!base::debug::BreakDebugger+0xc 04bae02c 6613bcfe 010b8378 68dc9e56 000000e4 chrome_child!?Run@?$Invoker@U?$BindState@P6AXPBDHV?$BasicStringPiece@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@base@@1@Z$$V@internal@base@@$$A6AXPBDHV?$BasicStringPiece@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@3@1@Z@internal@base@@SAXPAVBindStateBase@23@PBDH$$QAV?$BasicStringPiece@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@3@2@Z+0x1f 04bae4b8 689aef2f 00000003 01fe8200 68c0e174 chrome_child!logging::LogMessage::~LogMessage+0x40e 04bae584 689aeecf 048a54b0 04bae59c 689a30b5 chrome_child!WTF::ThreadCheckingCallbackWrapper<base::RepeatingCallback<v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>,v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>::~ThreadCheckingCallbackWrapper<base::RepeatingCallback<v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>,v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>+0x53 04bae590 689a30b5 048a54b0 04bae5ac 66137a89 chrome_child!std::unique_ptr<WTF::ThreadCheckingCallbackWrapper<base::RepeatingCallback<v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>,v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>,std::default_delete<WTF::ThreadCheckingCallbackWrapper<base::RepeatingCallback<v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>,v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)> > >::~unique_ptr<WTF::ThreadCheckingCallbackWrapper<base::RepeatingCallback<v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>,v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>,std::default_delete<WTF::ThreadCheckingCallbackWrapper<base::RepeatingCallback<v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>,v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)> > >+0x11 04bae59c 66137a89 048a54b0 05ef81b0 04bae5b8 chrome_child!base::internal::BindState<v8::MaybeLocal<v8::Script> (__thiscall WTF::ThreadCheckingCallbackWrapper<base::RepeatingCallback<v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>,v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>::*)(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *),std::unique_ptr<WTF::ThreadCheckingCallbackWrapper<base::RepeatingCallback<v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>,v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>,std::default_delete<WTF::ThreadCheckingCallbackWrapper<base::RepeatingCallback<v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)>,v8::MaybeLocal<v8::Script> __cdecl(v8::Isolate *,v8::Local<v8::String>,v8::ScriptOrigin,blink::InspectorCompileScriptEvent::V8CacheResult *)> > > >::Destroy+0x13 04bae5ac 6786bc28 047980b0 04bae5d0 66d35e7c chrome_child!base::internal::CallbackBase::~CallbackBase+0x19 04bae5b8 66d35e7c 00000001 04bae5d8 0482f618 chrome_child!base::internal::BindState<void (__thiscall blink::WebCompositeAndReadbackAsyncCallback::*)(SkBitmap const &),base::internal::UnretainedWrapper<blink::WebCompositeAndReadbackAsyncCallback>,SkBitmap>::Destroy+0x40 04bae5d0 66d40cdb 04bae5d8 0482f618 04850358 chrome_child!cc::LayerTreeImpl::ClearSwapPromises+0x4c 04bae894 66ddba7f 04bae8b8 00bae830 00000000 chrome_child!cc::LayerTreeHostImpl::DrawLayers+0xa0b 04bae9dc 66ddb860 00000000 04baea10 00000000 chrome_child!cc::ProxyImpl::DrawInternal+0x12f 04baead0 66de5a57 020a5da8 020a5c68 00000001 chrome_child!cc::ProxyImpl::ScheduledActionDrawIfPossible+0xc0 04baeaf0 66de2cb5 020a5da8 02083374 00000005 chrome_child!cc::Scheduler::DrawIfPossible+0x47 04baeb50 66de29e1 00000eec 00000000 a9bc2ed1 chrome_child!cc::Scheduler::ProcessScheduledActions+0x1d5 04baec40 67774a05 04778640 66f92f76 04778640 chrome_child!cc::Scheduler::OnBeginImplFrameDeadline+0xb1 04baed0c 652c3c78 04830740 0476fc00 04baee40 chrome_child!base::internal::Invoker<base::internal::BindState<void (__thiscall viz::DelayBasedTimeSource::*)(void),base::WeakPtr<viz::DelayBasedTimeSource> >,void __cdecl(void)>::Run+0x85 04baed1c 661ba755 04958768 a9bc2ed1 04baed4c chrome_child!??$ForwardRepeating@$$V@?$CancelableCallbackImpl@V?$RepeatingCallback@$$A6AXXZ@base@@@internal@base@@AAEXXZ+0x10 04baee40 66004ff9 68dd83b1 04baef08 6600f1e6 chrome_child!base::debug::TaskAnnotator::RunTask+0xe5 04baefd4 66003a16 020a0af0 00000000 02092f64 chrome_child!blink::scheduler::TaskQueueManager::ProcessTaskFromWorkQueue+0x409 04baf140 66006a4f 00000000 66003760 0200ece4 chrome_child!blink::scheduler::TaskQueueManager::DoWork+0x2b6 04baf15c 661ba755 02012c78 02092e70 04baf240 chrome_child!base::internal::Invoker<base::internal::BindState<void (__thiscall gpu::GpuWatchdogThread::*)(bool),base::WeakPtr<gpu::GpuWatchdogThread>,bool>,void __cdecl(void)>::Run+0x3d 04baf280 661d70f7 68e0c4db 04baf558 04baf2a8 chrome_child!base::debug::TaskAnnotator::RunTask+0xe5 04baf354 6616f0a9 04baf558 68dd8038 020963f8 chrome_child!base::internal::IncomingTaskQueue::RunTask+0x67 04baf474 6616f3d4 04baf558 02092e74 00000eec chrome_child!base::MessageLoop::RunTask+0x1c9 04baf550 6616f608 00000000 68dd8186 68dd8038 chrome_child!base::MessageLoop::DeferOrRunPendingTask+0xa4 04baf6b8 661da851 0209cd08 0209cd00 020963fc chrome_child!base::MessageLoop::DoWork+0x1f8 04baf6d4 6616ea97 020963f8 00000100 020963f8 chrome_child!base::MessagePumpDefault::Run+0xb1 04baf7ac 66165229 00000001 04baf968 04775d94 chrome_child!base::MessageLoop::Run+0xa7 04baf880 6617ad62 00000eec 00000eec 00000eec chrome_child!base::RunLoop::Run+0xc9 04baf94c 6617aff5 04baf968 00000000 00000000 chrome_child!base::Thread::Run+0xa2 04bafa2c 66173783 04775d88 000002b4 000002b4 chrome_child!base::Thread::ThreadMain+0x275 *** WARNING: Unable to verify checksum for kernel32.dll *** ERROR: Symbol file could not be found. Defaulted to export symbols for kernel32.dll - 04bafa50 7620336a 047b2580 04bafa9c 76ff92b2 chrome_child!base::PlatformThread::SetCurrentThreadPriority+0x203 WARNING: Stack unwind information not available. Following frames may be wrong. 04bafa5c 76ff92b2 047b2580 73d79d98 00000000 kernel32!BaseThreadInitThunk+0x12 04bafa9c 76ff9285 661736e0 047b2580 ffffffff ntdll!RtlInitializeExceptionChain+0x63 04bafab4 00000000 661736e0 047b2580 00000000 ntdll!RtlInitializeExceptionChain+0x36
,
Nov 17 2017
,
Nov 17 2017
There seem several users that hits the tighten DCHECK introduced by the new impl of WTF::Bind. E.g.: - FirstMeaningfulPaintDetector::RegisterNotifySwapTime - PaintTiming::RegisterNotifySwapTime
,
Nov 17 2017
Note that the revert landed in https://chromium.googlesource.com/chromium/src.git/+/a406b63494f748990356adb0ccbec1ebd42615c2 .
,
Nov 17 2017
The crash in comment 4 appears to be due to static function SelectCompileFunction in V8ScriptRunner.cpp called by V8ScriptRunner::CompileScript
,
Nov 17 2017
Hm, crash in #4 also has ClearSwapPromises in the stack, so the cause may be the same as #0's.
,
Nov 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c35af225d5bb6fefe50fdbdae6dd1bc8a9c0a3a9 commit c35af225d5bb6fefe50fdbdae6dd1bc8a9c0a3a9 Author: Yuta Kitamura <yutak@chromium.org> Date: Fri Nov 17 18:10:47 2017 Closures for swap promises must be cross-thread. Upon landing of <http://crrev.com/c/765749>, which tightened up the threading check of WTF::Function, a few assertion failures were reported. The stack traces indicated closures for swap promises (in PaintTiming.cpp and FirstMeaningfulPaintDetector.cpp) might end up getting destructed on the compositor thread. To allow this, WTF::CrossThreadBind() must be used instead of WTF::Bind(). Bug: 786194 Cq-Include-Trybots: master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2 Change-Id: I9252eff15ca00e1bc0dfc8f87629ccb9acb56714 Reviewed-on: https://chromium-review.googlesource.com/776343 Reviewed-by: Taiju Tsuiki <tzik@chromium.org> Reviewed-by: Kentaro Hara <haraken@chromium.org> Reviewed-by: enne <enne@chromium.org> Commit-Queue: enne <enne@chromium.org> Cr-Commit-Position: refs/heads/master@{#517471} [modify] https://crrev.com/c35af225d5bb6fefe50fdbdae6dd1bc8a9c0a3a9/third_party/WebKit/Source/core/paint/FirstMeaningfulPaintDetector.cpp [modify] https://crrev.com/c35af225d5bb6fefe50fdbdae6dd1bc8a9c0a3a9/third_party/WebKit/Source/core/paint/PaintTiming.cpp [modify] https://crrev.com/c35af225d5bb6fefe50fdbdae6dd1bc8a9c0a3a9/third_party/WebKit/Source/core/paint/PaintTiming.h
,
Nov 20 2017
This must be fixed by now. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by jbudorick@chromium.org
, Nov 17 2017