Flakiness in integration tests on macOS causes INVALID_TEST_RESULTS |
||||||
Issue descriptionCL: https://chromium-review.googlesource.com/c/chromium/src/+/1374965/4 Note that the first build fails, the second succeeds. Build with failures: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/207925 Example test suites that failed [ info_collection_tests ]: https://chromium-swarm.appspot.com/task?id=41c2902bf2c3ad10&refresh=10&show_raw=1 Example test suites that failed [ screenshot_sync_tests ]: https://chromium-swarm.appspot.com/task?id=41c28fff888d8510&refresh=10&show_raw=1 Both had the same failure: """ 0 Chromium Framework 0x0000000117a17d0f base::debug::StackTrace::StackTrace(unsigned long) + 31 1 Chromium Framework 0x0000000117a17b61 base::debug::(anonymous namespace)::StackDumpSignalHandler(int, __siginfo*, void*) + 2385 2 libsystem_platform.dylib 0x00007fff66ce1f5a _sigtramp + 26 3 ??? 0x0000000000000040 0x0 + 64 4 Chromium Framework 0x000000011cd76e87 blink::Page::RequestBeginMainFrameNotExpected(bool) + 343 5 Chromium Framework 0x0000000116d96211 blink::scheduler::MainThreadSchedulerImpl::DispatchRequestBeginMainFrameNotExpected(bool) + 289 6 Chromium Framework 0x0000000116d9d3ec base::internal::Invoker<base::internal::BindState<void (blink::scheduler::MainThreadSchedulerImpl::*)(bool), base::WeakPtr<blink::scheduler::MainThreadSchedulerImpl>, bool>, void ()>::RunOnce(base::internal::BindStateBase*) + 188 7 Chromium Framework 0x00000001178fdf51 base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) + 305 8 Chromium Framework 0x00000001179a9799 base::sequence_manager::internal::ThreadControllerImpl::DoWork(base::sequence_manager::internal::ThreadControllerImpl::WorkType) + 889 9 Chromium Framework 0x00000001179ab57b base::internal::Invoker<base::internal::BindState<void (base::sequence_manager::internal::ThreadControllerImpl::*)(base::sequence_manager::internal::ThreadControllerImpl::WorkType), base::WeakPtr<base::sequence_manager::internal::ThreadControllerImpl>, base::sequence_manager::internal::ThreadControllerImpl::WorkType>, void ()>::Run(base::internal::BindStateBase*) + 187 10 Chromium Framework 0x00000001178fdf51 base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) + 305 11 Chromium Framework 0x000000011792f2e0 base::MessageLoopImpl::RunTask(base::PendingTask*) + 272 12 Chromium Framework 0x000000011792f983 base::MessageLoopImpl::DoWork() + 387 13 Chromium Framework 0x0000000117931c63 base::MessagePumpCFRunLoopBase::RunWork() + 51 14 Chromium Framework 0x000000011791774a base::mac::CallWithEHFrame(void () block_pointer) + 10 15 Chromium Framework 0x000000011793143f base::MessagePumpCFRunLoopBase::RunWorkSource(void*) + 63 16 CoreFoundation 0x00007fff3ebb4a11 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17 17 CoreFoundation 0x00007fff3ec6e42c __CFRunLoopDoSource0 + 108 18 CoreFoundation 0x00007fff3eb97470 __CFRunLoopDoSources0 + 208 19 CoreFoundation 0x00007fff3eb968ed __CFRunLoopRun + 1293 20 CoreFoundation 0x00007fff3eb96153 CFRunLoopRunSpecific + 483 21 Foundation 0x00007fff40c92f26 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 277 22 Chromium Framework 0x00000001179324d1 base::MessagePumpNSRunLoop::DoRun(base::MessagePump::Delegate*) + 113 23 Chromium Framework 0x0000000117930daf base::MessagePumpCFRunLoopBase::Run(base::MessagePump::Delegate*) + 127 24 Chromium Framework 0x000000011792ee88 base::MessageLoopImpl::Run(bool) + 136 25 Chromium Framework 0x000000011796a229 base::RunLoop::Run() + 249 26 Chromium Framework 0x000000011de01367 content::RendererMain(content::MainFunctionParams const&) + 1399 27 Chromium Framework 0x00000001172e4393 content::ContentMainRunnerImpl::Run(bool) + 435 28 Chromium Framework 0x000000011a8d6ae5 service_manager::Main(service_manager::MainParams const&) + 3173 29 Chromium Framework 0x00000001172e34a4 content::ContentMain(content::ContentMainParams const&) + 68 30 Chromium Framework 0x000000011325c01f ChromeMain + 175 31 Chromium Helper 0x000000010b54972e main + 494 32 libdyld.dylib 0x00007fff669d3015 start + 1 [end of stack trace] """
,
Dec 14
The minidump symbolization produced one additional stack frame: Operating system: Mac OS X 10.13.6 17G65 CPU: amd64 family 6 model 70 stepping 1 8 CPUs GPU: UNKNOWN Crash reason: EXC_BAD_ACCESS / KERN_INVALID_ADDRESS Crash address: 0xfffffffffffffff8 Process uptime: 1 seconds Thread 0 (crashed) 0 Chromium Framework!__ZN7content13LayerTreeView13RequestDecodeERKN2cc10PaintImageEN4base12OnceCallbackIFvbEEE + 0x17 rax = 0x00007fd0ede2d860 rdx = 0xfffffffffffffff8 rcx = 0x000000011eac7cc8 rbx = 0x00000034970418c8 rsi = 0x0000000000000001 rdi = 0x00007fd0f1009200 rbp = 0x00007ffee1d73910 rsp = 0x00007ffee1d73790 r8 = 0x00007fd0f1004928 r9 = 0xffffffff00000000 r10 = 0x00007fd0f1004920 r11 = 0x00007fca4ad31830 r12 = 0x00007fd0ede23f40 r13 = 0x00007fd0f1005578 r14 = 0x0000000000000001 r15 = 0x00007fd0ede2d860 rip = 0x000000011cea88e7 Found by: given as instruction pointer in context 1 Chromium Framework!__ZN5blink4Page32RequestBeginMainFrameNotExpectedEb + 0x157 rbp = 0x00007ffee1d74a20 rsp = 0x00007ffee1d73920 rip = 0x000000011bd57e87 Found by: previous frame's frame pointer 2 Chromium Framework!__ZN5blink9scheduler23MainThreadSchedulerImpl40DispatchRequestBeginMainFrameNotExpectedEb + 0x121 rbp = 0x00007ffee1d74bb0 rsp = 0x00007ffee1d74a30 rip = 0x0000000115d77211 Found by: previous frame's frame pointer 3 Chromium Framework!__ZN4base8internal7InvokerINS0_9BindStateIMN5blink9scheduler23MainThreadSchedulerImplEFvbEJNS_7WeakPtrIS5_EEbEEEFvvEE7RunOnceEPNS0_13BindStateBaseE + 0xbc rbp = 0x00007ffee1d74d10 rsp = 0x00007ffee1d74bc0 rip = 0x0000000115d7e3ec Found by: previous frame's frame pointer 4 Chromium Framework!__ZN4base5debug13TaskAnnotator7RunTaskEPKcPNS_11PendingTaskE + 0x131 rbp = 0x00007ffee1d74ed0 rsp = 0x00007ffee1d74d20 rip = 0x00000001168def51 Found by: previous frame's frame pointer 5 Chromium Framework!__ZN4base16sequence_manager8internal20ThreadControllerImpl6DoWorkENS2_8WorkTypeE + 0x379 rbp = 0x00007ffee1d751a0 rsp = 0x00007ffee1d74ee0 rip = 0x000000011698a799 Found by: previous frame's frame pointer 6 Chromium Framework!__ZN4base8internal7InvokerINS0_9BindStateIMNS_16sequence_manager8internal20ThreadControllerImplEFvNS5_8WorkTypeEEJNS_7WeakPtrIS5_EES6_EEEFvvEE3RunEPNS0_13BindStateBaseE + 0xbb rbp = 0x00007ffee1d75300 rsp = 0x00007ffee1d751b0 rip = 0x000000011698c57b Found by: previous frame's frame pointer 7 Chromium Framework!__ZN4base5debug13TaskAnnotator7RunTaskEPKcPNS_11PendingTaskE + 0x131 rbp = 0x00007ffee1d754c0 rsp = 0x00007ffee1d75310 rip = 0x00000001168def51 Found by: previous frame's frame pointer ... From this shard: https://chromium-swarm.appspot.com/task?id=41c2902bf2c3ad10&refresh=10&show_raw=1 There's nothing in the compile log which would indicate that something was wrong, but something was clearly broken either in the build, or during the upload of the isolate.
,
Jan 11
Khushal, does the top stack trace in comment #2 ring any bells? __ZN7content13LayerTreeView13RequestDecodeERKN2cc10PaintImageEN4base12OnceCallbackIFvbEEE This sounds like maybe the prepaint stuff, or it could be a corrupt stack?
,
Jan 11
,
Jan 14
The stack at the top looks corrupt to me, because I don't see how it could go from Page::RequestBeginMainFrameNotExpected to LayerTreeView::RequestDecode. But the crash looks related to issue 838348 (though that one was reported on win and mac only). dcheng@, mind taking a look and checking if this could be related to issue 838348.
,
Jan 15
I filed issue 922069, which has similar symptoms. Not sure if it's the same problem.
,
Jan 15
I have a clusterfuzz repro for a problem in the same area. We've figured out the root cause and a fix, but it will take some time to finish the patch. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by erikc...@chromium.org
, Dec 14