For example, https://ci.chromium.org/buildbot/chromium.gpu.fyi/Mac%20FYI%20Experimental%20Release%20%28Intel%29/623.
Looks like there are a lot more crashes than expected during context_lost_tests, even in the passing runs.
I'm also not sure if the crash is the result of the test.
0 Chromium Framework 0x000000010dcd5c9c base::debug::StackTrace::StackTrace(unsigned long) + 28
1 Chromium Framework 0x000000010dcd5ad1 base::debug::(anonymous namespace)::StackDumpSignalHandler(int, __siginfo*, void*) + 2401
2 libsystem_platform.dylib 0x00007fff6a132f5a _sigtramp + 26
3 Chromium Framework 0x000000010e4ddef5 IPC::ChannelProxy::Context::TryFilters(IPC::Message const&) + 117
4 Chromium Framework 0x000000010b5c14b0 viz::mojom::GpuServiceStubDispatch::Accept(viz::mojom::GpuService*, mojo::Message*) + 2576
5 Chromium Framework 0x000000010e5e0dc6 mojo::InterfaceEndpointClient::HandleValidatedMessage(mojo::Message*) + 950
6 Chromium Framework 0x000000010e5e0696 mojo::FilterChain::Accept(mojo::Message*) + 150
7 Chromium Framework 0x000000010e5e21f5 mojo::InterfaceEndpointClient::HandleIncomingMessage(mojo::Message*) + 117
8 Chromium Framework 0x000000010e5e88dc mojo::internal::MultiplexRouter::ProcessIncomingMessage(mojo::internal::MultiplexRouter::MessageWrapper*, mojo::internal::MultiplexRouter::ClientCallBehavior, base::SequencedTaskRunner*) + 1420
9 Chromium Framework 0x000000010e5e7ce3 mojo::internal::MultiplexRouter::Accept(mojo::Message*) + 371
10 Chromium Framework 0x000000010e5e0696 mojo::FilterChain::Accept(mojo::Message*) + 150
11 Chromium Framework 0x000000010e5db955 mojo::Connector::ReadSingleMessage(unsigned int*) + 421
12 Chromium Framework 0x000000010e5dc3a1 mojo::Connector::ReadAllAvailableMessages() + 97
13 Chromium Framework 0x000000010e5dc259 mojo::Connector::OnHandleReadyInternal(unsigned int) + 137
14 Chromium Framework 0x000000010b5ae5a7 mojo::SimpleWatcher::DiscardReadyState(base::RepeatingCallback<void (unsigned int)> const&, unsigned int, mojo::HandleSignalsState const&) + 103
15 Chromium Framework 0x000000010e4fa4bc mojo::SimpleWatcher::OnHandleReady(int, unsigned int, mojo::HandleSignalsState const&) + 332
16 Chromium Framework 0x000000010e4fa907 mojo::SimpleWatcher::Context::Notify(unsigned int, MojoHandleSignalsState, unsigned int) + 855
17 Chromium Framework 0x000000010e4f94c3 mojo::SimpleWatcher::Context::CallNotify(MojoTrapEvent const*) + 35
18 Chromium Framework 0x000000010f88e8a8 mojo::edk::WatcherDispatcher::InvokeWatchCallback(unsigned long, unsigned int, mojo::HandleSignalsState const&, unsigned int) + 120
19 Chromium Framework 0x000000010f88deb2 mojo::edk::Watch::InvokeCallback(unsigned int, mojo::HandleSignalsState const&, unsigned int) + 194
20 Chromium Framework 0x000000010f888eb6 mojo::edk::RequestContext::~RequestContext() + 1030
21 Chromium Framework 0x000000010f879008 mojo::edk::NodeChannel::OnChannelMessage(void const*, unsigned long, std::__1::vector<mojo::edk::ScopedPlatformHandle, std::__1::allocator<mojo::edk::ScopedPlatformHandle> >) + 344
22 Chromium Framework 0x000000010f867f18 mojo::edk::Channel::OnReadComplete(unsigned long, unsigned long*) + 600
23 Chromium Framework 0x000000010f894527 mojo::edk::(anonymous namespace)::ChannelPosix::OnFileCanReadWithoutBlocking(int) + 535
24 Chromium Framework 0x000000010dd18560 base::MessagePumpLibevent::OnLibeventNotification(int, short, void*) + 352
25 Chromium Framework 0x000000010ddf39bc event_base_loop + 1196
26 Chromium Framework 0x000000010dd188d3 base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) + 435
27 Chromium Framework 0x000000010dd155e9 base::MessageLoop::Run(bool) + 169
28 Chromium Framework 0x000000010dd53c49 base::RunLoop::Run() + 249
29 Chromium Framework 0x000000010dda09be base::Thread::Run(base::RunLoop*) + 206
30 Chromium Framework 0x000000010dda0f7c base::Thread::ThreadMain() + 908
31 Chromium Framework 0x000000010dd9bc5f base::(anonymous namespace)::ThreadFunc(void*) + 95
32 libsystem_pthread.dylib 0x00007fff6a13c6c1 _pthread_body + 340
33 libsystem_pthread.dylib 0x00007fff6a13c56d _pthread_body + 0
34 libsystem_pthread.dylib 0x00007fff6a13bc5d thread_start + 13
Comment 1 by kbr@chromium.org
, Mar 10 2018Cc: a...@chromium.org
Owner: lgrey@chromium.org
Status: Assigned (was: Untriaged)
The error message is: Traceback (most recent call last): _RunGpuTest at content/test/gpu/gpu_tests/gpu_integration_test.py:132 self.RunActualGpuTest(url, *args) RunActualGpuTest at content/test/gpu/gpu_tests/context_lost_integration_test.py:90 getattr(self, test_name)(test_path) _GpuCrash_GPUProcessCrashesExactlyOncePerVisitToAboutGpuCrash at content/test/gpu/gpu_tests/context_lost_integration_test.py:214 self._KillGPUProcess(2, True) _KillGPUProcess at content/test/gpu/gpu_tests/context_lost_integration_test.py:144 self._CheckCrashCount(tab, expected_kills) _CheckCrashCount at content/test/gpu/gpu_tests/context_lost_integration_test.py:190 (expected_kills, number_of_crashes)) fail at .swarming_module/lib/python2.7/unittest/case.py:410 raise self.failureException(msg) AssertionError: Expected 1 gpu process crashes; got: 5 We do in fact see the NativeStackSamplerMac crashing here, causing at least some extraneous crashes: Operating system: Mac OS X 10.13.4 17E139j CPU: amd64 family 6 model 69 stepping 1 4 CPUs GPU: UNKNOWN Crash reason: EXC_BAD_ACCESS / KERN_INVALID_ADDRESS Crash address: 0x7fff6a19b000 Process uptime: 0 seconds Thread 8 (crashed) 0 libunwind.dylib + 0x494c rax = 0x0000000000000001 rdx = 0x00007fff6a169a3c rcx = 0x00000000ffffffff rbx = 0x00007fff6a19b000 rsi = 0x00007000098d1ff0 rdi = 0x00007fffa21e2a99 rbp = 0x00007000098d2020 rsp = 0x00007000098d1fd0 r8 = 0x00007fff6a19aff0 r9 = 0x0000000000000000 r10 = 0x0000000000000008 r11 = 0x00007fff6a19abf9 r12 = 0x00007fff6a18f969 r13 = 0x00007fff6a19b000 r14 = 0x00007fffa21e2a99 r15 = 0x00007fff6a19b368 rip = 0x00007fff6a16c94c Found by: given as instruction pointer in context 1 libunwind.dylib + 0x4768 rbp = 0x00007000098d2880 rsp = 0x00007000098d2030 rip = 0x00007fff6a16c768 Found by: previous frame's frame pointer 2 libunwind.dylib + 0xbe2 rbp = 0x00007000098d30d0 rsp = 0x00007000098d2890 rip = 0x00007fff6a168be2 Found by: previous frame's frame pointer 3 libunwind.dylib + 0xb2b rbp = 0x00007000098d30f0 rsp = 0x00007000098d30e0 rip = 0x00007fff6a168b2b Found by: previous frame's frame pointer 4 Chromium Framework!__ZN4base12_GLOBAL__N_120WalkStackFromContextIZNS0_21NativeStackSamplerMac27SuspendThreadAndRecordStackEPNS_18NativeStackSampler11StackBufferEPNS_21StackSamplingProfiler6SampleEE3$_1EEbP13unw_context_tmPmPNSt3__16vectorINS6_6ModuleENSD_9allocatorISF_EEEEPNSE_INS0_11ModuleIndexENSG_ISK_EEEERKT_ + 0x3b rbp = 0x00007000098d3600 rsp = 0x00007000098d3100 rip = 0x000000011a29c93b Found by: previous frame's frame pointer 5 Chromium Framework!__ZN4base12_GLOBAL__N_121NativeStackSamplerMac17RecordStackSampleEPNS_18NativeStackSampler11StackBufferEPNS_21StackSamplingProfiler6SampleE + 0x39c rbp = 0x00007000098d3ca0 rsp = 0x00007000098d3610 rip = 0x000000011a29bd3c Found by: previous frame's frame pointer 6 Chromium Framework!__ZN4base21StackSamplingProfiler14SamplingThread12RecordSampleEPNS1_17CollectionContextE + 0x43f rbp = 0x00007000098d3e10 rsp = 0x00007000098d3cb0 rip = 0x000000011a29f8bf Found by: previous frame's frame pointer 7 Chromium Framework!__ZN4base21StackSamplingProfiler14SamplingThread21PerformCollectionTaskEi + 0xea rbp = 0x00007000098d3fb0 rsp = 0x00007000098d3e20 rip = 0x000000011a29fc0a Found by: previous frame's frame pointer 8 Chromium Framework!__ZN4base5debug13TaskAnnotator7RunTaskEPKcPNS_11PendingTaskE + 0x104 rbp = 0x00007000098d4170 rsp = 0x00007000098d3fc0 rip = 0x000000011a225614 Found by: previous frame's frame pointer 9 Chromium Framework!__ZN4base8internal17IncomingTaskQueue7RunTaskEPNS_11PendingTaskE + 0x79 rbp = 0x00007000098d42c0 rsp = 0x00007000098d4180 rip = 0x000000011a260429 Found by: previous frame's frame pointer 10 Chromium Framework!__ZN4base11MessageLoop7RunTaskEPNS_11PendingTaskE + 0x257 rbp = 0x00007000098d44b0 rsp = 0x00007000098d42d0 rip = 0x000000011a264cc7 Found by: previous frame's frame pointer 11 Chromium Framework!__ZN4base11MessageLoop21DeferOrRunPendingTaskENS_11PendingTaskE + 0xba rbp = 0x00007000098d4660 rsp = 0x00007000098d44c0 rip = 0x000000011a26508a Found by: previous frame's frame pointer 12 Chromium Framework!__ZN4base11MessageLoop6DoWorkEv + 0x23c rbp = 0x00007000098d48f0 rsp = 0x00007000098d4670 rip = 0x000000011a2652fc Found by: previous frame's frame pointer 13 Chromium Framework!__ZN4base18MessagePumpDefault3RunEPNS_11MessagePump8DelegateE + 0xdc rbp = 0x00007000098d4940 rsp = 0x00007000098d4900 rip = 0x000000011a26687c Found by: previous frame's frame pointer 14 Chromium Framework!__ZN4base11MessageLoop3RunEb + 0xa9 rbp = 0x00007000098d4aa0 rsp = 0x00007000098d4950 rip = 0x000000011a2645e9 Found by: previous frame's frame pointer 15 Chromium Framework!__ZN4base7RunLoop3RunEv + 0xf9 rbp = 0x00007000098d4bf0 rsp = 0x00007000098d4ab0 rip = 0x000000011a2a2c49 Found by: previous frame's frame pointer 16 Chromium Framework!__ZN4base6Thread3RunEPNS_7RunLoopE + 0xce rbp = 0x00007000098d4d40 rsp = 0x00007000098d4c00 rip = 0x000000011a2ef9be Found by: previous frame's frame pointer 17 Chromium Framework!__ZN4base6Thread10ThreadMainEv + 0x38c rbp = 0x00007000098d4ec0 rsp = 0x00007000098d4d50 rip = 0x000000011a2eff7c Found by: previous frame's frame pointer 18 Chromium Framework!__ZN4base12_GLOBAL__N_110ThreadFuncEPv + 0x5f rbp = 0x00007000098d4ef0 rsp = 0x00007000098d4ed0 rip = 0x000000011a2eac5f Found by: previous frame's frame pointer 19 libsystem_pthread.dylib + 0x36c1 rbp = 0x00007000098d4f20 rsp = 0x00007000098d4f00 rip = 0x00007fff6a13c6c1 Found by: previous frame's frame pointer 20 libsystem_pthread.dylib + 0x356d rbp = 0x00007000098d4f50 rsp = 0x00007000098d4f30 rip = 0x00007fff6a13c56d Found by: previous frame's frame pointer 21 libsystem_pthread.dylib + 0x2c5d rbp = 0x00007000098d4f78 rsp = 0x00007000098d4f60 rip = 0x00007fff6a13bc5d Found by: previous frame's frame pointer 22 Chromium Framework!__ZN4base14PlatformThread6DetachENS_20PlatformThreadHandleE + 0x70 rsp = 0x00007000098d5028 rip = 0x000000011a2eac00 Found by: stack scanning Not sure which of these two this one is: gs://chrome-telemetry-output/minidump-2018-03-09_12-32-27-592818.dmp gs://chrome-telemetry-output/minidump-2018-03-09_12-32-40-318581.dmp lgrey@, can you please take a look and see if anything can be learned from these minidumps? The stack sampler needs to be more reliable.