Chrome Version: 70.0.3504.0
OS: Windows 10

came back to my computer today to find a totally hung browser UI.

in process explorer, two processes were 'suspended' one was the GPU process, one was a renderer process.

The browser UI was hung and attaching a debugger showed thread stuck waiting for GPU:

0:000> kn
 # Child-SP          RetAddr           Call Site
00 000000cf`5fbfd938 00007ffd`15b69252 ntdll!NtWaitForSingleObject+0x14
01 000000cf`5fbfd940 00007ffc`af1d874e KERNELBASE!WaitForSingleObjectEx+0xa2
02 000000cf`5fbfd9e0 00007ffc`af5bf1ab chrome_7ffcaf100000!base::WaitableEvent::Wait+0x4e [C:\b\c\b\win64_clang\src\base\synchronization\waitable_event_win.cc @ 66]
03 000000cf`5fbfda60 00007ffc`af7e8880 chrome_7ffcaf100000!gpu::GpuChannelHost::Send+0x1c1 [C:\b\c\b\win64_clang\src\gpu\ipc\client\gpu_channel_host.cc @ 85]
04 000000cf`5fbfdbc0 00007ffc`af7e87ca chrome_7ffcaf100000!gpu::GpuChannelHost::VerifyFlush+0x52 [C:\b\c\b\win64_clang\src\gpu\ipc\client\gpu_channel_host.cc @ 135]
05 000000cf`5fbfdc00 00007ffc`af6ab2a9 chrome_7ffcaf100000!gpu::gles2::GLES2Implementation::VerifySyncTokensCHROMIUM+0xb8 [C:\b\c\b\win64_clang\src\gpu\command_buffer\client\gles2_implementation.cc @ 6179]
06 000000cf`5fbfdc80 00007ffc`af868b6b chrome_7ffcaf100000!viz::DisplayResourceProvider::DeleteAndReturnUnusedResourcesToChild+0x629 [C:\b\c\b\win64_clang\src\components\viz\service\display\display_resource_provider.cc @ 727]
07 000000cf`5fbfddf0 00007ffc`af6ae2b7 chrome_7ffcaf100000!viz::DisplayResourceProvider::DestroyChildInternal+0xab [C:\b\c\b\win64_clang\src\components\viz\service\display\display_resource_provider.cc @ 757]
08 000000cf`5fbfde80 00007ffc`af6a4e06 chrome_7ffcaf100000!viz::SurfaceAggregator::ProcessAddedAndRemovedSurfaces+0xd7 [C:\b\c\b\win64_clang\src\components\viz\service\display\surface_aggregator.cc @ 815]
09 000000cf`5fbfdf00 00007ffc`af6a3ae8 chrome_7ffcaf100000!viz::SurfaceAggregator::Aggregate+0x556 [C:\b\c\b\win64_clang\src\components\viz\service\display\surface_aggregator.cc @ 1224]
0a 000000cf`5fbfe1d0 00007ffc`af6a37b0 chrome_7ffcaf100000!viz::Display::DrawAndSwap+0x1f8 [C:\b\c\b\win64_clang\src\components\viz\service\display\display.cc @ 287]
0b 000000cf`5fbfe480 00007ffc`af6a360e chrome_7ffcaf100000!viz::DisplayScheduler::DrawAndSwap+0x50 [C:\b\c\b\win64_clang\src\components\viz\service\display\display_scheduler.cc @ 212]
0c 000000cf`5fbfe560 00007ffc`af1248dc chrome_7ffcaf100000!viz::DisplayScheduler::OnBeginFrameDeadline+0x4e [C:\b\c\b\win64_clang\src\components\viz\service\display\display_scheduler.cc @ 500]
0d 000000cf`5fbfe660 00007ffc`af1242d7 chrome_7ffcaf100000!base::debug::TaskAnnotator::RunTask+0x12c [C:\b\c\b\win64_clang\src\base\debug\task_annotator.cc @ 101]
0e 000000cf`5fbfe780 00007ffc`af11d288 chrome_7ffcaf100000!base::MessageLoop::RunTask+0x247 [C:\b\c\b\win64_clang\src\base\message_loop\message_loop.cc @ 426]
0f 000000cf`5fbfe8e0 00007ffc`af21af49 chrome_7ffcaf100000!base::MessageLoop::DoWork+0x198 [C:\b\c\b\win64_clang\src\base\message_loop\message_loop.cc @ 483]
10 000000cf`5fbfead0 00007ffc`af11cf6e chrome_7ffcaf100000!base::MessagePumpForUI::DoRunLoop+0xa9 [C:\b\c\b\win64_clang\src\base\message_loop\message_pump_win.cc @ 172]
11 000000cf`5fbfeb80 00007ffc`af11ccd1 chrome_7ffcaf100000!base::MessagePumpWin::Run+0x4e [C:\b\c\b\win64_clang\src\base\message_loop\message_pump_win.cc @ 54]
12 000000cf`5fbfebd0 00007ffc`af4e1abe chrome_7ffcaf100000!base::RunLoop::Run+0x31 [C:\b\c\b\win64_clang\src\base\run_loop.cc @ 108]
13 000000cf`5fbfec00 00007ffc`af4e18c2 chrome_7ffcaf100000!ChromeBrowserMainParts::MainMessageLoopRun+0x92 [C:\b\c\b\win64_clang\src\chrome\browser\chrome_browser_main.cc @ 2096]
14 000000cf`5fbfecd0 00007ffc`af4e186d chrome_7ffcaf100000!content::BrowserMainLoop::RunMainMessageLoopParts+0x48 [C:\b\c\b\win64_clang\src\content\browser\browser_main_loop.cc @ 1036]
15 000000cf`5fbfed90 00007ffc`afd810ee chrome_7ffcaf100000!content::BrowserMainRunnerImpl::Run+0x11 [C:\b\c\b\win64_clang\src\content\browser\browser_main_runner_impl.cc @ 163]
16 000000cf`5fbfedc0 00007ffc`b02bb947 chrome_7ffcaf100000!content::BrowserMain+0xc6 [C:\b\c\b\win64_clang\src\content\browser\browser_main.cc @ 47]
17 000000cf`5fbfeea0 00007ffc`b02bbcf9 chrome_7ffcaf100000!content::RunBrowserProcessMain+0x6f [C:\b\c\b\win64_clang\src\content\app\content_main_runner_impl.cc @ 596]
18 000000cf`5fbfef10 00007ffc`af105a4b chrome_7ffcaf100000!content::ContentMainRunnerImpl::Run+0x309 [C:\b\c\b\win64_clang\src\content\app\content_main_runner_impl.cc @ 952]
19 000000cf`5fbff0d0 00007ffc`af105648 chrome_7ffcaf100000!service_manager::Main+0x336 [C:\b\c\b\win64_clang\src\services\service_manager\embedder\main.cc @ 472]
1a 000000cf`5fbff400 00007ffc`af101e5a chrome_7ffcaf100000!content::ContentMain+0x41 [C:\b\c\b\win64_clang\src\content\app\content_main.cc @ 19]
1b 000000cf`5fbff490 00007ff6`802f372c chrome_7ffcaf100000!ChromeMain+0x118 [C:\b\c\b\win64_clang\src\chrome\app\chrome_main.cc @ 104]
1c 000000cf`5fbff570 00007ff6`802f1699 chrome!MainDllLoader::Launch+0x26c [C:\b\c\b\win64_clang\src\chrome\app\main_dll_loader_win.cc @ 201]
1d 000000cf`5fbff660 00007ff6`803b4ca2 chrome!wWinMain+0x699 [C:\b\c\b\win64_clang\src\chrome\app\chrome_exe_main_win.cc @ 230]
1e 000000cf`5fbffa40 00007ffd`17b73034 chrome!__scrt_common_main_seh+0x106 [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283]
1f 000000cf`5fbffa80 00007ffd`197d1431 KERNEL32!BaseThreadInitThunk+0x14
20 000000cf`5fbffab0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

The GPU process seems to be hung on an exception in a suspended state, it appears to be an OOM:

0:000> kn
 # Child-SP          RetAddr           Call Site
00 000000c3`b49fc908 00007ffd`15b75e9a ntdll!NtDelayExecution+0x14
01 000000c3`b49fc910 00007ffc`e2402f21 KERNELBASE!SleepEx+0x9a
02 000000c3`b49fc9b0 00007ffd`15c1c0a0 chrome_elf!crashpad::`anonymous namespace'::UnhandledExceptionHandler+0xdd [C:\b\c\b\win64_clang\src\third_party\crashpad\crashpad\client\crashpad_client_win.cc @ 174]
03 000000c3`b49fcb00 00007ffd`198026d7 KERNELBASE!UnhandledExceptionFilter+0x190
04 000000c3`b49fcc10 00007ffd`197eab06 ntdll!RtlUserThreadStart$filt$0+0x38
05 000000c3`b49fcc40 00007ffd`197feced ntdll!_C_specific_handler+0x96
06 000000c3`b49fccb0 00007ffd`19766c86 ntdll!RtlpExecuteHandlerForException+0xd
07 000000c3`b49fcce0 00007ffd`197652ca ntdll!RtlDispatchException+0x3c6
08 000000c3`b49fd3e0 00007ffd`15b6a388 ntdll!RtlRaiseException+0x31a
09 000000c3`b49fdbc0 00007ffc`a5e148f4 KERNELBASE!RaiseException+0x68
0a 000000c3`b49fdca0 00007ffc`a6213d27 chrome_child!base::`anonymous namespace'::OnNoMemory+0x24 [C:\b\c\b\win64_clang\src\base\process\memory_win.cc @ 55]
0b 000000c3`b49fdcd0 00007ffc`a4651136 chrome_child!base::allocator::WinCallNewHandler+0x17 [C:\b\c\b\win64_clang\src\base\allocator\winheap_stubs_win.cc @ 90]
0c 000000c3`b49fdd00 00007ffc`a82f948b chrome_child!malloc+0x46 [C:\b\c\b\win64_clang\src\base\allocator\allocator_shim_override_ucrt_symbols_win.h @ 53]
0d 000000c3`b49fdd40 00007ffc`a742537c chrome_child!operator new+0x1f [f:\dd\vctools\crt\vcstartup\src\heap\new_scalar.cpp @ 35]
0e 000000c3`b49fdd70 00007ffc`a6ccad74 chrome_child!gpu::gles2::GLES2DecoderImpl::ClearLevel+0x236 [C:\b\c\b\win64_clang\src\gpu\command_buffer\service\gles2_cmd_decoder.cc @ 13185]
0f 000000c3`b49fdef0 00007ffc`a6cca962 chrome_child!gpu::gles2::Texture::ClearLevel+0x3ba [C:\b\c\b\win64_clang\src\gpu\command_buffer\service\texture_manager.cc @ 1710]
10 000000c3`b49fe000 00007ffc`a7420b24 chrome_child!gpu::gles2::Texture::ClearRenderableLevels+0x88 [C:\b\c\b\win64_clang\src\gpu\command_buffer\service\texture_manager.cc @ 1575]
11 000000c3`b49fe060 00007ffc`a74221a8 chrome_child!gpu::gles2::GLES2DecoderImpl::ClearUnclearedTextures+0xd4 [C:\b\c\b\win64_clang\src\gpu\command_buffer\service\gles2_cmd_decoder.cc @ 10270]
12 000000c3`b49fe0c0 00007ffc`a73f7420 chrome_child!gpu::gles2::GLES2DecoderImpl::DoDrawElements+0x282 [C:\b\c\b\win64_clang\src\gpu\command_buffer\service\gles2_cmd_decoder.cc @ 10873]
13 000000c3`b49fe160 00007ffc`a7415929 chrome_child!gpu::gles2::GLES2DecoderImpl::HandleDrawElements+0x38 [C:\b\c\b\win64_clang\src\gpu\command_buffer\service\gles2_cmd_decoder.cc @ 10944]
14 000000c3`b49fe1b0 00007ffc`a741523d chrome_child!gpu::gles2::GLES2DecoderImpl::DoCommandsImpl<0>+0xfd [C:\b\c\b\win64_clang\src\gpu\command_buffer\service\gles2_cmd_decoder.cc @ 5649]
15 000000c3`b49fe330 00007ffc`a73cd551 chrome_child!gpu::gles2::GLES2DecoderImpl::DoCommands+0x25 [C:\b\c\b\win64_clang\src\gpu\command_buffer\service\gles2_cmd_decoder.cc @ 5703]
16 000000c3`b49fe360 00007ffc`a6cc4a38 chrome_child!gpu::CommandBufferService::Flush+0xef [C:\b\c\b\win64_clang\src\gpu\command_buffer\service\command_buffer_service.cc @ 90]
17 000000c3`b49fe4b0 00007ffc`a6cc4842 chrome_child!gpu::CommandBufferStub::OnAsyncFlush+0x112 [C:\b\c\b\win64_clang\src\gpu\ipc\service\command_buffer_stub.cc @ 620]
18 000000c3`b49fe600 00007ffc`a6cc353b chrome_child!IPC::MessageT<GpuCommandBufferMsg_AsyncFlush_Meta,std::tuple<int,unsigned int>,void>::Dispatch<gpu::CommandBufferStub,gpu::CommandBufferStub,void,void (gpu::CommandBufferStub::*)(int, unsigned int)>+0x92 [C:\b\c\b\win64_clang\src\ipc\ipc_message_templates.h @ 146]
19 000000c3`b49fe700 00007ffc`a65eba12 chrome_child!gpu::CommandBufferStub::OnMessageReceived+0x219 [C:\b\c\b\win64_clang\src\gpu\ipc\service\command_buffer_stub.cc @ 280]
1a 000000c3`b49fe930 00007ffc`a65ea53e chrome_child!gpu::GpuChannel::HandleMessageHelper+0x32 [C:\b\c\b\win64_clang\src\gpu\ipc\service\gpu_channel.cc @ 541]
1b 000000c3`b49fe970 00007ffc`a65dd607 chrome_child!gpu::GpuChannel::HandleMessage+0x5e [C:\b\c\b\win64_clang\src\gpu\ipc\service\gpu_channel.cc @ 517]
1c 000000c3`b49fea10 00007ffc`a469d74c chrome_child!gpu::Scheduler::RunNextTask+0x2dd [C:\b\c\b\win64_clang\src\gpu\command_buffer\service\scheduler.cc @ 526]
1d 000000c3`b49feb80 00007ffc`a469cfb7 chrome_child!base::debug::TaskAnnotator::RunTask+0x12c [C:\b\c\b\win64_clang\src\base\debug\task_annotator.cc @ 101]
1e 000000c3`b49feca0 00007ffc`a4697f48 chrome_child!base::MessageLoop::RunTask+0x247 [C:\b\c\b\win64_clang\src\base\message_loop\message_loop.cc @ 426]
1f 000000c3`b49fee00 00007ffc`a4697d89 chrome_child!base::MessageLoop::DoWork+0x198 [C:\b\c\b\win64_clang\src\base\message_loop\message_loop.cc @ 483]
20 000000c3`b49feff0 00007ffc`a4697aa1 chrome_child!base::MessagePumpDefault::Run+0x99 [C:\b\c\b\win64_clang\src\base\message_loop\message_pump_default.cc @ 37]
21 000000c3`b49ff050 00007ffc`a61b6afa chrome_child!base::RunLoop::Run+0x31 [C:\b\c\b\win64_clang\src\base\run_loop.cc @ 108]
22 000000c3`b49ff080 00007ffc`a5df0342 chrome_child!content::GpuMain+0x396 [C:\b\c\b\win64_clang\src\content\gpu\gpu_main.cc @ 346]
23 000000c3`b49ff380 00007ffc`a4654bdb chrome_child!content::ContentMainRunnerImpl::Run+0x1ee [C:\b\c\b\win64_clang\src\content\app\content_main_runner_impl.cc @ 951]
24 000000c3`b49ff530 00007ffc`a46547d8 chrome_child!service_manager::Main+0x336 [C:\b\c\b\win64_clang\src\services\service_manager\embedder\main.cc @ 472]
25 000000c3`b49ff860 00007ffc`a4651c3d chrome_child!content::ContentMain+0x41 [C:\b\c\b\win64_clang\src\content\app\content_main.cc @ 19]
26 000000c3`b49ff8f0 00007ff6`802f372c chrome_child!ChromeMain+0x118 [C:\b\c\b\win64_clang\src\chrome\app\chrome_main.cc @ 104]
27 000000c3`b49ff9d0 00007ff6`802f1699 chrome!MainDllLoader::Launch+0x26c [C:\b\c\b\win64_clang\src\chrome\app\main_dll_loader_win.cc @ 201]
28 000000c3`b49ffac0 00007ff6`803b4ca2 chrome!wWinMain+0x699 [C:\b\c\b\win64_clang\src\chrome\app\chrome_exe_main_win.cc @ 230]
29 000000c3`b49ffea0 00007ffd`17b73034 chrome!__scrt_common_main_seh+0x106 [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283]
2a 000000c3`b49ffee0 00007ffd`197d1431 kernel32!BaseThreadInitThunk+0x14
2b 000000c3`b49fff10 00000000`00000000 ntdll!RtlUserThreadStart+0x21

I also have a WER report for crashpad process crashing around the same time:

0:006> kn
 # Child-SP          RetAddr           Call Site
00 00000064`f93feb30 00007ff6`80329354 KERNELBASE!RaiseException+0x68
01 00000064`f93fec10 00007ff6`80322bc7 chrome!base::`anonymous namespace'::OnNoMemory+0x24 [C:\b\c\b\win64_clang\src\base\process\memory_win.cc @ 55]
02 00000064`f93fec40 00007ff6`8030d476 chrome!base::allocator::WinCallNewHandler+0x17 [C:\b\c\b\win64_clang\src\base\allocator\winheap_stubs_win.cc @ 90]
03 00000064`f93fec70 00007ff6`803b1e1f chrome!malloc+0x46 [C:\b\c\b\win64_clang\src\base\allocator\allocator_shim_override_ucrt_symbols_win.h @ 53]
04 00000064`f93fecb0 00007ff6`80396ee5 chrome!operator new+0x1f [f:\dd\vctools\crt\vcstartup\src\heap\new_scalar.cpp @ 35]
05 00000064`f93fece0 00007ff6`80396c1a chrome!std::vector<_MINIDUMP_MEMORY_INFO,std::allocator<_MINIDUMP_MEMORY_INFO> >::_Emplace_reallocate<const _MINIDUMP_MEMORY_INFO &>+0xb5 [C:\b\c\b\win64_clang\src\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\VC\Tools\MSVC\14.14.26428\include\vector @ 1005]
06 00000064`f93fed40 00007ff6`8037a58b chrome!crashpad::MinidumpMemoryInfoListWriter::InitializeFromSnapshot+0x5a [C:\b\c\b\win64_clang\src\third_party\crashpad\crashpad\minidump\minidump_memory_info_writer.cc @ 35]
07 00000064`f93fed90 00007ff6`80353fe6 chrome!crashpad::MinidumpFileWriter::InitializeFromSnapshot+0x32b [C:\b\c\b\win64_clang\src\third_party\crashpad\crashpad\minidump\minidump_file_writer.cc @ 130]
08 00000064`f93ff030 00007ff6`8035632b chrome!crashpad::CrashReportExceptionHandler::ExceptionHandlerServerException+0x1b6 [C:\b\c\b\win64_clang\src\third_party\crashpad\crashpad\handler\win\crash_report_exception_handler.cc @ 107]
09 00000064`f93ff470 00007ffd`197a2e5b chrome!crashpad::ExceptionHandlerServer::OnCrashDumpEvent+0x2b [C:\b\c\b\win64_clang\src\third_party\crashpad\crashpad\util\win\exception_handler_server.cc @ 545]
0a 00000064`f93ff4b0 00007ffd`197a2d04 ntdll!RtlpTpWaitCallback+0x9b
0b 00000064`f93ff520 00007ffd`1977f5c9 ntdll!TppExecuteWaitCallback+0xa0
0c 00000064`f93ff570 00007ffd`17b73034 ntdll!TppWorkerThread+0x2a9
0d 00000064`f93ff860 00007ffd`197d1431 kernel32!BaseThreadInitThunk+0x14
0e 00000064`f93ff890 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Also an OOM...

So what seems to have happened is

1. GPU ooms
2. Crashpad tries to capture the process, but the crashpad handler crashes due to OOM
3. GPU process never un-suspends and never gets terminated
4. Browser process is waiting on a GPU process that has totally hung, and so the browser UI hangs.

It seems we should be more resilient to this happening, perhaps we need to timeout the crashpad wait if the crashpad service never connects to get the dump and unsuspend the process?

The dumps for all these processes are all in drive folder:

http://shortn/_OVQYXqjR4j


 

Deleted: chrome-processes.png 19.2 KB

	chrome-processes.png 19.2 KB View Download