A similar deadlock/crash upon shutdown has also been seen in the wild on Windows

issue 781714

 Issue 783434  has been merged into this issue.

GpuProcessHostBrowserTest.Shutdown exposes a similar shutdown deadlock when running as Viz. See duped  issue 783434  for the stack trace.

As discussed in triage I'm assigning to kylechar@

I've duped in one  issue 783434 , the other issue 781714 is tracking a version of this in the wild. I can be duped in if appropriate

This is more problematic than I first though. The test is triggering a context loss error to test recovery. On some platforms that is treated as a fatal error for the GPU process. The GPU main thread RunLoop is stopped here:

https://cs.chromium.org/chromium/src/gpu/ipc/service/gpu_channel_manager.cc?l=204&rcl=98c33f635308a47061a9bc35f6a638f9f78eac16

That triggers the VizMainImpl destructor, which tries to destroy the VizCompositorThread. The browser isn't involved in this and hasn't destroyed RootCompositorFrameSinkImpls yet. As a result, when RootCompositorFrameSinkImpl is destroyed the InProcessCommandBuffer causes a deadlock. Since the browser process isn't involved, it can't destroy the RootCompositorFrameSinkImpls first.

+piman

The quickest solution would be in the FrameSinkManagerImpl destructor check if  any RootCompositorFrameSinkImpls still exist and CHECK(false) to crash. The GPU is shutting down anyways at this point. This situation shouldn't happen for browser process initiated the GPU shutdown, only GPU process initiated GPU shutdown.

The other solution is to use a nested message loop to keep GPU main thread responsive. piman objected to this when I landed changes to VizMainImpl originally so this probably isn't an option.

+piman for real, please see above.

We could do a non-nested message loop in GpuMain when viz is on, to make sure we can tear-down the compositor thread with the gpu thread still working.

Longer term, we need to rework these threads, having the GPU thread double as the main thread is problematic.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0a7b11f6cd1d844204ccd814a4b40b63c106c26f

commit 0a7b11f6cd1d844204ccd814a4b40b63c106c26f
Author: kylechar <kylechar@chromium.org>
Date: Wed Mar 21 20:33:05 2018

viz: Enforce destruction order.

Enforce that all mojom::CompositorFrameSinks are destroyed before
FrameSinkManagerImpl is destroyed. Running with VizDisplayCompositor
this will ensure that no RootCompositorFrameSinkImpls exist that could
deadlock the compositor and GPU thread on destruction. For normal Chrome
this will only effect offscreen canvas. Fix offscreen canvas tests so
they destroy everything that is created.

Bug:  820437 
Change-Id: I5bd71456ea86fb6b1d89347f4d39de75a72dd86a
Reviewed-on: https://chromium-review.googlesource.com/969590
Commit-Queue: Fady Samuel <fsamuel@chromium.org>
Commit-Queue: kylechar <kylechar@chromium.org>
Reviewed-by: Fady Samuel <fsamuel@chromium.org>
Reviewed-by: Olivia Lai <xlai@chromium.org>
Cr-Commit-Position: refs/heads/master@{#544827}
[modify] https://crrev.com/0a7b11f6cd1d844204ccd814a4b40b63c106c26f/components/viz/service/frame_sinks/frame_sink_manager_impl.cc
[modify] https://crrev.com/0a7b11f6cd1d844204ccd814a4b40b63c106c26f/content/browser/renderer_host/offscreen_canvas_provider_impl_unittest.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/56ddcdc22b051a2ed4064e0dc15f05da365404c4

commit 56ddcdc22b051a2ed4064e0dc15f05da365404c4
Author: kylechar <kylechar@chromium.org>
Date: Mon May 28 16:45:38 2018

viz: Allow GPU to exit cleanly with OOP-D.

The GPU process will exit and restart in response to certain errors or
an IPC from the browser process. This is complicated with OOP-D as the
compositor thread needs to tear down RootCompositorFrameSinkImpls while
the GPU thread is still running and isn't blocked. This is because
InProcessCommandBuffer straddles the two threads.

Add functionality to GpuServiceImpl and VizMainImpl to allow for clean
shutdown. This entails the following:

1. GPU thread: Closing mojom::VizMain binding. This will prevent the
   browser process from reconnecting to FrameSinkManagerImpl.
2. Compositor thread: Closing mojom::FrameSinkManager binding. This will
   prevent the browser process from creating more CompositorFrameSinks.
3. Compositor thread: Destroying all CompositorFrameSinks.
4. GPU thread: Quit the RunLoop to run process shutdown code.

Bug:  820437 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: I86e29f8719dc2b44561c9a071a5cca3f9239d514
Reviewed-on: https://chromium-review.googlesource.com/1070216
Commit-Queue: kylechar <kylechar@chromium.org>
Reviewed-by: Jonathan Backer <backer@chromium.org>
Reviewed-by: Sadrul Chowdhury <sadrul@chromium.org>
Cr-Commit-Position: refs/heads/master@{#562279}
[modify] https://crrev.com/56ddcdc22b051a2ed4064e0dc15f05da365404c4/components/viz/service/frame_sinks/frame_sink_manager_impl.cc
[modify] https://crrev.com/56ddcdc22b051a2ed4064e0dc15f05da365404c4/components/viz/service/frame_sinks/frame_sink_manager_impl.h
[modify] https://crrev.com/56ddcdc22b051a2ed4064e0dc15f05da365404c4/components/viz/service/gl/gpu_service_impl.cc
[modify] https://crrev.com/56ddcdc22b051a2ed4064e0dc15f05da365404c4/components/viz/service/gl/gpu_service_impl.h
[modify] https://crrev.com/56ddcdc22b051a2ed4064e0dc15f05da365404c4/components/viz/service/gl/gpu_service_impl_unittest.cc
[modify] https://crrev.com/56ddcdc22b051a2ed4064e0dc15f05da365404c4/components/viz/service/main/viz_main_impl.cc
[modify] https://crrev.com/56ddcdc22b051a2ed4064e0dc15f05da365404c4/components/viz/service/main/viz_main_impl.h
[modify] https://crrev.com/56ddcdc22b051a2ed4064e0dc15f05da365404c4/gpu/ipc/service/gpu_channel_manager.cc
[modify] https://crrev.com/56ddcdc22b051a2ed4064e0dc15f05da365404c4/gpu/ipc/service/gpu_channel_manager_delegate.h
[modify] https://crrev.com/56ddcdc22b051a2ed4064e0dc15f05da365404c4/gpu/ipc/service/gpu_channel_test_common.cc
[modify] https://crrev.com/56ddcdc22b051a2ed4064e0dc15f05da365404c4/services/ui/ws/gpu_host_unittest.cc
[modify] https://crrev.com/56ddcdc22b051a2ed4064e0dc15f05da365404c4/testing/buildbot/filters/viz.content_browsertests.filter