I was able to repro this, so I cut out waitable_event_mac.cc and switched back to waitable_event_posix.cc, and the problem persisted.

Re #1 about memory-infra.
I think the thing we can reasonably fix is to exclude hung renderers from the dump and continue with the other processes. Tracked bug for this is  Issue 727785 .
Instead, excluding individual MDPs and saving the rest of the dump for the process will be harder, because for reasons we allow some MDPs to take a look to the result injected by other MDPs in the same dump, so we can't easily make the dump process non-serial (i.e. we can't easily turn the sequence of hops into a "broadcast tasks to all task runniners and then join with timeout") 

I don't know anything about gpu architecture, but anecdotally renderer hangs like this (multiple threads stuck on a waitable event) are due to our code interlocking the task runners and getting in a deadlock.
A classical pattern I've seen in the past is something like: TaskRunner A blocking itself on a event.Wait(), waiting for a task posted on TaskRunner B to signal it. TaskRunner B waiting for a task posted on A to execute. --> deadlock.

+ ericrk, ccameron, piman

No response from vmiura [and I've pinged], so ccing some other GPU folks. This seems like a potentially serious problem. 

Here's the flow:

1- ClientGpuMemoryBufferManager::CreateGpuMemoryBuffer posts a task to the GpuMemoryThread, calling AllocateGpuMemoryBufferOnThread, and waiting for the result (WaitableEvent).
2- ClientGpuMemoryBufferManager::AllocateGpuMemoryBufferOnThread sends a message to the browser process, expecting an async reply
3- This request should be served by a content::GpuClient, running on the browser IO thread, owned by the RenderProcessHostImpl (see RenderProcessHostImpl::CreateMusGpuRequest).
4- GpuClient::CreateGpuMemoryBuffer talks to BrowserGpuMemoryBufferManager::AllocateGpuMemoryBufferForChildProcess, which on Mac should go through the "native buffer" path (assuming default flags).
5- BrowserGpuMemoryBufferManager should then connect to the GPU process (instantiating it if necessary, this shouldn't be the case here unless it crashed), and sending an async message (see GpuProcessHost::CreateGpuMemoryBuffer)
6- the message is serviced on the GPU process IO thread (see GpuServiceImpl::CreateGpuMemoryBuffer).

At this point the callback flow should send back the response.

7- GpuServiceImpl::CreateGpuMemoryBuffer will run the mojo callback, which should be received by GpuProcessHost::OnGpuMemoryBufferCreated on the browser IO thread, calling back BrowserGpuMemoryBufferManager (unless GpuProcessHost was destroyed because of a GPU crash, in which case the callback should have been called by GpuProcessHost::SendOutstandingReplies with an error). 
8- in case of error because the gpu process died, BrowserGpuMemoryBufferManager::GpuMemoryBufferCreatedOnIO may retry creating it (going back to #5), but will otherwise call back GpuClient
9- GpuClient::OnCreateGpuMemoryBuffer should be called - unless the GpuClient was destroyed in the mean time, but see below - which will run the mojo callback.
10- ClientGpuMemoryBufferManager::OnGpuMemoryBufferAllocatedOnThread will be called on the GpuMemoryThread, which will signal the WaitableEvent and unlock CreateGpuMemoryBuffer.

If the GpuClient was destroyed in step 9, the mojo interface should be disconnected, and ClientGpuMemoryBufferManager::DisconnectGpuOnThread should be called which should signal the WaitableEvent and unlock CreateGpuMemoryBuffer.



Obviously, a bug in any of these steps where a message gets lost without ensuring the corresponding callback is called might cause a situation like this, but the fact that attaching a debugger make things go makes me think that it's not what happens. Possibly live-lock situations or thread descheduling/starvation or something that would just resolve itself after attaching the debugger got things to cool off.

Some additional data that would be helpful:
1- in the blocked situation, do we have any evidence that any of the other involved threads (browser IO, GPU IO) are busy?
2- what triggers signaling the WaitableEvent after attaching the debugger? Is that ClientGpuMemoryBufferManager::OnGpuMemoryBufferAllocatedOnThread or ClientGpuMemoryBufferManager::DisconnectGpuOnThread

Thanks for the feedback!

If I let the test sit there for >1min, it eventually completes. The browser process is spinning at 1 full core, so we're looking at a live lock. Sampling the process returns almost no CPU usage [very suspicious]. 

Doing some more debugging, the browser process to try to compute its own resident memory. On macOS, this crawls the virtual address space. This causes ASAN to blow up. Thus explaining why the renderer never gets the callback it's expecting.

Hm. There's more to this story. The hang [really like a 70s wait] still occurs, even with my fixes. I've confirmed we're no longer crawling the virtual address space. I'm going to see if this renderer hang happens even without my code.

+ sadrul, samans

I can reproduce this issue non-deterministically on ToT [~50% of the time]. The easiest way to do so is to take the test in question, add a while(true){sleep(1000);} to the end, and then use "ps aux | grep Chromium" to see whether there's a renderer process sticking around that should have died.

I added a bunch of log statements. Here's what I observe:

In the renderer process:
 * The main thread is waiting in ProxyMain::Stop.
 * The impl thread is waiting in ProxyImpl::~ProxyImpl
 * The compositor tile worker is waiting in ClientGpuMemoryBufferManager::CreateGpuMemoryBuffer.
 * The IO thread and GpuMemoryThread are not blocked.

In the browser process:
 * The GpuClient receives the request, and sends a reply. 
 * The GpuClient is then destroyed.

In the renderer process:
 * The reply is not received, and the connection_error_handler is not called. This causes the deadlock.
 * AFAICT, the InterfacePtr is bound to the GpuMemoryThread, which is not blocked.

The only thing that's a little bit suspicious is that InterfacePtr is first bound on the main thread, and then transferred to the GpuMemoryThread via PassInterface. reillyg@ says this should work [as long as no messages are passed in the interim, which I've confirmed]. But then, there's a second InterfacePtr bound on the main thread to the same connector. 

I tried cloning the connector that gets used for the GpuMemoryThread InterfacePtr, but that doesn't make a difference. I'm out of ideas.

I agree with your assessment that the GpuPtr is bound on the GpuMemoryThread. I also don't see any obvious races between signaling and ClientGpuMemoryBufferManager shutdown that aren't accounted for. I do know there have been bugs here in the past though, and the logic is a bit subtle.

Just to rule out one remotely possible class of surprises, can you verify that at the time of the deadlock, |gpu_| is still bound?

I added a repeating timer to the GpuMemoryThread that checks on gpu_.is_bound(). I've confirmed that:

1) The GpuMemoryThread is not blocked.
2) gpu_ is bound.

*sigh*. This hole keeps digging deeper.

I think there may be several different issues. At least, I'm observing different symptoms when I dig down.

question: If the pipe is disconnected, a task is added to MultiplexRouter::tasks_. Is it possible to deadlock if MultiplexRouter::ProcessTasks doesn't process the message [let's say b/c something needs to touch the main thread]?

I think I'm seeing the following:
1) Browser process - GpuClient is Destroyed.
2) Renderer process calls Connector::HandleError:
3) Renderer process calls MultiplexRouter::OnPipeConnectionError:
4) Renderer process calls MultiplexRouter::ProcessNotifyErrorTask:
5) Renderer process ClientGpuMemoryBufferManager::DisconnectGpuOnThread is NOT called.
My repeating timer on the GpuMemoryThread shows that pending_allocation_waiters_ is 1, and gpu_ is bound. 

I'm also observing a different case where my timer on the GPUMemoryThread is indicating the gpu_ is not bound, but the process is still not shutting down.

Okay, I have a theory. First, observation of timing for context:

1) Renderer process calls CreateGpuMemoryBuffer on the GpuMemoryThread.
2) Browser process receives the call, processes it, and invokes the callback. [the callback never makes it to the renderer process].
3) The renderer process begins to shut down. Main thread blocked, compositor thread blocked, etc. GpuMemoryThread still unblocked.
4) Browser process destroys RenderProcessHostImpl and GpuClient
5) Browser process destroys the mojom::Gpu Binding. This calls Node::ClosePort on the underlying ports.
6) Renderer process receives Node::OnObserveClosure for the same ports.
7) Renderer process begins dispatching callbacks for Node::OnObserveClosure, via the Dispatch/Watcher system.

This is where it gets weird.
8) There is a GpuInfo connector bound to the GpuMemoryThread. It has two watchers: handle_watcher_ and peer_remoteness_tracker_. The latter receives a callback on the GpuMemoryThread. No callback is queued for the former, even though IsWatching() is true, and it's armed [confirmed with logging]. 
9) Making the following change fixes the problem. The appropriate callback is queued and run on the appropriate thread. There are no hung renderer processes.

Before:
"""
void Connector::WaitToReadMore() {    
...
389   MojoResult rv = handle_watcher_->Watch(                                       
390       message_pipe_.get(), MOJO_HANDLE_SIGNAL_READABLE,                                                                                                                                                                           
391       base::Bind(&Connector::OnWatcherHandleReady, base::Unretained(this)));   
"""

After:
"""
void Connector::WaitToReadMore() {    
...
389   MojoResult rv = handle_watcher_->Watch(                                       
390       message_pipe_.get(), MOJO_HANDLE_SIGNAL_READABLE | MOJO_HANDLE_SIGNAL_PEER_CLOSED,                                                                                                                                                                                                            
391       base::Bind(&Connector::OnWatcherHandleReady, base::Unretained(this)));   
"""

This makes logical sense. handler_watcher_ should be watching for the PEER_CLOSED signal. But this begs the question...how does this work at all without the change I proposed? I threw up a CL to run through the CQ: https://chromium-review.googlesource.com/c/chromium/src/+/714370. I'm going to sleep on this, and confer more with rockot@ tomorrow.

+siggi, FYI, since he just pinged me about hangs w/ certain sites, under SyzyASAN Canary.

We believe this has been tracked down to OS X specific behavior around Mach
port transfer, so probably not likely to be related to any SyzyASAN hangs.

And just to follow-up from comment #12, most of that is correct, but |handle_watcher_| is fine as-is; it should not be watching for PEER_CLOSED.

The issue here was around how Mach port transfer is handled in Mojo internals, combined with pipe signaling semantics. A pipe with a closed peer remains readable as long as it knows there are still unreceived messages in transit.

Meanwhile, if a Mach port fails to transfer on the sending side for an outgoing message, we silently drop that message instead of sending it. The result in this case is that the renderer has been told to expect a message on the Gpu pipe, but that message is never going to arrive. It knows the peer is closed but still sees the pipe as readable, indefinitely.

The solution we discussed is to ensure that messages are still delivered in such cases, albeit with invalid mach ports attached.

Issue 762463 has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/55090272269b587b30cc3eb277184644c4b16088

commit 55090272269b587b30cc3eb277184644c4b16088
Author: erikchen <erikchen@chromium.org>
Date: Fri Oct 13 18:41:28 2017

Messages with Mach Ports that fail to broker should still be sent.

Mojo enforces strict message ordering. Failing to send a message will prevent
all future messages from being read, and will also prevent the pipe disconnect
from being processed.

Bug:  771805 
Change-Id: I1d9db2b9d9f280eebd58bb8793df7b53eb4e77c1
Reviewed-on: https://chromium-review.googlesource.com/716857
Commit-Queue: Erik Chen <erikchen@chromium.org>
Reviewed-by: Ken Rockot <rockot@chromium.org>
Cr-Commit-Position: refs/heads/master@{#508765}
[modify] https://crrev.com/55090272269b587b30cc3eb277184644c4b16088/mojo/edk/system/mach_port_relay.cc
[modify] https://crrev.com/55090272269b587b30cc3eb277184644c4b16088/mojo/edk/system/mach_port_relay.h
[modify] https://crrev.com/55090272269b587b30cc3eb277184644c4b16088/mojo/edk/system/node_channel.cc
[modify] https://crrev.com/55090272269b587b30cc3eb277184644c4b16088/mojo/edk/system/node_controller.cc
[modify] https://crrev.com/55090272269b587b30cc3eb277184644c4b16088/mojo/public/cpp/system/platform_handle.cc