I would be very surprised to learn that this is caused by raciness in Mojo, but surely it's possible.

Can you repro locally at all? We have similar crash reports in the wild but the only reasonable interpretation we have so far is that some code is likely using bindings objects incorrectly (e.g. from multiple threads despite not being thread-safe)

While I don't have an explanation for why the SetProperty (or whatever) stack frame might be missed, I am doubtful that this is actually different from the bug captured by  issue 749804  comment #2; i.e. I think it's probably a crash inside the CoordinatorUnitImpl rather than a UAF on the CoordinatorUnitImpl instance itself.

Let's get the lifetime issues fixed there and see if these crashes disappear from the CQ.

Okay. I can reproduce this semi-reliably. The fun news: There's actually a slew of related errors.

Repro steps:
Patch in https://chromium-review.googlesource.com/c/578482. Build browser_tests on Windows with 
"""
use_goma=true
symbol_level=1
is_component_build=true
is_debug = true
is_win_fastlink = true
"""
Run browser_tests. I get a *ton* of different errors.


"""
[3648:3652:0720/140159.640:FATAL:kill_win.cc(46)] GetExitCodeProcess() failed: The handle is invalid. (0x6)

Backtrace:

base::debug::StackTrace::StackTrace [0x024C21A7+55]

base::debug::StackTrace::StackTrace [0x0248434A+10]

logging::Win32ErrorLogMessage::~Win32ErrorLogMessage [0x0243ABCA+106]

base::GetTerminationStatus [0x024AFED1+113]

content::internal::ChildProcessLauncherHelper::GetTerminationStatus [0x01831740+20]

content::ChildProcessLauncher::GetChildTerminationStatus [0x01830623+128]

content::RenderProcessHostImpl::ProcessDied [0x019FB947+198]

content::RenderProcessHostImpl::FastShutdownIfPossible [0x019F8982+162]

browser_shutdown::OnShutdownStarting [0x02CB9970+250]

Browser::OnWindowClosing [0x03A622C8+59]

BrowserCloseManager::CloseBrowsers [0x03D5B503+141]

BrowserCloseManager::CheckForDownloadsInProgress [0x03D5B438+28]

BrowserCloseManager::TryToCloseBrowsers [0x03D5B7FC+111]

chrome::CloseAllBrowsers [0x02C859F3+83]

chrome::AttemptExit [0x02C85811+65]

base::MessageLoop::RunTask [0x0244B34E+1374]
"""

"""
3624:3628:0720/140159.749:FATAL:scoped_handle.cc(116)] Check failed: false. 

Backtrace:

base::debug::StackTrace::StackTrace [0x024C21A7+55]

base::debug::StackTrace::StackTrace [0x0248434A+10]

std::_Hash<std::_Umap_traits<void *,`anonymous namespace'::Info,std::_Uhash_compare<void *,`anonymous namespace'::HandleHash,std::equal_to<void *> >,std::allocator<std::pair<void * const,`anonymous namespace'::Info> >,0> >::_Hash<std::_Umap_traits<void *, [0x02493318+280]

base::win::HandleTraits::CloseHandle [0x02493391+33]

base::Process::Close [0x0243BCC7+55]

content::ChildProcessLauncher::GetChildTerminationStatus [0x0183063E+155]

content::RenderProcessHostImpl::ProcessDied [0x019FB947+198]

content::RenderProcessHostImpl::FastShutdownIfPossible [0x019F8982+162]

browser_shutdown::OnShutdownStarting [0x02CB9970+250]

Browser::OnWindowClosing [0x03A622C8+59]

BrowserCloseManager::CloseBrowsers [0x03D5B503+141]

BrowserCloseManager::CheckForDownloadsInProgress [0x03D5B438+28]

BrowserCloseManager::TryToCloseBrowsers [0x03D5B7FC+111]

chrome::CloseAllBrowsers [0x02C859F3+83]

chrome::AttemptExit [0x02C85811+65]

base::MessageLoop::RunTask [0x0244B34E+1374]
"""


"""
4616:4824:0720/140200.732:FATAL:scoped_handle.cc(116)] Check failed: false. 

Backtrace:

base::debug::StackTrace::StackTrace [0x024C21A7+55]

base::debug::StackTrace::StackTrace [0x0248434A+10]

std::_Hash<std::_Umap_traits<void *,`anonymous namespace'::Info,std::_Uhash_compare<void *,`anonymous namespace'::HandleHash,std::equal_to<void *> >,std::allocator<std::pair<void * const,`anonymous namespace'::Info> >,0> >::_Hash<std::_Umap_traits<void *, [0x02493318+280]

base::win::HandleTraits::CloseHandle [0x02493391+33]

base::WaitableEvent::~WaitableEvent [0x0245DDAA+42]

mojo::internal::MultiplexRouter::InterfaceEndpoint::~InterfaceEndpoint [0x02E485C7+318]

mojo::internal::MultiplexRouter::InterfaceEndpoint::`scalar deleting destructor' [0x02E4895B+11]

base::RefCountedThreadSafe<content::WebFileWriterImpl::WriterBridge,base::DefaultRefCountedThreadSafeTraits<content::WebFileWriterImpl::WriterBridge> >::Release [0x048DA21E+31]

mojo::internal::MultiplexRouter::ProcessTasks [0x02E4A440+348]

mojo::internal::MultiplexRouter::OnPipeConnectionError [0x02E49BEA+250]

base::Callback<void __cdecl(void),0,0>::Run [0x00CB8F27+20]

mojo::Connector::HandleError [0x02E4D9EA+182]

mojo::Connector::OnHandleReadyInternal [0x02E4DAAF+115]

base::internal::Invoker<base::internal::BindState<void (__thiscall OneGoogleBarFetcherImpl::*)(net::URLFetcher const *),base::internal::UnretainedWrapper<OneGoogleBarFetcherImpl> >,void __cdecl(net::URLFetcher const *)>::RunOnce [0x018251D1+17]

mojo::SimpleWatcher::DiscardReadyState [0x0327A216+22]

base::internal::Invoker<base::internal::BindState<void (__cdecl*)(GURL const &,PermissionRequestCreator *,base::Callback<void __cdecl(bool),1,1> const &),GURL>,void __cdecl(PermissionRequestCreator *,base::Callback<void __cdecl(bool),1,1> const &)>::Run [0x03804445+21]

mojo::SimpleWatcher::OnHandleReady [0x02E56A9F+191]

mojo::SimpleWatcher::Context::Notify [0x02E56970+143]

mojo::SimpleWatcher::Context::CallNotify [0x02E56677+23]

mojo::edk::WatcherDispatcher::InvokeWatchCallback [0x033F896B+79]

mojo::edk::Watch::InvokeCallback [0x03402BAC+135]

mojo::edk::RequestContext::~RequestContext [0x033F679A+207]

mojo::edk::NodeChannel::OnChannelError [0x03400AF9+217]

mojo::edk::Channel::Create [0x03403AB2+451]

base::MessagePumpForIO::WaitForWork [0x024C7717+199]

base::MessagePumpForIO::DoRunLoop [0x024C6B80+176]

base::MessagePumpWin::Run [0x024C744A+74]
"""

"""
Backtrace:
        RtlAcquireSRWLockExclusive [0x00007FF82187DA73+19]
        base::internal::LockImpl::Lock [0x0000000000DB5D76+38]
        base::Lock::Acquire [0x0000000000B18EBA+26]
        base::AutoLock::AutoLock [0x0000000000B158E8+40]
        base::ThreadCheckerImpl::CalledOnValidThread [0x0000000000E567EB+27]
        resource_coordinator::ResourceCoordinatorInterface::AddChild [0x0000000014ED411E+46]
        ResourceCoordinatorWebContentsObserver::DidFinishNavigation [0x000000014B06138A+234]
        content::WebContentsImpl::DidFinishNavigation [0x000000000A14D6BE+142]
        content::NavigationHandleImpl::~NavigationHandleImpl [0x0000000009275811+513]
        content::NavigationHandleImpl::`vector deleting destructor' [0x0000000009276AF8+104]
        std::default_delete<content::NavigationHandleImpl>::operator() [0x00000000076375FE+62]
        std::unique_ptr<content::NavigationHandleImpl,std::default_delete<content::NavigationHandleImpl> >::reset [0x000000000923E97E+78]
        content::NavigatorImpl::DidNavigate [0x000000000928DDBB+1691]
        content::RenderFrameHostImpl::OnDidCommitProvisionalLoad [0x00000000092FD6EA+3210]
        content::RenderFrameHostImpl::OnMessageReceived [0x00000000092FFE45+2597]
        content::RenderProcessHostImpl::OnMessageReceived [0x000000000814F118+2072]
        IPC::ChannelProxy::Context::OnDispatchMessage [0x0000000006BBFF12+146]
        base::internal::FunctorTraits<void (__cdecl IPC::ChannelProxy::Context::*)(IPC::Message const & __ptr64) __ptr64,void>::Invoke<scoped_refptr<IPC::ChannelProxy::Context> const & __ptr64,IPC::Message const & __ptr64> [0x0000000006BB751A+74]
        base::internal::InvokeHelper<0,void>::MakeItSo<void (__cdecl IPC::ChannelProxy::Context::*const & __ptr64)(IPC::Message const & __ptr64) __ptr64,scoped_refptr<IPC::ChannelProxy::Context> const & __ptr64,IPC::Message const & __ptr64> [0x0000000006BB7D29+105][0727/210535.280:ERROR:process_win.cc(24)] Process::Process: 4294967294

        base::internal::Invoker<base::internal::BindState<void (__cdecl IPC::ChannelProxy::Context::*)(IPC::Message const & __ptr64) __ptr64,scoped_refptr<IPC::ChannelProxy::Context>,IPC::Message>,void __cdecl(void)>::RunImpl<void (__cdecl IPC::ChannelProxy::Cont [0x0000000006BB8243+115]
        base::internal::Invoker<base::internal::BindState<void (__cdecl IPC::ChannelProxy::Context::*)(IPC::Message const & __ptr64) __ptr64,scoped_refptr<IPC::ChannelProxy::Context>,IPC::Message>,void __cdecl(void)>::Run [0x0000000006BC0CE3+51]
        base::Callback<void __cdecl(void),0,0>::Run [0x0000000000B1F7D0+64]
        base::debug::TaskAnnotator::RunTask [0x0000000000BAE83D+765]
        base::MessageLoop::RunTask [0x0000000000C67855+773]
        base::MessageLoop::DeferOrRunPendingTask [0x0000000000C65252+66]
        base::MessageLoop::DoWork [0x0000000000C65B44+420]
        base::MessagePumpForUI::DoRunLoop [0x0000000000C70CA1+97]
        base::MessagePumpWin::Run [0x0000000000C7225D+157]
        base::MessageLoop::Run [0x0000000000C67400+256]
        base::RunLoop::Run [0x0000000000D711AF+255]
        content::RunThisRunLoop [0x000000014487D637+39]
        content::MessageLoopRunner::Run [0x000000014487CE89+217]
        content::WindowedNotificationObserver::Wait [0x000000014487D75A+170]
        content::WaitForLoadStopWithoutSuccessCheck [0x0000000140F2B4D5+117]
        content::WaitForLoadStop [0x0000000140F2B443+19]
        InProcessBrowserTest::PreRunTestOnMainThread [0x00000001442B77EC+124]
        content::BrowserTestBase::ProxyRunTestOnMainThreadLoop [0x000000014488F98A+330]
"""

This last one is interesting! It implies that there are threading issues with the clients of the resource_coordinator interface.

Note that the previous errors I get on almost *every single test run*. Okay, so now I comment out all clients of GRC.

 modified:   chrome/browser/resource_coordinator/resource_coordinator_web_contents_observer.cc
 modified:   chrome/browser/resource_coordinator/tab_manager.cc
 modified:   content/browser/renderer_host/render_process_host_impl.cc

The errors *mostly* go away. Interestingly enough, I now get the original error [opening comment] about once every 300 tests. 

oysteine@ - can you deal with the GRC client threading issues?
rockot: I seem to be able to repro the issue from the opening comment semi-reliably. Let me know if you want help putting the repro together.

Hm. It looks like the following lines in my CL were incorrect:

base::ProcessHandle process_handle = GetHandle();
base::Process(process_handle).Pid()

On Windows, base::Process takes ownership of the handle [causing a double release]. This isn't the case on POSIX. Confusing!

I had copied-pasted this from https://codereview.chromium.org/2926563005, so I guess this logic has never worked!

oysteine@ - that means you're in the clear, for now. Although you still may want to check the threading on your ResourceCoordinator clients, especially in browser_test environments. 

rockot@ - Unfortunately, we're still in the swamps. The initial error was from macOS, and I was able to encounter this error on Windows even with the logic in question commented out.

everything from #3 looks like the result of handles being closed incorrectly out from under random parts of the system.

to what error and what commented-out logic are you referring in your very last statement?

ps#28: https://chromium-review.googlesource.com/c/578482/28
Fixes the double-close handle. I built browser_tests and let it run for 1500 tests. There was exactly 1 error:

"""
[ RUN      ] FirstRunMasterPrefsImportBookmarksFile.ImportBookmarksFile
[27380:24868:0727/230950.330:ERROR:direct_composition_surface_win.cc(1039)] Failing to detect HDR, couldn't retrieve D3D11 device from ANGLE.
Backtrace:
        resource_coordinator::CoordinationUnitImpl::SetProperty [0x000000000A0E6413+259]
        resource_coordinator::mojom::CoordinationUnitStubDispatch::Accept [0x00000000144B8120+3472]
        resource_coordinator::mojom::CoordinationUnitStub<mojo::RawPtrImplRefTraits<resource_coordinator::mojom::CoordinationUnit> >::Accept [0x000000000A0E2B06+70]
        mojo::InterfaceEndpointClient::HandleValidatedMessage [0x0000000006342756+1670]
        mojo::InterfaceEndpointClient::HandleIncomingMessageThunk::Accept [0x0000000006340821+33]
        mojo::FilterChain::Accept [0x000000000633391F+399]
        mojo::InterfaceEndpointClient::HandleIncomingMessage [0x000000000634208E+222]
        mojo::internal::MultiplexRouter::ProcessIncomingMessage [0x000000000636A7FA+1354]
        mojo::internal::MultiplexRouter::Accept [0x00000000063667F2+594]
        mojo::FilterChain::Accept [0x000000000633391F+399]
        mojo::Connector::ReadSingleMessage [0x0000000006328B30+848]
        mojo::Connector::ReadAllAvailableMessages [0x0000000006328478+88]
        mojo::Connector::OnHandleReadyInternal [0x0000000006327A71+241]
        mojo::Connector::OnWatcherHandleReady [0x0000000006327C8B+27]
        base::internal::FunctorTraits<void (__cdecl mojo::Connector::*)(unsigned int) __ptr64,void>::Invoke<mojo::Connector * __ptr64,unsigned int> [0x0000000006323C5B+43]
        base::internal::InvokeHelper<0,void>::MakeItSo<void (__cdecl mojo::Connector::*const & __ptr64)(unsigned int) __ptr64,mojo::Connector * __ptr64,unsigned int> [0x0000000006324493+83]
        base::internal::Invoker<base::internal::BindState<void (__cdecl mojo::Connector::*)(unsigned int) __ptr64,base::internal::UnretainedWrapper<mojo::Connector> >,void __cdecl(unsigned int)>::RunImpl<void (__cdecl mojo::Connector::*const & __ptr64)(unsigned i [0x0000000006324515+101]
        base::internal::Invoker<base::internal::BindState<void (__cdecl mojo::Connector::*)(unsigned int) __ptr64,base::internal::UnretainedWrapper<mojo::Connector> >,void __cdecl(unsigned int)>::Run [0x0000000006329012+82]
        base::Callback<void __cdecl(unsigned int),1,1>::Run [0x00000000063FD0D5+69]
        mojo::SimpleWatcher::DiscardReadyState [0x00000000063FCE80+32]
        base::internal::FunctorTraits<void (__cdecl*)(base::Callback<void __cdecl(unsigned int),1,1> const & __ptr64,unsigned int,mojo::HandleSignalsState const & __ptr64),void>::Invoke<base::Callback<void __cdecl(unsigned int),1,1> const & __ptr64,unsigned int,m [0x00000000063FBEB6+86]
        base::internal::InvokeHelper<0,void>::MakeItSo<void (__cdecl*const & __ptr64)(base::Callback<void __cdecl(unsigned int),1,1> const & __ptr64,unsigned int,mojo::HandleSignalsState const & __ptr64),base::Callback<void __cdecl(unsigned int),1,1> const & __pt [0x00000000063FC02F+111]
        base::internal::Invoker<base::internal::BindState<void (__cdecl*)(base::Callback<void __cdecl(unsigned int),1,1> const & __ptr64,unsigned int,mojo::HandleSignalsState const & __ptr64),base::Callback<void __cdecl(unsigned int),1,1> >,void __cdecl(unsigned  [0x00000000063FC17F+127]
        base::internal::Invoker<base::internal::BindState<void (__cdecl*)(base::Callback<void __cdecl(unsigned int),1,1> const & __ptr64,unsigned int,mojo::HandleSignalsState const & __ptr64),base::Callback<void __cdecl(unsigned int),1,1> >,void __cdecl(unsigned  [0x00000000063FD160+112]
        base::Callback<void __cdecl(unsigned int,mojo::HandleSignalsState const & __ptr64),1,1>::Run [0x00000000064068F1+97]
        mojo::SimpleWatcher::OnHandleReady [0x000000000640665C+396]
        base::internal::FunctorTraits<void (__cdecl mojo::SimpleWatcher::*)(int,unsigned int,mojo::HandleSignalsState const & __ptr64) __ptr64,void>::Invoke<base::WeakPtr<mojo::SimpleWatcher> const & __ptr64,int const & __ptr64,unsigned int const & __ptr64,mojo:: [0x00000000064038FD+109]
        base::internal::InvokeHelper<1,void>::MakeItSo<void (__cdecl mojo::SimpleWatcher::*const & __ptr64)(int,unsigned int,mojo::HandleSignalsState const & __ptr64) __ptr64,base::WeakPtr<mojo::SimpleWatcher> const & __ptr64,int const & __ptr64,unsigned int cons [0x0000000006403BBE+158]
        base::internal::Invoker<base::internal::BindState<void (__cdecl mojo::SimpleWatcher::*)(int,unsigned int,mojo::HandleSignalsState const & __ptr64) __ptr64,base::WeakPtr<mojo::SimpleWatcher>,int,unsigned int,mojo::HandleSignalsState>,void __cdecl(void)>::R [0x0000000006403CB3+195]
        base::internal::Invoker<base::internal::BindState<void (__cdecl mojo::SimpleWatcher::*)(int,unsigned int,mojo::HandleSignalsState const & __ptr64) __ptr64,base::WeakPtr<mojo::SimpleWatcher>,int,unsigned int,mojo::HandleSignalsState>,void __cdecl(void)>::R [0x0000000006406943+51]
        base::Callback<void __cdecl(void),0,0>::Run [0x000000018004F7D0+64]
        base::debug::TaskAnnotator::RunTask [0x00000001800DE83D+765]
        base::MessageLoop::RunTask [0x0000000180197855+773]
        base::MessageLoop::DeferOrRunPendingTask [0x0000000180195252+66]
        base::MessageLoop::DoWork [0x0000000180195B44+420]
        base::MessagePumpForIO::DoRunLoop [0x00000001801A0AC7+39]
        base::MessagePumpWin::Run [0x00000001801A225D+157]
        base::MessageLoop::Run [0x0000000180197400+256]
        base::RunLoop::Run [0x00000001802A0D1F+255]
        base::Thread::Run [0x0000000180384323+355]
        content::BrowserThreadImpl::IOThreadRun [0x00000000085C63AF+47]
        content::BrowserThreadImpl::Run [0x00000000085C88CA+522]
        base::Thread::ThreadMain [0x0000000180385B87+1239]
        base::PlatformThread::Sleep [0x000000018034EDC1+513]
        BaseThreadInitThunk [0x00007FF81FB48102+34]
        RtlUserThreadStart [0x00007FF8218AC5B4+52]
"""

Naturally, the error is not deterministic. Note that this is the same error I originally posted about occurring on a macOS CQ trybot. This was also the error that you indicated should *not* be caused by client threading issues, I believe?

I just ran browser_tests for another 800 tests or so. Was able to reproduce the same error again! So I believe I now have a probabilistic repro. :)

I am using ToT:  a225c0882644b4f81d48638fcbe5fa0045772c67
With the following CL patched in: https://chromium-review.googlesource.com/c/578482/28

gn args:
"""
use_goma=true
symbol_level=1
is_component_build=true
is_debug = true
is_win_fastlink = true
"""

binary: browser_tests

Right that's not a threading issue, that's (AFAICT) a UAF within the
SetProperty impl; and I'd expect it to be fixed by bounding all unit impl
lifetimes to that of the service rather than to individual pipes.
Presumably the unit impl is making invalid assumptions about how long other
things live.

Uh, duh. Of course. That's what I get for making posts at odd hours of day.