This issue is first seen on M70-70.0.3508.0. But, recently spiked in latest Beta M70-70.0.3538.16. Below link gives in detail of the number of instances in which the crash occurred for associated builds:

https://crash.corp.google.com/browse?q=product_name%3D%27Chrome%27+AND+expanded_custom_data.ChromeCrashProto.ptype%3D%27browser%27+AND+expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BOut+of+Memory%5D+base%3A%3Ainternal%3A%3AMessageLoopTaskRunner%3A%3AAddToIncomingQueue%27#-productname:1000,productversion:100,-magicsignature:50,-magicsignature2:50,-stablesignature:50,-magicsignaturesorted:50

Note:
=====
1. This is Browser crash listed under Beta build 70.0.3538.16 for Windows on 11 different client ID's.
2. Currently this crash is ranked as number #29 with 12 instances.
3. Since there are low instances of crashes, not marking this crash as RB-Stable. Please feel free to edit if this is not the case.

Crash data:
-----------
71.0.3552.3	1.64%	2 - Canary
71.0.3551.3	3.28%	4 - Dev
70.0.3538.16	10.66%	13 - Beta

From the regression range - https://chromium.googlesource.com/chromium/src/+log/69.0.3497.0..70.0.3538.0?pretty=fuller&n=10000
Suspecting the CL - https://chromium.googlesource.com/chromium/src/+/02b1cd6abe987473248a3454a51efc5aefda0637

gab@ -- Could you please check if this is caused with respect to your change, if not please help us in reassigning the issue to the right owner.

Thanks!

Issue 860608 has been merged into this issue.

This is the same as issue 860608 (signature merely changed in my CL).

This is an OOM on an alloc posting into the task queue, not sure we can do much about it.

Looking into some of the crashes to see if this is just the alloc tipping the scale or if the task queue is really unreasonably large (e.g. not being processed for some reason) could be interesting -- I don't plan to do this at the moment.

Just to update the latest behavior of this issue in the latest channels:

Still seeing 1087 crashes from 979 clients so far on latest Stable - 70.0.3538.67 on Windows OS. This crash is ranked as #29 in 'Browser' Stable crashes. 

72.0.3594.0	0.04%	1 - Previous Canary
72.0.3590.0	0.30%	7 - Dev
71.0.3578.20	0.47%	11 - Beta
70.0.3538.77	46.41%	1092 - Stable

Link to the list of builds:
-------------------------
https://crash.corp.google.com/browse?q=product_name%3D%27Chrome%27+AND+expanded_custom_data.ChromeCrashProto.ptype%3D%27browser%27+AND+expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BOut+of+Memory%5D+base%3A%3Ainternal%3A%3AMessageLoopTaskRunner%3A%3AAddToIncomingQueue%27

Thanks!

Just to update the latest behavior of this issue in the latest channels:

Still seeing 37 crashes from 35 clients so far on latest Beta - 72.0.3626.64 on Windows OS. This crash is ranked as #13 in 'Browser' Beta crashes. 

73.0.3679.0	0.00%	1 - Canary
73.0.3673.0	0.07%	24 - Dev
72.0.3626.64	0.11%	38 - Beta
71.0.3578.98	37.97%	13340 - Stable

Crashes on latest Stable spiked than earlier stable builds.

Link to the list of builds:
-------------------------
https://crash.corp.google.com/browse?q=product_name%3D%27Chrome%27+AND+expanded_custom_data.ChromeCrashProto.ptype%3D%27browser%27+AND+expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BOut+of+Memory%5D+base%3A%3Ainternal%3A%3AMessageLoopTaskRunner%3A%3AAddToIncomingQueue%27#-productname:1000,productversion:100,-magicsignature:50,-magicsignature2:50,-stablesignature:50,-magicsignaturesorted:50

Thanks!

Wow, the task queues are humongous!!

1402802 tasks in |incoming_queue_| (multi-threaded incoming queue) and 334371 tasks in |outgoing_queue_| (single-threaded processing queue; reloaded from |incoming_queue_| when empty).

That's a total of 1,737,173 pending tasks on a single thread..!

@etienneb : any chance you can see this in memlog? (we had seen delayed tasks in the past but I don't recall seeing the main task queue being this full)

Above data from http://crash/1b18d2bfa98b87ed

From that crash, it looks like the UI thread is stuck trying to post back to itself and being unable to acquire the lock for its task queue (see next stack for reason).

lock_impl_win.cc:36 :  base::internal::LockImpl::Lock()
message_loop_task_runner.cc:152 :  base::internal::MessageLoopTaskRunner::AddToIncomingQueue(base::Location const &,base::OnceCallback<void >,base::TimeDelta,base::Nestable)
message_loop_task_runner.cc:55 : base::internal::MessageLoopTaskRunner::PostDelayedTask(base::Location const &,base::OnceCallback<void >,base::TimeDelta)
navigation_monitor_impl.cc:65 :  download::NavigationMonitorImpl::NotifyNavigationFinished()
navigation_monitor_impl.cc:56 :  download::NavigationMonitorImpl::OnNavigationEvent(download::NavigationEvent)
download_navigation_observer.cc:36 : download::DownloadNavigationObserver::DidStopLoading()
web_contents_impl.cc:4977 :  content::WebContentsImpl::LoadingStateChanged(bool,bool,content::LoadNotificationDetails *)
web_contents_impl.cc:5628 :  content::WebContentsImpl::DidStopLoading()
frame_tree_node.cc:475 : content::FrameTreeNode::DidStopLoading()
render_frame_host_impl.cc:3180 : content::RenderFrameHostImpl::OnDidStopLoading()
ipc_message_templates.h:146 :  IPC::MessageT<FrameHostMsg_DidStopLoading_Meta,std::tuple<>,void>::Dispatch<content::RenderFrameHostImpl,content::RenderFrameHostImpl,void,void (content::RenderFrameHostImpl::*)() __attribute__((thiscall))>
render_frame_host_impl.cc:1271 : content::RenderFrameHostImpl::OnMessageReceived(IPC::Message const &)
render_process_host_impl.cc:3373 : content::RenderProcessHostImpl::OnMessageReceived(IPC::Message const &)
ipc_channel_proxy.cc:320 : IPC::ChannelProxy::Context::OnDispatchMessage(IPC::Message const &)
bind_internal.h:658 :  base::internal::Invoker<base::internal::BindState<base::internal::IgnoreResultHelper<bool (content::BrowserMessageFilter::Internal::*)(const IPC::Message &) __attribute__((thiscall))>,scoped_refptr<content::BrowserMessageFilter::Internal>,IPC::Message>,void ()>::RunOnce
task_annotator.cc:99 : base::debug::TaskAnnotator::RunTask(char const *,base::PendingTask *)
message_loop.cc:434 :  base::MessageLoop::RunTask(base::PendingTask *)
message_loop.cc:445 :  base::MessageLoop::DeferOrRunPendingTask(base::PendingTask)
message_loop.cc:517 :  base::MessageLoop::DoWork()
message_pump_win.cc:179 :  base::MessagePumpForUI::DoRunLoop()
message_pump_win.cc:52 : base::MessagePumpWin::Run(base::MessagePump::Delegate *)
message_loop.cc:386 :  base::MessageLoop::Run(bool)
run_loop.cc:102 :  base::RunLoop::Run()
chrome_browser_main.cc:1903 :  ChromeBrowserMainParts::MainMessageLoopRun(int *)
browser_main_loop.cc:998 : content::BrowserMainLoop::RunMainMessageLoopParts()
browser_main_runner_impl.cc:165 :  content::BrowserMainRunnerImpl::Run()
browser_main.cc:47 : content::BrowserMain(content::MainFunctionParams const &)
content_main_runner_impl.cc:535 :  content::RunBrowserProcessMain(content::MainFunctionParams const &,content::ContentMainDelegate *)
content_main_runner_impl.cc:900 :  content::ContentMainRunnerImpl::Run(bool)
content_service_manager_main_delegate.cc:53 :  content::ContentServiceManagerMainDelegate::RunEmbedderProcess()
main.cc:472 :  service_manager::Main(service_manager::MainParams const &)
content_main.cc:19 : content::ContentMain(content::ContentMainParams const &)
chrome_main.cc:102 : ChromeMain

Meanwhile the IO thread owns the UI thread's task lock while flushing messages via mojo::Connector::ReadAllAvailableMessages() (22569/35646 reports have mojo as the poster) :
(seems like this is the core issue)

memory_win.cc:51 : base::`anonymous namespace'::OnNoMemory
winheap_stubs_win.cc:90 :  base::allocator::WinCallNewHandler(unsigned int)
allocator_shim_override_ucrt_symbols_win.h:53 :  malloc
circular_deque.h:960 : base::circular_deque<base::PendingTask>::ExpandCapacityIfNecessary(unsigned int)
message_loop_task_runner.cc:162 :  base::internal::MessageLoopTaskRunner::AddToIncomingQueue(base::Location const &,base::OnceCallback<void >,base::TimeDelta,base::Nestable)
message_loop_task_runner.cc:55 : base::internal::MessageLoopTaskRunner::PostDelayedTask(base::Location const &,base::OnceCallback<void >,base::TimeDelta)
task_runner.cc:44 :  base::TaskRunner::PostTask(base::Location const &,base::OnceCallback<void >)
ipc_mojo_bootstrap.cc:838 :  IPC::`anonymous namespace'::ChannelAssociatedGroupController::Accept
connector.cc:475 : mojo::Connector::ReadSingleMessage(unsigned int *)
connector.cc:505 : mojo::Connector::ReadAllAvailableMessages()
connector.cc:387 : mojo::Connector::OnHandleReadyInternal(unsigned int)
bind_internal.h:658 :  base::internal::Invoker<base::internal::BindState<bool (DownloadsListTracker::*)(const download::DownloadItem &) __attribute__((thiscall)) const,base::internal::UnretainedWrapper<DownloadsListTracker> >,bool (const download::DownloadItem &)>::Run
simple_watcher.h:194 : favicon::FaviconService::FaviconResultsCallbackRunner(base::RepeatingCallback<void > const &,std::vector<favicon_base::FaviconRawBitmapResult,std::allocator<favicon_base::FaviconRawBitmapResult> > const *)
bind_internal.h:671 :  base::internal::Invoker<base::internal::BindState<(anonymous namespace)::ComparisonType (*)(const base::RepeatingCallback<bool (const download::DownloadItem &)> &, const download::DownloadItem &, const download::DownloadItem &),base::RepeatingCallback<bool (const download::DownloadItem &)> >,(anonymous namespace)::ComparisonType (const download::DownloadItem &, const download::DownloadItem &)>::Run
simple_watcher.cc:273 :  mojo::SimpleWatcher::OnHandleReady(int,unsigned int,mojo::HandleSignalsState const &)
bind_internal.h:671 :  base::internal::Invoker<base::internal::BindState<void (mojo::SimpleWatcher::*)(int, unsigned int, const mojo::HandleSignalsState &) __attribute__((thiscall)),base::WeakPtr<mojo::SimpleWatcher>,int,unsigned int,mojo::HandleSignalsState>,void ()>::Run
task_annotator.cc:99 : base::debug::TaskAnnotator::RunTask(char const *,base::PendingTask *)
message_loop.cc:434 :  base::MessageLoop::RunTask(base::PendingTask *)
message_loop.cc:445 :  base::MessageLoop::DeferOrRunPendingTask(base::PendingTask)
message_loop.cc:517 :  base::MessageLoop::DoWork()
message_pump_win.cc:512 :  base::MessagePumpForIO::DoRunLoop()
message_pump_win.cc:52 : base::MessagePumpWin::Run(base::MessagePump::Delegate *)
message_loop.cc:386 :  base::MessageLoop::Run(bool)
run_loop.cc:102 :  base::RunLoop::Run()
thread.cc:262 :  base::Thread::Run(base::RunLoop *)
browser_process_sub_thread.cc:174 :  content::BrowserProcessSubThread::IOThreadRun(base::RunLoop *)
thread.cc:357 :  base::Thread::ThreadMain()
platform_thread_win.cc:100 : base::`anonymous namespace'::ThreadFunc

And there's also the history thread being unable to post to the UI thread (probably just a symptom) :

lock_impl_win.cc:36 :  base::internal::LockImpl::Lock()
message_loop_task_runner.cc:152 :  base::internal::MessageLoopTaskRunner::AddToIncomingQueue(base::Location const &,base::OnceCallback<void >,base::TimeDelta,base::Nestable)
message_loop_task_runner.cc:55 : base::internal::MessageLoopTaskRunner::PostDelayedTask(base::Location const &,base::OnceCallback<void >,base::TimeDelta)
task_runner.cc:44 :  base::TaskRunner::PostTask(base::Location const &,base::OnceCallback<void >)
history_backend.cc:163 : history::QueuedHistoryDBTask::DoneRun()
history_backend.cc:2437 :  history::HistoryBackend::ProcessDBTaskImpl()
history_backend.cc:2653 :  history::HistoryBackend::ProcessDBTask(std::unique_ptr<history::HistoryDBTask,std::default_delete<history::HistoryDBTask> >,scoped_refptr<base::SingleThreadTaskRunner>,base::RepeatingCallback<bool > const &)
bind_internal.h:516 :  base::internal::FunctorTraits<void (history::HistoryBackend::*)(std::unique_ptr<history::HistoryDBTask,std::default_delete<history::HistoryDBTask> >, scoped_refptr<base::SingleThreadTaskRunner>, const base::RepeatingCallback<bool ()> &) __attribute__((thiscall)),void>::Invoke<void (history::HistoryBackend::*)(std::unique_ptr<history::HistoryDBTask,std::default_delete<history::HistoryDBTask> >, scoped_refptr<base::SingleThreadTaskRunner>, const base::RepeatingCallback<bool ()> &) __attribute__((thiscall)),scoped_refptr<history::HistoryBackend>,std::unique_ptr<history::HistoryDBTask,std::default_delete<history::HistoryDBTask> >,scoped_refptr<base::SingleThreadTaskRunner>,base::RepeatingCallback<bool ()> >
bind_internal.h:658 :  base::internal::Invoker<base::internal::BindState<void (history::HistoryBackend::*)(std::unique_ptr<history::HistoryDBTask,std::default_delete<history::HistoryDBTask> >, scoped_refptr<base::SingleThreadTaskRunner>, const base::RepeatingCallback<bool ()> &) __attribute__((thiscall)),scoped_refptr<history::HistoryBackend>,std::unique_ptr<history::HistoryDBTask,std::default_delete<history::HistoryDBTask> >,scoped_refptr<base::SingleThreadTaskRunner>,base::RepeatingCallback<bool ()> >,void ()>::RunOnce
task_annotator.cc:99 : base::debug::TaskAnnotator::RunTask(char const *,base::PendingTask *)
message_loop.cc:434 :  base::MessageLoop::RunTask(base::PendingTask *)
message_loop.cc:445 :  base::MessageLoop::DeferOrRunPendingTask(base::PendingTask)
message_loop.cc:517 :  base::MessageLoop::DoWork()
message_pump_default.cc:37 : base::MessagePumpDefault::Run(base::MessagePump::Delegate *)
message_loop.cc:386 :  base::MessageLoop::Run(bool)
run_loop.cc:102 :  base::RunLoop::Run()
thread.cc:262 :  base::Thread::Run(base::RunLoop *)
thread.cc:357 :  base::Thread::ThreadMain()

Another interesting crash is http://crash/9cc52053ebdb8c9c in this case the IO thread is hung with 2 tasks in |outgoing_queue_| and 3,424,806 tasks in |incoming_queue_|.

The IO thread is hung in a system call to "closesocket", there aren't many instances of this particular hang and it might be a manifestation of known Windows issue with KB4338818 https://forum.filezilla-project.org/viewtopic.php?t=49308.

So basically we've just discovered a new class of bugs currently all bucketed under this single signature :

"Thread hangs which manifest themselves as OOMs before being caught by a hang watcher"

I think we need to catch those (e.g. DCHECK tasks queue < 1M task when posting) and have server-side smarts to triage those based on the hung thread's signature like we triage hangs.