Adding more info from https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20product.version%3D%2750.0.2661.0%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20OMIT%20RECORD%20IF%20SUM(CrashedStackTrace.StackFrame.SourceFileName%3D%27c%3A%5C%5Cb%5C%5Cbuild%5C%5Cslave%5C%5Cwin%5C%5Cbuild%5C%5Csrc%5C%5Cbase%5C%5Cwin%5C%5Cscoped_handle.cc%27)%20%3D%200&ignore_case=false&stbtiq=&reportid=&index=0#0

Stack Trace
==========
Thread 0 CRASHED [EXCEPTION_BREAKPOINT @ 0x00cefe76 ] MAGIC SIGNATURE THREAD
0x00cefe76	(chrome.exe -scoped_handle.cc:210 )	`anonymous namespace'::ActiveVerifier::OnHandleBeingClosed(void *)
0x5ec44f8e	(chrome_child.dll -close_handle_hook_win.cc:41 )	`anonymous namespace'::CloseHandleHook(void *)
0x75e55096	(SHLWAPI.dll + 0x00015096 )	_ProcessDetach
0x75e55057	(SHLWAPI.dll + 0x00015057 )	EtwUnregisterTraceGuids
0x75e5a2d7	(SHLWAPI.dll + 0x0001a2d7 )	_DllMainCRTStartup
0x7779af23	(ntdll.dll + 0x0005af23 )	LdrpCallInitRoutine
0x777a3851	(ntdll.dll + 0x00063851 )	RtlExitUserProcess
0x777a37c4	(ntdll.dll + 0x000637c4 )	RtlExitUserProcess
0x75c72ae3	(KERNEL32.dll + 0x00052ae3 )	ExitProcessStub
0x00d3865a	(chrome.exe -crt0dat.c:774 )	__crtExitProcess

git CL range: https://chromium.googlesource.com/chromium/src/+log/50.0.2660.3..50.0.2661.0?pretty=fuller&n=10000

* Suspecting https://chromium.googlesource.com/chromium/src/+/5556e56290c41891b0830af9b6b9153b745dd226 [base/win/scoped_handle.cc], 

wfh@ can you pls take a look? Thanks.

I see some coming through UnmapViewOfFile e.g. crash/eee2b8d400000000. These contain no pc info for the original creator.

0:000> kv
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr  Args to Child              
(Inline) -------- -------- -------- -------- chrome!base::debug::BreakDebugger+0x9 (Inline Function @ 0134fe76) (CONV: cdecl) [c:\b\build\slave\win\build\src\base\debug\debugger_win.cc @ 21]
*** WARNING: Unable to verify checksum for KERNELBASE.dll
0030fadc 757d6c11 00877f50 0030faf8 60df4f8f chrome!`anonymous namespace'::ActiveVerifier::OnHandleBeingClosed+0x66 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\build\slave\win\build\src\base\win\scoped_handle.cc @ 210]
*** WARNING: Unable to verify checksum for SHLWAPI.dll
0030faf8 774b4b8f 0000018c 00000001 00000000 KERNELBASE!UnmapViewOfFile+0x14 (FPO: [Non-Fpo])
0030fb0c 774b4b50 00000001 0030fb78 774b9c34 SHLWAPI!_ProcessDetach+0x35 (FPO: [Non-Fpo])
0030fb18 774b9c34 774a0000 00000000 00000001 SHLWAPI!DllMain+0x28 (FPO: [Non-Fpo])
0030fb78 776089d8 774a0000 00000000 00000001 SHLWAPI!_CRT_INIT+0x26d (FPO: [Non-Fpo])
0030fb98 7760e104 774b9ba6 774a0000 00000000 ntdll!LdrpCallInitRoutine+0x14
0030fc3c 7760e19f 00a97834 7760cd10 00a97838 ntdll!LdrShutdownProcess+0x1aa (FPO: [Non-Fpo])
*** WARNING: Unable to verify checksum for kernel32.dll
0030fc50 77412163 00000000 77e8f3b0 ffffffff ntdll!RtlExitUserProcess+0x74 (FPO: [Non-Fpo])
0030fc64 0139865b 00000000 0030fcb8 013988ea kernel32!ExitProcessStub+0x12 (FPO: [1,0,0])
0030fc70 013988e9 00000000 f8c77643 00000000 chrome!__crtExitProcess+0x15 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt\crtw32\startup\crt0dat.c @ 774]
0030fcb8 0139890e 00000000 00000000 00000000 chrome!doexit+0x119 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt\crtw32\startup\crt0dat.c @ 678]
0030fccc 01396fb9 00000000 f8c777f7 00000000 chrome!exit+0xf (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt\crtw32\startup\crt0dat.c @ 417]
0030fd0c 77413c45 7ffda000 0030fd58 776137f5 chrome!__tmainCRTStartup+0x10c (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt\crtw32\startup\crt0.c @ 264]
0030fd18 776137f5 7ffda000 77d91bd1 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0030fd58 776137c8 01397024 7ffda000 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0030fd70 00000000 01397024 7ffda000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])
0:000> dt -r other
Local var @ 0x30fac0 Type `anonymous-namespace'::Info
   +0x000 owner            : 0x7740db13 Void
   +0x004 pc1              : 0x60df4f6e Void
   +0x008 pc2              : (null) 
   +0x00c thread_id        : 0x3b5cc08
0:000> ln 0x60df4f6e
*** WARNING: Unable to verify checksum for chrome_child.dll
c:\b\build\slave\win\build\src\base\debug\close_handle_hook_win.cc(40)
SrcSrv Command: cmd /c "mkdir "C:\ProgramData\dbg\src\base\debug\close_handle_hook_win.cc\ef6f6ae5e4c96622286b563658d5cd62a6cf1197" & python -c "import urllib2, base64;url = \"https://chromium.googlesource.com/chromium/src.git/+/ef6f6ae5e4c96622286b563658d5cd62a6cf1197/base/debug/close_handle_hook_win.cc?format=TEXT\";u = urllib2.urlopen(url);open(r\"C:\ProgramData\dbg\src\base\debug\close_handle_hook_win.cc\ef6f6ae5e4c96622286b563658d5cd62a6cf1197\close_handle_hook_win.cc\", \"wb\").write(base64.b64decode(u.read(
(60df4f6e)   chrome_child!`anonymous namespace'::CloseHandleHook   |  (60df4f9d)   chrome_child!`anonymous namespace'::DuplicateHandleHook
Exact matches:
    chrome_child!`anonymous namespace'::CloseHandleHook (void *)

Others coming through a thread leaking:

0:000> kv
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr  Args to Child              
(Inline) -------- -------- -------- -------- chrome!base::debug::BreakDebugger+0x9 (Inline Function @ 0109fe76) (CONV: cdecl) [c:\b\build\slave\win\build\src\base\debug\debugger_win.cc @ 21]
*** WARNING: Unable to verify checksum for chrome_child.dll
001cf8a4 52f64f8f 00000190 00000000 001cf8c8 chrome!`anonymous namespace'::ActiveVerifier::OnHandleBeingClosed+0x66 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\build\slave\win\build\src\base\win\scoped_handle.cc @ 210]
*** WARNING: Unable to verify checksum for SHLWAPI.dll
(Inline) -------- -------- -------- -------- chrome_child!base::win::OnHandleBeingClosed+0x1d (Inline Function @ 52f64f8f) (CONV: cdecl) [c:\b\build\slave\win\build\src\base\win\scoped_handle.cc @ 244]
001cf8b4 75365097 00000190 00000001 00000000 chrome_child!`anonymous namespace'::CloseHandleHook+0x21 (FPO: [Non-Fpo]) (CONV: stdcall) [c:\b\build\slave\win\build\src\base\debug\close_handle_hook_win.cc @ 42]
001cf8c8 75365058 00000001 001cf934 7536a2d8 SHLWAPI!_ProcessDetach+0x35 (FPO: [Non-Fpo])
001cf8d4 7536a2d8 75350000 00000000 00000001 SHLWAPI!DllMain+0x28 (FPO: [Non-Fpo])
001cf934 775daf24 75350000 00000000 00000001 SHLWAPI!_CRT_INIT+0x26d (FPO: [Non-Fpo])
001cf954 775e3852 7536a24a 75350000 00000000 ntdll!LdrpCallInitRoutine+0x14
001cf9f8 775e37c5 00d07604 775df020 00d07608 ntdll!LdrShutdownProcess+0x1aa (FPO: [Non-Fpo])
*** WARNING: Unable to verify checksum for kernel32.dll
001cfa0c 75402b03 00000000 77e8f3b0 ffffffff ntdll!RtlExitUserProcess+0x74 (FPO: [Non-Fpo])
001cfa20 010e865b 00000000 001cfa74 010e88ea kernel32!ExitProcessStub+0x12 (FPO: [1,0,0])
001cfa2c 010e88e9 00000000 4ad26be8 00000000 chrome!__crtExitProcess+0x15 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt\crtw32\startup\crt0dat.c @ 774]
001cfa74 010e890e 00000000 00000000 00000000 chrome!doexit+0x119 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt\crtw32\startup\crt0dat.c @ 678]
001cfa88 010e6fb9 00000000 4ad26b54 00000000 chrome!exit+0xf (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt\crtw32\startup\crt0dat.c @ 417]
001cfac8 75401194 7ffda000 001cfb14 775db3f5 chrome!__tmainCRTStartup+0x10c (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt\crtw32\startup\crt0.c @ 264]
001cfad4 775db3f5 7ffda000 776679e9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
001cfb14 775db3c8 010e7024 7ffda000 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
001cfb2c 00000000 010e7024 7ffda000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])
0:000> ln 0x52914b73
c:\b\build\slave\win\build\src\base\threading\thread.cc(73)+0xd2
SrcSrv Command: cmd /c "mkdir "C:\ProgramData\dbg\src\base\threading\thread.cc\ef6f6ae5e4c96622286b563658d5cd62a6cf1197" & python -c "import urllib2, base64;url = \"https://chromium.googlesource.com/chromium/src.git/+/ef6f6ae5e4c96622286b563658d5cd62a6cf1197/base/threading/thread.cc?format=TEXT\";u = urllib2.urlopen(url);open(r\"C:\ProgramData\dbg\src\base\threading\thread.cc\ef6f6ae5e4c96622286b563658d5cd62a6cf1197\thread.cc\", \"wb\").write(base64.b64decode(u.read()))"
(52914aa1)   chrome_child!base::Thread::Thread+0xd2   |  (52914ba2)   chrome_child!base::Histogram::FactoryGet
0:000> ln 0x52907d7d
c:\b\build\slave\win\build\src\base\win\scoped_handle.h(78)+0x7
SrcSrv Command: cmd /c "mkdir "C:\ProgramData\dbg\src\base\win\scoped_handle.h\ef6f6ae5e4c96622286b563658d5cd62a6cf1197" & python -c "import urllib2, base64;url = \"https://chromium.googlesource.com/chromium/src.git/+/ef6f6ae5e4c96622286b563658d5cd62a6cf1197/base/win/scoped_handle.h?format=TEXT\";u = urllib2.urlopen(url);open(r\"C:\ProgramData\dbg\src\base\win\scoped_handle.h\ef6f6ae5e4c96622286b563658d5cd62a6cf1197\scoped_handle.h\", \"wb\").write(base64.b64decode(u.read()))"
(52907d4c)   chrome_child!base::win::GenericScopedHandle<base::win::HandleTraits,base::win::VerifierTraits>::Set+0x31   |  (52907db8)   chrome_child!base::win::GenericScopedHandle<base::win::HandleTraits,base::win::VerifierTraits>::Close

It's non trivial to work out what this could be. I am analyzing all the crashes now, so far almost all of them appear to be a leaking thread on shutdown, it's not possible to know what is creating the thread but it's still running on shutdown and this is tripping the handle verifier. The code in question would be creating a base::Thread object somehow.

Not quite ready to pick out individual CLs, but maybe one of:

* https://codereview.chromium.org/1439053002
* https://codereview.chromium.org/1713143002
* https://codereview.chromium.org/1736703002

I analyzed 1148 crashes from 50.0.2661.0 matching this magic signature and they fit into three buckets for the pc1/pc2 - but I think they are all the same since they go through SHLWAPI!_ProcessDetach+0x35

In all cases it was an Event that was being closed, the event is created in thread.cc#73 and is called start_event_ - the event is not being closed because the base::Thread object is not being destructed.

All cases break out as follows:

+---+-----------------------------------------------------+------------------------------------------------+-----+
| N | pc1                                                 | pc2                                            | num |
+---+-----------------------------------------------------+------------------------------------------------+-----+
| 1 | chrome_child!base::Thread::Thread+0xd2              | chrome_child!base::win::ScopedHandle::Set+0x31 | 788 |
| 2 | chrome_child!`anonymous namespace'::CloseHandleHook | (null)                                         | 283 |
| 3 | chrome_child!base::Thread::Thread+0x70              | chrome_child!base::win::ScopedHandle::Set+0x31 |  77 |
+---+-----------------------------------------------------+------------------------------------------------+-----+

chrome_child!base::Thread::Thread+0x70 and 0xd2 both resolve to same source line - base\threading\thread.cc(73)

https://chromium.googlesource.com/chromium/src/+/50.0.2661.0/base/threading/thread.cc#73

stack 1 - e.g. crash/0058dc9800000000:

 # ChildEBP RetAddr
00 (Inline) -------- chrome_child!base::debug::BreakDebugger+0x9
01 001cf86c 61e54f8f chrome_child!`anonymous namespace'::ActiveVerifier::OnHandleBeingClosed+0x66
02 (Inline) -------- chrome_child!base::win::OnHandleBeingClosed+0x1d
03 001cf87c 76475097 chrome_child!`anonymous namespace'::CloseHandleHook+0x21
04 001cf890 76475058 SHLWAPI!_ProcessDetach+0x35
05 001cf89c 7647a2d8 SHLWAPI!DllMain+0x28

stack 2 - e.g. crash/00c2ad7000000000:

 # ChildEBP RetAddr
00 (Inline) -------- chrome!base::debug::BreakDebugger+0x9
01 0013f8dc 75886c11 chrome!`anonymous namespace'::ActiveVerifier::OnHandleBeingClosed+0x66
02 0013f8f8 77664b8f KERNELBASE!UnmapViewOfFile+0x14
03 0013f90c 77664b50 SHLWAPI!_ProcessDetach+0x35
04 0013f918 77669c34 SHLWAPI!DllMain+0x28

stack 3 - e.g. crash/01c45c9800000000

 # ChildEBP RetAddr
00 (Inline) -------- chrome!base::debug::BreakDebugger+0x9
01 002af87c 5fc64f8f chrome!`anonymous namespace'::ActiveVerifier::OnHandleBeingClosed+0x66
02 (Inline) -------- chrome_child!base::win::OnHandleBeingClosed+0x1d
03 002af88c 77b84b8f chrome_child!`anonymous namespace'::CloseHandleHook+0x21
04 002af8a0 77b84b50 SHLWAPI!_ProcessDetach+0x35
05 002af8ac 77b89c34 SHLWAPI!DllMain+0x28

raw data can be found here: https://docs.google.com/spreadsheets/d/1-9XgN0kk-wGmX22EG_JbY1NAv75bCmaqnCpGtHPrrOM/edit

Just to update, this crash has more than 600 Instances reported so far on Build 51.0.2663.0 on Windows . below link gives in detail about the same.

https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20product.version%3D%2751.0.2663.0%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%60anonymous%20namespace%5C%27%3A%3AActiveVerifier%3A%3AOnHandleBeingClosed%27&ignore_case=false

Will, would it better to revert the CL while getting it further investigated? 

There is no candidate CL to revert. I need more people to look at their CLs in the range 50.0.2660.0 - 50.0.2661.0 to see if any changes have been made to thread lifetime.

Marking the bug ad M50 dev blocker since the spike started since 50.0.2661.0 and we are planning a Dev release tomorrow.

Here is the crash rate. Regressed after - 50.0.2658.0 and introduced renderer & extension crash.

Renderer - 44.50% with 1028 instances
Extension - 96.61% with 2280 instances.

51.0.2663.1	0.03%	375	
51.0.2663.0	0.15%	1777	
51.0.2662.1	0.11%	1280	
51.0.2662.0	0.42%	4954	
50.0.2661.1	0.09%	1028	
50.0.2661.0	0.38%	4491	
50.0.2658.0	0.00%	1	
50.0.2657.0	0.00%	2	
50.0.2656.0	0.00%	3	

CL
==
https://chromium.googlesource.com/chromium/src/+log/50.0.2658.0..50.0.2661.0?pretty=fuller&n=10000

Manually checked the CL, hard to pinpoint any suspects, wondering whether the below mentioned CLs have any impact.

https://chromium.googlesource.com/chromium/src/+/20232c23c12a81d98b69d10c9e18c0b4292606cb

https://chromium.googlesource.com/chromium/src/+/1cdbf437f9f45896ccff18c834c7e8b7bc3e8061

Assigning to stability sheriff for more updates. We need to resolve this crash by eod as we planning a Dev release tomorrow.

I've already trawled the memory of the crash dumps looking for base::Thread structures with start_event_ matching the handle that is being closed on shutdown and not managed to find the thread name. I've done all that I can do without either:

1. More people looking at their CLs in the range to see if they have made any changes to thread lifetime
2. Adding more diagnostic code in handle verifier to try and walk up the creation stack further, to see what caller is creating the scopedhandle.

I am investigating 2. but it is probably a few days at least away. This bug feel a bit lonely with just me and the release TPMs commenting on it. It would be nice if other Chromium developers also helped here.

Thanks Will for the update. 

I talked to stability sheriff of the day he will also help in investigating.

FYI - the regression range in #10 is too wide, a crasher of this magnitude would appear immediately in a new version so the range is just 2660 - 2661 which is

https://chromium.googlesource.com/chromium/src/+log/50.0.2660.0..50.0.2661.0?pretty=fuller&n=10000

for stability sheriff, it's worth reading through other handle verifier regressions such as  issue 574325  and  issue 559238  for important background, before diving into this bug.

I'll take a look at this too.

On Mon, Feb 29, 2016 at 12:22 PM, wfh@chromium.org via Monorail <
monorail@chromium.org> wrote:

Thanks guys. Pls get a fix in by 5pm today for the first M50 branched dev push.

Just to make sure I understand, since my head isn't quite yet wrapped around the handle hooks we have: this is definitely an Event handle, and this particular CHECK indicates a leak rather than a double-close?

this particular CHECK is because on Windows 7 during DllMain DLL_PROCESS_DETACH of SHLWAPI.dll it will attempt to close any remaining open event handles. Since handle verifier has hooked CloseHandle this trips the handle verifier.

The event handle in question was originally created as part of a base::Thread, and hasn't been closed previously because the base::Thread has presumably not been destructed at process shutdown (or at least, this is what happened in previous cases of this spiking).

And thanks for looking into this also rockot@. It would be good if the authors of the CLs I had tentatively identified as candidates in #4 could take a look and confirm that their CL does not introduce any potential for leaving a base::Thread running during shutdown.

Probably my fault: https://codereview.chromium.org/1748093002

Thanks for finding it Ken.

On a side note, I'm wondering if it would be worth to add more information to the verifier. Having a full callstack as opposed to only the last 2 pc would make it a lot easier to track this down.

yes, I am planning on adding more callstacks to the verifier - see #12. This would make it easier to debug in future.

Re #16: We're cutting it close. The fix is in the CQ but likely won't land for another hour or so. Do we have any wiggle room for the exact time of branch?

We are planning to cut M50 dev branch 2661 at 6 pm today.It would be great if you could merge before that.

rockot@, pls have it landed in trunk/master to catch up with tonight's 8pm canary build, once verified then request the merge to M50. Thanks.

I'm moving out tonight's canary build out a bit (to 10pm), to ensure it can get rockot@'s CL in, then we can get some stability data on it tomorrow.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6cb09bb59dd7ed4dcd37e14c32586ea25490f896

commit 6cb09bb59dd7ed4dcd37e14c32586ea25490f896
Author: Ken Rockot <rockot@chromium.org>
Date: Tue Mar 01 04:17:49 2016

Clean up MojoShellConnection in render processes

MojoShellConnectionImpl ultimately owns a RunnerConnection, which in
turn spins up a new thread. If we don't explicitly tear this down then
we leak that thread.

BUG= 590472 
R=ben@chromium.org
TBR=ben@chromium.org

Review URL: https://codereview.chromium.org/1748093002 .

Cr-Commit-Position: refs/heads/master@{#378394}

[modify] https://crrev.com/6cb09bb59dd7ed4dcd37e14c32586ea25490f896/content/renderer/renderer_main.cc
[modify] https://crrev.com/6cb09bb59dd7ed4dcd37e14c32586ea25490f896/mojo/shell/runner/host/child_process_host.cc
[modify] https://crrev.com/6cb09bb59dd7ed4dcd37e14c32586ea25490f896/mojo/shell/runner/host/out_of_process_native_runner.cc

Just to update, this crash is still (8:00 PM IST) observed with more than 190 instances on Canary 51.0.2664.0 on Windows. Below link gives in detail about the same:

https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20product.version%3D%2751.0.2664.0%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%60anonymous%20namespace%5C%27%3A%3AActiveVerifier%3A%3AOnHandleBeingClosed%27&ignore_case=false

It looks like that CL didn't make it on to the Canary channel last night.  If we could please merge that CL to branch 2664, ASAP and kick off a build to verify the fix.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c5003b063fe4daffbc47fabd8573718ec53c4e73

commit c5003b063fe4daffbc47fabd8573718ec53c4e73
Author: Ken Rockot <rockot@chromium.org>
Date: Tue Mar 01 15:38:10 2016

Clean up MojoShellConnection in render processes

MojoShellConnectionImpl ultimately owns a RunnerConnection, which in
turn spins up a new thread. If we don't explicitly tear this down then
we leak that thread.

BUG= 590472 
R=ben@chromium.org
TBR=ben@chromium.org

Review URL: https://codereview.chromium.org/1748093002 .

Cr-Commit-Position: refs/heads/master@{#378394}
(cherry picked from commit 6cb09bb59dd7ed4dcd37e14c32586ea25490f896)

Review URL: https://codereview.chromium.org/1746263003 .

Cr-Commit-Position: refs/branch-heads/2664@{#1}
Cr-Branched-From: bb9c28a9df2e0b7915926372d832452455b3d5a0-refs/heads/master@{#378386}

[modify] https://crrev.com/c5003b063fe4daffbc47fabd8573718ec53c4e73/content/renderer/renderer_main.cc
[modify] https://crrev.com/c5003b063fe4daffbc47fabd8573718ec53c4e73/mojo/shell/runner/host/child_process_host.cc
[modify] https://crrev.com/c5003b063fe4daffbc47fabd8573718ec53c4e73/mojo/shell/runner/host/out_of_process_native_runner.cc

Triggering a build for 2664 now.

Can we please merge this with 2661 as well? ChromeOS needs to dev tomorrow and this change is critical for that release.

I'll merge to 2661 now.

A quick glance seems to indicate that this may indeed have resolved the crash in 51.0.2664.1. Can someone else verify this?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/064c42a1a3010651c4dcbe8e84a62365e7261b8a

commit 064c42a1a3010651c4dcbe8e84a62365e7261b8a
Author: Ken Rockot <rockot@chromium.org>
Date: Tue Mar 01 21:36:05 2016

Clean up MojoShellConnection in render processes

MojoShellConnectionImpl ultimately owns a RunnerConnection, which in
turn spins up a new thread. If we don't explicitly tear this down then
we leak that thread.

BUG= 590472 
R=ben@chromium.org
TBR=ben@chromium.org

Review URL: https://codereview.chromium.org/1748093002 .

Cr-Commit-Position: refs/heads/master@{#378394}
(cherry picked from commit 6cb09bb59dd7ed4dcd37e14c32586ea25490f896)

Review URL: https://codereview.chromium.org/1755843002 .

Cr-Commit-Position: refs/branch-heads/2661@{#34}
Cr-Branched-From: ef6f6ae5e4c96622286b563658d5cd62a6cf1197-refs/heads/master@{#378081}

[modify] https://crrev.com/064c42a1a3010651c4dcbe8e84a62365e7261b8a/content/renderer/renderer_main.cc
[modify] https://crrev.com/064c42a1a3010651c4dcbe8e84a62365e7261b8a/mojo/shell/runner/host/child_process_host.cc
[modify] https://crrev.com/064c42a1a3010651c4dcbe8e84a62365e7261b8a/mojo/shell/runner/host/out_of_process_native_runner.cc

Yes can confirm this crash is gone in 51.0.2664.1. Thanks rockot@

Verified in latest canary -51.0.2664.1

Awesome it's fixed and confirmed gone in 51.0.2664.1, thanks rockot, wfh and everyone!
(rockot@ pls use merge-request label to request merge first after it's verified, and merge after it's approved. Thanks!)

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/bling/chromium.git/+/6cb09bb59dd7ed4dcd37e14c32586ea25490f896

commit 6cb09bb59dd7ed4dcd37e14c32586ea25490f896
Author: Ken Rockot <rockot@chromium.org>
Date: Tue Mar 01 04:17:49 2016