Users experienced this crash on the following builds:

Win Dev 51.0.2704.4 -  0.83 CPM, 175 reports, 33 clients (signature RtlProcessFlsData)
Win Canary 52.0.2708.0 -  76.83 CPM, 1148 reports, 174 clients (signature RtlProcessFlsData)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Seems to be this got spiked in M51#51.0.2707.0 and working on finding a owner.

52.0.2708.0	4.76%	114	
52.0.2707.2	1.17%	28	
52.0.2707.0	5.97%	143	
52.0.2706.0	0.21%	5	
52.0.2705.0	0.50%	12	
51.0.2704.4	0.04%	1	
51.0.2704.0	0.17%	4	

Thank you!

Typo: Spiked in M52#52.0.2707.0.

Seems like Ricardo (rvargas@) is not with Chrome team anymore.

this appears to be Comodo Internet Security. Still investigating.

taking the lock on this bug, until I have further analysis to post.

I analyzed a representative sample of 100 reports with this crash signature of Chrome 52.0.2704.0 and above.

100% of these crash reports have unloaded module guard64.dll on the callstack.

> kn
  *** Stack trace for last set context - .thread/.cxr resets it
 # Child-SP          RetAddr           Call Site
00 00000063`fd17f888 00007ffc`ea6bb509 <Unloaded_guard64.dll>+0x2c7a0
01 00000063`fd17f890 00007ffc`ea6bb23f ntdll!RtlProcessFlsData+0x129
02 00000063`fd17f8e0 00007ffc`ea6bb17a ntdll!LdrShutdownProcess+0x9f
03 00000063`fd17f9f0 00007ffc`e84f4d8a ntdll!RtlExitUserProcess+0xda
04 00000063`fd17fae0 00007ff7`380840d8 KERNEL32!ExitProcessImplementation+0xa
05 00000063`fd17fb10 00007ff7`38084074 chrome!exit_or_terminate_process+0x48
06 00000063`fd17fb40 00007ff7`3807a703 chrome!common_exit+0x150
07 00000063`fd17fb70 00007ffc`e84e8102 chrome!__scrt_common_main_seh+0x137
08 00000063`fd17fbb0 00007ffc`ea6bc5b4 KERNEL32!BaseThreadInitThunk+0x22
09 00000063`fd17fbe0 00000000`00000000 ntdll!RtlUserThreadStart+0x34

guard64.dll is a component of Comodo Internet Security.

What's the follow-up for this, given it seems to be caused by Comodo?

This is still marked as a P0, since it's already triaged, are we going to do anything about this third-party or should close as WontFix?

Adding more windows experts here. This is currently at 10% of our canary renderer crashes.

Uggh. Possible mitigations include TerminateProcess instead of ExitProcess, or incrementing the reference count on guard64.dll so that it doesn't unload.

TerminateProcess is much more tempting. Could also be done only in the case where guard64.dll is noticed.

Could follow-up with Comodo.

I have a similar but not identical crash when I run Chrome with Comodo Internet Security (8, 2, 0, 5005 file version info on the DLL)

4:051> k
Child-SP          RetAddr           Call Site
000000ff`abd4eae0 00007ffa`92db2ad2 ntdll!KiRaiseUserExceptionDispatcher+0x3a
000000ff`abd4ebb0 00007ffa`92db2dd9 guard64!Exported+0x39df2
000000ff`abd4ed50 00007ffa`92db2ea1 guard64!Exported+0x3a0f9
000000ff`abd4ede0 00007ffa`92db1cba guard64!Exported+0x3a1c1
000000ff`abd4ee10 00007ffa`92db2507 guard64!Exported+0x38fda
000000ff`abd4f660 00007ffa`92da6fd4 guard64!Exported+0x39827
000000ff`abd4f690 00007ffa`92da72fd guard64!Exported+0x2e2f4
000000ff`abd4f710 00007ffa`92da639a guard64!Exported+0x2e61d
000000ff`abd4f790 00007ffa`92da6013 guard64!Exported+0x2d6ba
000000ff`abd4f7f0 00007ffa`92d755d2 guard64!Exported+0x2d333
000000ff`abd4f820 00007ffa`92d9b653 guard64+0x55d2
000000ff`abd4f850 00007ffa`95e5c0f4 guard64!Exported+0x22973
000000ff`abd4f890 00007ffa`95e5b502 ntdll!LdrpCallInitRoutine+0x4c
000000ff`abd4f8f0 00007ffa`95e58528 ntdll!LdrShutdownProcess+0x142
000000ff`abd4fa00 00007ffa`938b516a ntdll!RtlExitUserProcess+0x98
000000ff`abd4faf0 00007ff6`30c334d0 KERNEL32!ExitProcessImplementation+0xa
000000ff`abd4fb20 00007ff6`30c3346c chrome!exit_or_terminate_process+0x48 [d:\th\minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 129]
000000ff`abd4fb50 00007ff6`30c29ce3 chrome!common_exit+0x150 [d:\th\minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 269]
000000ff`abd4fb80 00007ffa`938b13d2 chrome!__scrt_common_main_seh+0x137 [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 266]
000000ff`abd4fbc0 00007ffa`95e554e4 KERNEL32!BaseThreadInitThunk+0x22
000000ff`abd4fbf0 00000000`00000000 ntdll!RtlUserThreadStart+0x34

in my ppapi process guard64.dll is not unloaded, but it's possible there is some kind of race happening on shutdown anyway.

Either way, I'm going to try adding guard64.dll to the sandboxed process blacklist and see if it gets any better (or worse).

For reference:

00007ff8`aced0000 00007ff8`acf66000   guard64.dll
    Timestamp: Wed Apr 06 03:44:13 2016 (5704E87D)
    Checksum:  00094E79
    ImageSize:  00096000

Signed: ‎Wednesday, ‎April ‎6, ‎2016 5:16:32 AM (Pacific).

hmm patching https://codereview.chromium.org/1902563004/ into my tree gives:

[3540:4312:0418/160117:FATAL:module_list.cc(17)] Check failed: result. C:\Windows\system32\guard64.dll."

this is when calling FreeLibrary on guard64.dll

so clearly guard64 doesn't like being closed, or meddled with. Sigh.

adding guard64.dll to the sandboxed target DLL blacklist prevents the crash I can reproduce in my testing, so I'll be landing a CL to do that.

+chrome-blacklisting-announce

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/437dd9f37f0fde6e746a2a305229ca35a4414600

commit 437dd9f37f0fde6e746a2a305229ca35a4414600
Author: wfh <wfh@chromium.org>
Date: Tue Apr 19 19:29:19 2016

Add guard64.dll to sandbox target DLL blacklist.

BUG= 603701 

Review URL: https://codereview.chromium.org/1902853003

Cr-Commit-Position: refs/heads/master@{#388277}

[modify] https://crrev.com/437dd9f37f0fde6e746a2a305229ca35a4414600/content/common/sandbox_win.cc

Is this for M51 or M52? Should this be ReleaseBlock-Beta or Dev?

This should be RB-Dev as crash spiked in M52 chrome version.

Feel free to lower if someone thinks otherwise.

Thank you!

hi All,

I'm representative of Comodo Internet Security (CIS) product development.

We're aware about this issue and working on it. Suppose to address it in the nearest CIS release.

Thanks!

@wfh, please remove the Stability-Sheriff-Desktop label when you're satisfied with the fix and close the bug.

Someone needs to dremel through crashes on dev/beta/stable and determine whether this just affected canary 64-bit or also other channels on 64-bit and was just lose in the noise. Then, if it did, it needs to be merged.

Crash instances were observed on Google Chrome Stable (50.0.2661.87), Beta (51.0.2704.22), Dev (51.0.2704.19) & Canary (52.0.2715.0) channels.

Link - https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27RtlProcessFlsData%27#samplereports:5,productversion:1000

Reply to #23.This crash is affected only in Win64.

Canary:52.0.2715.0
=================
71 out of 39957 = .18%

Link : https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.name%3D%27RtlProcessFlsData%27%20AND%20product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.channel%3D%27canary%27%20AND%20cpu.Architecture%3D%27amd64%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion:1000

Dev: 51.0.2704.19
==================
9898 out of 28985 = 34.15%

Link : https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.name%3D%27RtlProcessFlsData%27%20AND%20product.name%3D%27Chrome%27%20AND%20(custom_data.ChromeCrashProto.channel%3D%27dev-m%27%20OR%20custom_data.ChromeCrashProto.channel%3D%27dev%27)%20AND%20cpu.Architecture%3D%27amd64%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion:1000

Beta:51.0.2704.22
=================
323 out of 7393 = 4.37%

Link : https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.name%3D%27RtlProcessFlsData%27%20AND%20product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.channel%20CONTAINS%20%27beta%27%20AND%20cpu.Architecture%3D%27amd64%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion:1000

Stable
=======

No instances

Link : https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.name%3D%27RtlProcessFlsData%27%20AND%20product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.channel%20CONTAINS%20%27stable%27%20AND%20cpu.Architecture%3D%27amd64%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D

The impact in Dev channel is higher, Will would you mind investigating further and merge the patch in #8 if needed.

To be clear:

+---------------+----------+---------+-----------+
| version       | specific | general | perc      |
+---------------+----------+---------+-----------+
| 52.0.2715.0   |        3 |    2466 |  0.121655 |
| 52.0.2713.0   |       12 |     974 |  1.232033 | <- fix went in.
| 52.0.2712.0   |     2601 |    3574 | 72.775602 |
| 52.0.2711.0   |     3141 |    4094 | 76.722032 |
| 52.0.2710.0   |     5456 |    7535 | 72.408759 |
| 52.0.2709.0   |     3165 |    4310 | 73.433875 |
| 52.0.2708.0   |     2723 |    3739 | 72.826959 |
| 52.0.2707.2   |      203 |     299 | 67.892977 |
| 52.0.2707.0   |     1445 |    2136 | 67.649813 |
| 52.0.2706.0   |      104 |     937 | 11.099253 |
| 52.0.2705.0   |      176 |    1054 | 16.698292 |
| 51.0.2704.7   |     4755 |   22817 | 20.839725 |

ppapi Crashes droped from >70% to < 1% after 437dd9f37f0fde6e746a2a305229ca35a4414600 landed, so this is fixed in trunk and M52.

Still trying to determine whether this ever affected M50 Stable or M51 Beta.

It seems this only ever spiked on canary, and not on any other channel, so I don't /think/ any merges are needed.

go/wfh_link_2