That change (https://chromium.googlesource.com/chromium/src/+/357c17552fb353ea9f3de6eca8a47b2d009067c8) was Linux-only, and I don't think it would be causing this issue on Windows.  Please re-add me if you suspect otherwise

tonikitoo@, could you please look into this change (https://codereview.chromium.org/2619773003) if possible?

Thank you!

In https://codereview.chromium.org/2573283002 @rockot *removed* the following code (from services/service_manager/standalone/desktop/launcher_process.cc).:

 (..)
 23 #include "services/service_manager/switches.h"	
 24 	
 25 namespace service_manager {	
 26 	
 27 int LauncherProcessMain() {	
 28 #if !defined(OFFICIAL_BUILD)	
 29   base::debug::EnableInProcessStackDumping();	
 30 #endif	
 31   base::CommandLine* command_line = base::CommandLine::ForCurrentProcess();	
 32   //  http://crbug.com/546644 	
 33   command_line->AppendSwitch(switches::kNoSandbox);
 (..)

See that line 33, references switches::kNoSandbox, declared by the file included in line 23.

What https://codereview.chromium.org/2619773003 does is removing the declaration of switches::kNoSandbox, since there is no code that references it anymore.

Not sure it would cause such crash regression.

Btw, AFAICU that code path is only exercised with launching chrome with --mash paramater, which is only available for ChromeOS builds.

Sure, thank you so much for the update. Adding 'Stability-Sheriff-Desktop' for further inputs.

pennymac@: Could you help triage this?  Thanks!

Users experienced this crash on the following builds:

Win Dev 57.0.2970.0 -  0.46 CPM, 420 reports, 77 clients (signature TargetNtCreateFile)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Just to update, crashes are seen only on the latest Dev release of Windows Clang build: 57.0.2979.2.

Link to the list of the builds:
===============================
https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.name%3D%27sandbox%3A%3ATargetNtCreateFile%27%20AND%20product.name%3D%27Chrome%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D

pennymac@: Could you please review the blocker label and update accordingly as we approach the M-57 branch date few days from now.

Since the crashes are seen only on the recent Clang build(57.0.2979.2), adjusting the blocker to Stable. Feel free to revert if anyone thinks otherwise.

Handing off to Niko - as this is only seen on clang Windows test releases.  Removing releaseblock label.

Removing from Stability-Sheriff-Desktop queue

Users experienced this crash on the following builds:

Win Dev 58.0.3000.4 -  0.86 CPM, 53 reports, 21 clients (signature TargetNtCreateFile)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

The crashes aren't specific to Clang builds I see crashes on latest and previous Chrome beta builds on M57 and all crashes are w.r.t Chrome 64bit with 99.9% just on Windows 7 , Please find the Chrome version and number of crashes on respective builds :


Beta :
57.0.2987.37	2.44%	578	
57.0.2987.21	3.00%	712	
57.0.2987.19	0.84%	199


Please find all Chrome versions where this crash was observed : https://goto.google.com/tufjh 


Note : I am not sure how actionable the stack trace would be with the stack quality at 5%(on crashes_id's so far what I have seen)

Niko, could you please confirm whether there were any clang builds released for these versions above?

We never shipped clang builds to beta, only to dev so far.

wfh,

Can I get your third party stability expertise on this?

https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20cpu.architecture%3D%27amd64%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27TargetNtCreateFile%27%20AND%20product.Version%3D%2757.0.2987.21%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,-productname,-component,-author,-changelist,-magicsignature2,-url,-simplifiedurl,-extensions,3rdparty,3rdpartystripped

Looks like Trend Micro is in > 90% of these crashes.
Trend Micro is system32\tmumh product, which has 2 dlls in our processes.  1) tmumevt64.dll and tmmon64.dll.

Given that this is a site of one of our sandbox hooks, my gut tells me hook clash.

Any thoughts Will?

(And note that this is a win7 x64 hook clash.)

Just threw a minidump from user into windbg.

Wanted to mention more exception details:

ExceptionAddress: 000000013f619707 (chrome!TargetNtCreateFile+0x000000000000033f)
   ExceptionCode: c0000409 (Security check failure or stack buffer overrun)  STACK_BUFFER_OVERRUN

  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 0000000000000002
Subcode: 0x2 FAST_FAIL_STACK_COOKIE_CHECK_FAILURE


In particular: 
BUCKET_ID_PREFIX_STR:  X64_STACK_BUFFER_OVERRUN_MISSING_GSFRAME_MISSING_GSFRAME_


Looks like the hook clash is wrecking the stack (blatting a stack guard canary) for our hook function (TargetNtCreateFile) which gets called instead of NtCreateFile in our processes.  Since our hook is in place and working, Trend Micro is probably patching our function (but think they're patching NtCreateFile)... and their hook doesn't preserve the stack appropriately.


For now, I'm removing the ReleaseBlock label.  Longer-term fix is to prevent third-party injection.  Also removing some folks from this ticket to prevent spam (feel free to add yourself back if interested).
-thakis, -tonikitoo, -rockot.

wfh@, have you had to reach out to TrendMicro (and have any sort of contact) before?  Or would you recommend I send a "good intention" warning with info to one of their external email addresses?  Happy to do so if you think that's the best way forward.

I can see if I can come up with a contact for TM. Do we know if the regression happened at a particular time, or in a particular version? do we know if it's tied to a particular version of TM software?

All I know is what can be gleaned from crash data (and I haven't spent lots of time).  Looks like this really started in M57, which has since moved from Dev to Beta branch, and the crashes are moving with it.

I could try to repro with their products and try to narrow down a version, but that's time.  
And I figure they should do that work to fix their product!  We know it's x64 Win7.  And we know the DLL names (if that helps them narrow down a product).

trendmicro seems to have a twitter account... but it's mostly high-level marketing.  No email address on their website, just phone numbers.

I reached out to Trend Micro today, and I hope we can work to jointly resolve this issue.

0:002> kv
 # Child-SP          RetAddr           : Args to Child                                                           : Call Site
00 00000000`0130e7a8 000007fe`fd671203 : 00000000`0130e868 00000000`7521c053 00000000`10004022 00000000`00000000 : ntdll!ZwDelayExecution+0xa
*** WARNING: Unable to verify timestamp for tmmon64.dll
*** ERROR: Module load completed but symbols could not be loaded for tmmon64.dll
01 00000000`0130e7b0 00000000`75188248 : 00000000`0130e930 00000000`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!SleepEx+0xab
02 00000000`0130e850 00000000`0130e930 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : tmmon64+0x18248
03 00000000`0130e858 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x130e930
0:002> lmvm tmmon64
Browse full module list
start             end                 module name
00000000`75170000 00000000`75285000   tmmon64  T (no symbols)           
    Loaded symbol image file: tmmon64.dll
    Image path: C:\Windows\system32\tmumh\20019\TmMon\2.5.0.2030\tmmon64.dll
    Image name: tmmon64.dll
    Browse all global symbols  functions  data
    Timestamp:        Thu Sep 29 06:28:00 2016 (57ED16E0)
    CheckSum:         00000000
    ImageSize:        00115000
    File version:     2.5.0.2030
    Product version:  2.5.0.2030
    File flags:       0 (Mask 0)
    File OS:          0 Unknown Base
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

pbommana -> I wonder if it would be possible for you to try and install "Trend Micro Maximum Security" 11.0 and try and reproduce this crash locally? It should manifest on M57 on beta/dev.

Sorry for delayed reply, I have installed and used Chrome 57.0.2987.54 on Windows 7(x64) VM for past few days with "Trend Micro Maximum Security" installed and haven't seen any crashes so far.

I will keep running that VM to see if I can reproduce the crashes. 

symbols for the stack in #0 are:

tmmon.dll 2.5.0.2029 DB68E7D6935948CE9AD256B63350D0D61
TmUmEvt.dll 7.0.0.1099 5D2ABE8E6007442BBA36213C5F16384E1
kernel32.dll 6.1.7601.23392 990CFD856F6E4DD783E726BDD2024A232
KERNELBASE.dll 6.1.7601.23392 CBBCE991C69047DEBD28D615EF22EFE52
ntdll.dll 6.1.7601.23572 285FB3D7D8F14FA684E7BE0CDC6448832

Just to update, Prestable 57.0.2987.98 has reported more than 2633 instances of this crash on Windows OS. Below link gives in detail about the same:

https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27TargetNtCreateFile%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion:1000

Thanks.!

Removing "blocking" since this happens for non-clang builds too.

Just to update the latest behavior of the crash.
Link to the list of builds:
-------------------------
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20sandbox%3A%3ATargetNtCreateFile%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion:1000

62.0.3178.0	1.64%	3	-Canary
62.0.3175.3	2.73%	5	-Dev

Issue not seen on beta & stable.
wfh@, Could you please take a look into it.
Thanks..!!

Users experienced this crash on the following builds:

Win Canary 64.0.3248.2 -  0.20 CPM, 4 reports, 4 clients (signature [GPU hang] sandbox::TargetNtOpenFile)
Win Canary 64.0.3248.2 -  0.15 CPM, 3 reports, 3 clients (signature [Renderer hang] sandbox::TargetNtCreateFile)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Just to update the latest behavior of the crash.

Magic Signature: '[Renderer hang] sandbox::TargetNtCreateFile'

This is top#20th renderer crash on latest dev-64.0.3260.2 & still seeing 22 instances from 22 clients so far.

64.0.3260.2	5.41%	22	-Dev

Link to the list of builds:
-------------------------
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BRenderer%20hang%5D%20sandbox%3A%3ATargetNtCreateFile%27&sql_dialect=dremelsql&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#productversion:1000

wfh@, Could you please take a look into it.

Thanks..!

Just to update:

[GPU hang] sandbox::TargetNtOpenFile

Still seeing 32 instances from 31 clients so far on latest dev-64.0.3282.24.

64.0.3282.24	0.96%	30	-Dev

Link to the list of builds:
--------------------------
https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU%20hang%5D%20sandbox%3A%3ATargetNtOpenFile%27&sql_dialect=googlesql&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D

wfh@, Could you please take a look into it.
Thanks..!

Users experienced this crash on the following builds:

Win Dev 65.0.3322.3 -  0.24 CPM, 21 reports, 14 clients (signature [GPU hang] sandbox::TargetNtOpenFile)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Dev 66.0.3343.3 -  0.29 CPM, 103 reports, 81 clients (signature [GPU hang] sandbox::TargetNtOpenFile)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Dev 67.0.3377.1 -  0.28 CPM, 62 reports, 39 clients (signature [GPU hang] sandbox::TargetNtOpenFile)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Beta 66.0.3359.45 -  0.32 CPM, 269 reports, 231 clients (signature [GPU hang] sandbox::TargetNtOpenFile)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Just to update the latest behavior of this issue in the latest channels:

Still seeing 185 crashes from 162 clients so far on latest beta - 68.0.3440.68 on Windows OS. This crash is ranked as number #23 in 'GPU-Process' beta crashes. 

70.0.3499.0	0.00%	5 - Canary
69.0.3493.3	0.01%	100 - Dev
68.0.3440.68	0.03%	185 - Beta
67.0.3396.99	19.59%	132035 - Stable

Link to the list of builds:
-------------------------
https://crash.corp.google.com/browse?q=product_name%3D%27Chrome%27+AND+expanded_custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27+AND+expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU+hang%5D+sandbox%3A%3ATargetNtOpenFile%27#-productname:1000,productversion:1000,-magicsignature:50,-magicsignature2:50,-stablesignature:50,-magicsignaturesorted:50

Thanks!