Abdul, can you help us triage this issue in 68 Windows?

A few more crash reports from the customer:
https://drive.google.com/open?id=1WjEYknGGTc0ESNqBuCIgypSYZJ1VRjAh

632ca1c1-e2d2-4a67-b37f-0147bc6e0846.dmp:

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.
EXCEPTION_CODE_STR:  c0000005

40de2e06-9c90-4be5-810d-c2a18b81373d.dmp

ERROR_CODE: (NTSTATUS) 0x517a7ed - <Unable to get error code text>
EXCEPTION_CODE: (Win32) 0x517a7ed (85436397) - <Unable to get error code text>
EXCEPTION_CODE_STR:  517a7ed

it looks like at least two crashes are related to security software, but initial crash and 40de2e06-9c90-4be5-810d-c2a18b81373d.dmp are different:
https://drive.google.com/open?id=1bC7udYEKXoC8EV8FD0JwmclP_grpWDBfrMB71YYUJvM  

new crash reports provided by customer: https://drive.google.com/open?id=1Y_tFACPIebqWP3QaNONnPpBC-_sLoEq9

+ bheenan and privard

Can we get a browser resource assign to this issue? This is a critical Chrome browser customer experiencing the crashes.

to clarify, from what I can see in those latter crashes - they have different error codes. For example:

*** WARNING: Unable to verify timestamp for igvk64.dll
*** ERROR: Module load completed but symbols could not be loaded for igvk64.dll
*** WARNING: Unable to verify checksum for KERNEL32.DLL
*** WARNING: Unable to verify checksum for USER32.dll
*** WARNING: Unable to verify timestamp for chrome_elf.dll
*** ERROR: Module load completed but symbols could not be loaded for chrome_elf.dll
*** WARNING: Unable to verify checksum for ole32.dll
*** WARNING: Unable to verify timestamp for chrome.exe
*** ERROR: Module load completed but symbols could not be loaded for chrome.exe
*** WARNING: Unable to verify timestamp for chrome_child.dll
*** ERROR: Module load completed but symbols could not be loaded for chrome_child.dll
*** WARNING: Unable to verify checksum for combase.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for combase.dll - 
GetUrlPageData2 (WinHttp) failed: 12007.
Probably caused by : igvk64.dll ( igvk64+a23002 )

which might be related to https://bugs.chromium.org/p/chromium/issues/detail?id=863086

  

Checking now. 

Just to clarify, new Google mail UI = new Gmail?

abdulsyed@ yes, new Gmail UI

Also being tracked internally at b/112432627.

+khushal@ - could this be related to GPU hardware acceleration issue similar to crbug/872117?

One of the internal bugs linked to a crash report, I'm looking through crashes with that magic signature:

https://crash.corp.google.com/browse?q=EXISTS+%28SELECT+1+FROM+UNNEST%28CrashedStackTrace.StackFrame%29+WHERE+FunctionName%3D%27blink%3A%3AFrame%3A%3AGetSettings%28%29+const%27%29#samplereports

And I see crashes coming from blink::NetworkInformation. +tbansal, could you take a look?

Re#16: This seems like Issue 848172, which was fixed for M-69, but not M-68. At that point, we were not seeing any crashes in M-68.

Is this bug about browser crash or renderer crash? Issue 848172 was about renderer crash.

Per #1, appears to be browser crash and happening sporadically on Windows. 

OK, I do not think NetInfo crashes from #16 (or Issue 848172) can cause browser crash. I added "PROCESS TYPE" to the query in #16, and I see ~1.92% browser crashes. Further filtering by browser crashes, shows about ~80 crashes, but they do not seem to be related to NetInfo (http://shortn/_szHnZm7MBd).

Its really odd for a gmail content change to cause a browser crash.

vkasatkin@, could you clarify whether the whole browser is crashing or just the gmail tab?

Yes, I'm suspecting this is a renderer crash. Another bug, crbug/873268. 

Crash: 
https://crash.corp.google.com/browse?q=product_name%3D%27Chrome%27+AND+expanded_custom_data.ChromeCrashProto.url.raw%3D%27https%3A%2F%2Fmail.google.com%2Fmail%2Fu%2F0%2F%23inbox%27+AND+expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27viz%3A%3AClientLayerTreeFrameSink%3A%3ASubmitCompositorFrame%27+AND+expanded_custom_data.ChromeCrashProto.ptype%3D%27renderer%27&compProp=product.Version&v1=67.0.3396.99&v2=68.0.3440.106

I don't see anything sticking out for browser. 

khushalsagar@ - yes, from customer description whole browser is crashing

to add on #23: customer confirmed:
- all users are running Gmail with new UI
- Chrome crashes completely, requires restart 
   

two more crash dump files from customer. Chome is running with HW acceleration disabled.
https://drive.google.com/open?id=1wizo8eZb-qroFoo1ZOuxfPEWi_KnN25e
https://drive.google.com/open?id=1xO3Pcwt0y5pyHg4eP_NY_weNKdJcf78X 

+ bruce

is this related to crbug/870054?

I doubt this is related to  crbug.com/870054 .

BTW, when summarizing crash dumps the exception code is not sufficient. A call stack is needed as well. Here's the summary of the two crashes from comment #25. The first looks like heap corruption which could be caused by a Chrome bug or by a bug in any third-party module in the process. The second one looks like a pattern I have seen before of crashes deep in Microsoft's crypto code. I suspect a bug in Microsoft's code but I have never been able to get a repro. If the bug can be reproduced reliably then running Chrome under Application Verifier may help find it. This requires 64-bit Chrome and a lot of memory but it will reliably catch heap corruption. Instructions are in https://www.chromium.org/developers/testing/page-heap-for-chrome - you'll want to disable Leak, Handles, and Locks checks.

Note that both of these crashes are happening deep inside Microsoft's code.

Here are the summaries of those two crashes. If we get similar summaries from more of the crash dumps then we can bucket them into different signatures and perhaps figure out what is going on.

0:041> .ecxr
rax=0000000000000000 rbx=0000020cf5580000 rcx=000000007a836c4b
rdx=0000020cfb3f8750 rsi=0000020cfb3f8760 rdi=0000000000000000
rip=00007ffafe465c19 rsp=0000007a413fdb80 rbp=0000000000000000
 r8=0000020cfb3f8760  r9=0000000000000020 r10=00000fff5fbdb2c8
r11=0000000000040100 r12=0000000000000000 r13=00007ffafb30a630
r14=0000020cfb3f74a0 r15=000000000000012d
iopl=0         nv up ei pl nz na po nc
cs=0033  ss=0000  ds=0000  es=0000  fs=0053  gs=002b             efl=00010206
ntdll!RtlSizeHeap+0xb9:
00007ffa`fe465c19 440fb74824      movzx   r9d,word ptr [rax+24h] ds:00000000`00000024=????
0:041> kc
  *** Stack trace for last set context - .thread/.cxr resets it
 # Call Site
00 ntdll!RtlSizeHeap
01 windows_storage!kfapi::CFolderCache::GetIDList
02 windows_storage!kfapi::CKFFacade::GetFolderIDList
03 windows_storage!SHILAliasTranslate
04 windows_storage!CDesktopFolder::ParseDisplayName
05 windows_storage!CRegFolder::ParseDisplayName
06 windows_storage!SHParseDisplayName
07 windows_storage!ILCreateFromPathEx
08 windows_storage!CShellLink::_SetPIDLPath
09 windows_storage!CShellLink::SetPath
0a chrome!`anonymous namespace'::AddShellLink
0b chrome!JumpListUpdater::AddCustomCategory
0c chrome!JumpList::CreateNewJumpListAndNotifyOS
0d chrome!JumpList::RunUpdateJumpList
0e chrome!base::internal::Invoker<...>()>::Run
0f chrome!base::`anonymous namespace'::PostTaskAndReplyRelay::RunTaskAndPostReply
10 chrome!base::internal::Invoker<base::internal::BindState<void (*)(base::(anonymous namespace)::PostTaskAndReplyRelay),base::(anonymous namespace)::PostTaskAndReplyRelay>,void ()>::RunOnce
11 chrome!base::debug::TaskAnnotator::RunTask
12 chrome!base::internal::TaskTracker::RunOrSkipTask
13 chrome!base::internal::TaskTracker::RunAndPopNextTask
14 chrome!base::internal::SchedulerWorker::RunWorker
15 chrome!base::internal::SchedulerWorker::RunSharedCOMWorker
*** WARNING: Unable to verify checksum for KERNEL32.DLL
16 chrome!base::`anonymous namespace'::ThreadFunc
17 KERNEL32!BaseThreadInitThunk
18 ntdll!RtlUserThreadStart


And here's the other one:

0:039> .ecxr
rax=0000000000000000 rbx=0000021964bb3c00 rcx=0000021966dc7dc0
rdx=0000000000000000 rsi=0000000000000001 rdi=00007ffe507e30c0
rip=00007ffe50718b12 rsp=000000e53ddfe6f0 rbp=000000e53ddfe7a9
 r8=00000000e7ffffff  r9=0000000000000002 r10=0000000000000001
r11=000000000000db40 r12=000002195f3b3260 r13=000000e53ddfe830
r14=00000000ffffffff r15=0000000000002400
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=0000  ds=0000  es=0000  fs=0053  gs=002b             efl=00010202
CRYPT32!CryptVerifyCertificateSignatureEx+0x182:
00007ffe`50718b12 803805          cmp     byte ptr [rax],5 ds:00000000`00000000=??
*** WARNING: Unable to verify checksum for chrome.dll
0:039> kc
  *** Stack trace for last set context - .thread/.cxr resets it
 # Call Site
00 CRYPT32!CryptVerifyCertificateSignatureEx
01 CRYPT32!ChainGetSubjectStatus
02 CRYPT32!CCertIssuerList::CreateElement
03 CRYPT32!CCertIssuerList::AddIssuer
04 CRYPT32!CChainPathObject::FindAndAddIssuersFromCacheByMatchType
05 CRYPT32!CChainPathObject::FindAndAddIssuersByMatchType
06 CRYPT32!CChainPathObject::FindAndAddIssuers
07 CRYPT32!CChainPathObject::CChainPathObject
08 CRYPT32!ChainCreatePathObject
09 CRYPT32!CCertChainEngine::CreateChainContextFromPathGraph
0a CRYPT32!CCertChainEngine::GetChainContext
0b CRYPT32!CertGetCertificateChain
0c chrome!net::CertVerifyProcWin::VerifyInternal
0d chrome!net::CertVerifyProc::Verify
0e chrome!net::DoVerifyOnWorkerThread
0f chrome!base::internal::Invoker<...> ()>::RunOnce
10 chrome!base::internal::ReturnAsParamAdapter<std::unique_ptr<net::(anonymous namespace)::ResultHelper,std::default_delete<net::(anonymous namespace)::ResultHelper> > >
11 chrome!base::internal::Invoker<base::internal::BindState<void (*)(base::OnceCallback<base::File ()>, base::File *),base::OnceCallback<base::File ()>,base::File *>,void ()>::RunOnce
12 chrome!base::`anonymous namespace'::PostTaskAndReplyRelay::RunTaskAndPostReply
13 chrome!base::internal::Invoker<base::internal::BindState<void (*)(base::(anonymous namespace)::PostTaskAndReplyRelay),base::(anonymous namespace)::PostTaskAndReplyRelay>,void ()>::RunOnce
14 chrome!base::debug::TaskAnnotator::RunTask
15 chrome!base::internal::TaskTracker::RunOrSkipTask
16 chrome!base::internal::TaskTracker::RunAndPopNextTask
17 chrome!base::internal::SchedulerWorker::RunWorker
18 chrome!base::internal::SchedulerWorker::RunPooledWorker
*** WARNING: Unable to verify checksum for KERNEL32.DLL
19 chrome!base::`anonymous namespace'::ThreadFunc
1a KERNEL32!BaseThreadInitThunk
1b ntdll!RtlUserThreadStart

we've got two more crash reports today:
https://drive.google.com/open?id=1NRlxIu2ueEmf4YfM2RPov1vW0JkiVwHY


0:018> .ecxr
rax=0000d53b3fe8b25a rbx=0000000000000000 rcx=0000007f021ff720
rdx=00007ffd5cb2ff1c rsi=0000007f021ff720 rdi=0000000000000000
rip=00007ffd5cb2dc93 rsp=0000007f021ff700 rbp=0000000000000000
 r8=0000000000000000  r9=00007ffd5cb2ff1c r10=00000fffab965fe3
r11=ffffffffffffffff r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl nz na po nc
cs=0033  ss=0000  ds=0000  es=0000  fs=0053  gs=002b             efl=00000206
chrome_elf+0x2dc93:
00007ffd`5cb2dc93 4889f1          mov     rcx,rsi
0:018> kc
  *** Stack trace for last set context - .thread/.cxr resets it
 # Call Site
00 chrome_elf

https://drive.google.com/open?id=1rrE5pqpOgFFYpTx5kPH8Hm3PgJYOEjVG

0:002> .excr
rax=000001b0b08ed4b0 rbx=000001b0b08ed4f0 rcx=0000000000001b80
rdx=0000000000010001 rsi=000001b09f3502f0 rdi=0000000000000140
rip=00007ffe961fbcbf rsp=0000006bac5fc920 rbp=000001b09f350cf0
 r8=ffffffffffffffff  r9=000000000000000b r10=000001b0b08eb930
r11=0000000000000006 r12=0000000000000016 r13=000001b09f359240
r14=000001b09f350000 r15=0000000000000001
iopl=0         nv up ei pl nz na po nc
cs=0033  ss=0000  ds=0000  es=0000  fs=0053  gs=002b             efl=00010206
ntdll!RtlpLowFragHeapAllocFromContext+0x2bf:
00007ffe`961fbcbf f6430f3f        test    byte ptr [rbx+0Fh],3Fh ds:000001b0`b08ed4ff=??
0:002>  kc
  *** Stack trace for last set context - .thread/.cxr resets it
 # Call Site
00 ntdll!RtlpLowFragHeapAllocFromContext
01 ntdll!RtlpAllocateHeapInternal
02 mswsock!SockSocket
03 mswsock!WSPSocket
04 WS2_32!WSASocketW
05 chrome
06 0x0
07 0x0
08 0x0

To get full call stacks you need to set up Chrome's symbols. There are some instructions here:

https://www.chromium.org/developers/how-tos/debugging-on-windows

In windbg I believe it works out to using this command:

.sympath+ SRV*c:\Symbols*https://chromium-browser-symsrv.commondatastorage.googleapis.com

If the problem is third-party software then we need to figure out what third-party software is common to these crashes. I don't know an easy way of doing this. I see EventReporting.ApplicationFilter.Monitor.Win64.dll in one of the crashes but I don't know a scalable way of finding patterns in these anecdotal reports.

With Chrome's symbol server enabled when I look at the call stack for the first crash I see this:

0:018> kc
  *** Stack trace for last set context - .thread/.cxr resets it
 # Call Site
00 chrome_elf!crash_reporter::DumpWithoutCrashing
01 chrome_elf!crash_reporter::internal::DumpProcessForHungInputThread
02 KERNEL32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

So it's not actually a crash, it's a hang. Still a serious bug.


The second one also appears to be heap corruption, and Chrome's symbols give a more complete call stack (only the top few are relevant, but here they all are):

0:002> kc
  *** Stack trace for last set context - .thread/.cxr resets it
 # Call Site
00 ntdll!RtlpLowFragHeapAllocFromContext
01 ntdll!RtlpAllocateHeapInternal
02 mswsock!SockSocket
03 mswsock!WSPSocket
04 WS2_32!WSASocketW
05 chrome!net::CreatePlatformSocket
06 chrome!net::TCPSocketWin::Open
07 chrome!net::TCPClientSocket::OpenSocket
08 chrome!net::TCPClientSocket::DoConnect
09 chrome!net::TCPClientSocket::DoConnectLoop
0a chrome!net::TCPClientSocket::Connect
0b chrome!net::TransportConnectJob::DoTransportConnect
0c chrome!net::TransportConnectJob::DoLoop
0d chrome!net::ConnectJob::Connect
0e chrome!net::internal::ClientSocketPoolBaseHelper::RequestSocketInternal
0f chrome!net::internal::ClientSocketPoolBaseHelper::RequestSocket
10 chrome!net::ClientSocketPoolBase<net::TransportSocketParams>::RequestSocket
11 chrome!net::ClientSocketHandle::Init<net::TransportClientSocketPool>
12 chrome!net::SSLConnectJob::DoTransportConnect
13 chrome!net::SSLConnectJob::DoLoop
14 chrome!net::ConnectJob::Connect
15 chrome!net::internal::ClientSocketPoolBaseHelper::RequestSocketInternal
16 chrome!net::internal::ClientSocketPoolBaseHelper::RequestSocket
17 chrome!net::ClientSocketPoolBase<net::SSLSocketParams>::RequestSocket
18 chrome!net::ClientSocketHandle::Init<net::TransportClientSocketPool>
19 chrome!net::`anonymous namespace'::InitSocketPoolHelper
1a chrome!net::InitSocketHandleForHttpRequest
1b chrome!net::HttpStreamFactoryImpl::Job::DoInitConnectionImpl
1c chrome!net::HttpStreamFactoryImpl::Job::DoInitConnection
1d chrome!net::HttpStreamFactoryImpl::Job::DoLoop
1e chrome!net::HttpStreamFactoryImpl::Job::RunLoop
1f chrome!net::HttpStreamFactoryImpl::JobController::DoCreateJobs
20 chrome!net::HttpStreamFactoryImpl::JobController::DoLoop
21 chrome!net::HttpStreamFactoryImpl::JobController::RunLoop
22 chrome!base::RepeatingCallback<void (int)>::Run
23 chrome!net::ProxyResolutionService::Request::QueryComplete
24 chrome!base::RepeatingCallback<void (int)>::Run
25 chrome!network::`anonymous namespace'::ProxyResolverMojo::Job::CompleteRequest
26 chrome!proxy_resolver::mojom::ProxyResolverRequestClientStubDispatch::Accept
27 chrome!mojo::internal::MultiplexRouter::ProcessIncomingMessage
28 chrome!mojo::internal::MultiplexRouter::Accept
29 chrome!mojo::Connector::ReadSingleMessage
2a chrome!mojo::Connector::ReadAllAvailableMessages
2b chrome!base::RepeatingCallback<void (unsigned int, const mojo::HandleSignalsState &)>::Run
2c chrome!mojo::SimpleWatcher::OnHandleReady
2d chrome!mojo::SimpleWatcher::Context::Notify
2e chrome!mojo::SimpleWatcher::Context::CallNotify
2f chrome!mojo::edk::WatcherDispatcher::InvokeWatchCallback
30 chrome!mojo::edk::Watch::InvokeCallback
31 chrome!mojo::edk::RequestContext::~RequestContext
32 chrome!mojo::edk::NodeChannel::OnChannelMessage
33 chrome!mojo::edk::Channel::OnReadComplete
34 chrome!mojo::edk::`anonymous namespace'::ChannelWin::OnReadDone
35 chrome!mojo::edk::`anonymous namespace'::ChannelWin::OnIOCompleted
36 chrome!base::MessagePumpForIO::WaitForIOCompletion
37 chrome!base::MessagePumpForIO::DoRunLoop
38 chrome!base::MessagePumpWin::Run
39 chrome!base::RunLoop::Run
3a chrome!content::BrowserProcessSubThread::IOThreadRun
3b chrome!base::Thread::ThreadMain

Also, the hung browser has its main thread on the stack below - deep in some JavaScript regex stuff. You'd need to talk to the v8 team to see if there is a v8 bug or if it is just badly behaved JavaScript. It's only worth doing this if we get multiple crashes/hangs with the same signature.

0:000> kc
 # Call Site
00 chrome_child!v8::internal::StringHasher::AddCharacter
01 chrome_child!v8::internal::StringHasher::AddCharacters
02 chrome_child!v8::internal::IteratingStringHasher::VisitOneByteString
03 chrome_child!v8::internal::String::VisitFlat<v8::internal::IteratingStringHasher>
04 chrome_child!v8::internal::IteratingStringHasher::Hash
05 chrome_child!v8::internal::String::ComputeAndSetHash
06 chrome_child!v8::internal::Name::Hash
07 chrome_child!v8::internal::CompilationCacheShape::RegExpHash
08 chrome_child!v8::internal::RegExpKey::RegExpKey
09 chrome_child!v8::internal::CompilationCacheTable::LookupRegExp
0a chrome_child!v8::internal::CompilationCacheRegExp::Lookup
0b chrome_child!v8::internal::CompilationCache::LookupRegExp
0c chrome_child!v8::internal::RegExpImpl::Compile
0d chrome_child!v8::internal::JSRegExp::Initialize
0e chrome_child!v8::internal::JSRegExp::Initialize
0f chrome_child!v8::internal::__RT_impl_Runtime_RegExpInitializeAndCompile
10 chrome_child!v8::internal::Runtime_RegExpInitializeAndCompile

+ hablich@ from V8. Can you please check the crashes/hangs in #30 in V8?

+ pastormovj@ - do you know if there are any known enterprise issues around 3RD Pty and browser crashes, particularly with DLP apps like McAfee and Dtex?

Adding Yang and Jakob regarding Regex.

Some particular RegExp patterns could indeed cause hangs, both during regexp compilation and execution. In many cases, the fix is the rewrite the regexp in a more efficient form.

But the trace in #30 does not look suspicious. It actually leads outside regexp code and into string hashing, which should be cheap.

Do other hangs also point into similar RegExp code? Without multiple hangs pointing to the same area, I would not suspect a regexp as the root cause.

Do we have a way to reproduce? 

brucedawson@ - thank you for the details on how to get full call stacks! 

Just a fresh example of a crash https://drive.google.com/open?id=1ulpksaY0pw9C4Wt86jboldwJvnRy1URa

0:031> .excr
rax=0043005c00720065 rbx=0000026407573e50 rcx=000002640c1a8af0
rdx=000002640c1aa520 rsi=0000026407523a00 rdi=000002640c1a8af0
rip=0000026402953b2e rsp=00000088bfafe610 rbp=0000026407523790
 r8=0000000000000001  r9=000002640c1a8af0 r10=0000000000000000
r11=00000088bfafe718 r12=000002640c19c730 r13=0000026407523790
r14=0000000000000004 r15=0000026407523a00
iopl=0         nv up ei pl nz na po nc
cs=0033  ss=0000  ds=0000  es=0000  fs=0053  gs=002b             efl=00010206
CRYPT32!ChainIsValidPubKeyMatchForIssuer+0x16:
00000264`02953b2e 833800          cmp     dword ptr [rax],0 ds:0043005c`00720065=????????
0:031> kc
  *** Stack trace for last set context - .thread/.cxr resets it
 # Call Site
00 CRYPT32!ChainIsValidPubKeyMatchForIssuer
01 CRYPT32!CCertIssuerList::AddIssuer
02 CRYPT32!CChainPathObject::FindAndAddIssuersByMatchType
03 CRYPT32!CChainPathObject::FindAndAddIssuers
04 CRYPT32!CChainPathObject::CChainPathObject
05 CRYPT32!ChainCreatePathObject
06 CRYPT32!CCertIssuerList::AddIssuer
07 CRYPT32!CChainPathObject::FindAndAddIssuersByMatchType
08 CRYPT32!CChainPathObject::FindAndAddIssuers
09 CRYPT32!CChainPathObject::CChainPathObject
0a CRYPT32!ChainCreatePathObject
0b CRYPT32!CCertChainEngine::CreateChainContextFromPathGraph
0c CRYPT32!CCertChainEngine::GetChainContext
0d CRYPT32!CertGetCertificateChain
0e chrome!net::CertVerifyProcWin::VerifyInternal
0f chrome!net::CertVerifyProc::Verify
10 chrome!net::DoVerifyOnWorkerThread
11 chrome!base::internal::Invoker<base::internal::BindState<std::unique_ptr<net::(anonymous namespace)::ResultHelper,std::default_delete<net::(anonymous namespace)::ResultHelper> > (*)(const scoped_refptr<net::CertVerifyProc> &, const scoped_refptr<net::X509Certificate> &, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > &, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > &, int, const scoped_refptr<net::CRLSet> &, const std::vector<scoped_refptr<net::X509Certificate>,std::allocator<scoped_refptr<net::X509Certificate> > > &),scoped_refptr<net::CertVerifyProc>,scoped_refptr<net::X509Certificate>,std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::basic_string<char,std::char_traits<char>,std::allocator<char> >,int,scoped_refptr<net::CRLSet>,std::vector<scoped_refptr<net::X509Certificate>,std::allocator<scoped_refptr<net::X509Certificate> > > >,std::unique_ptr<net::(anonymous namespace)::ResultHelper,std::default_delete<net::(anonymous namespace)::ResultHelper> > ()>::RunOnce
12 chrome!base::internal::ReturnAsParamAdapter<std::unique_ptr<net::(anonymous namespace)::ResultHelper,std::default_delete<net::(anonymous namespace)::ResultHelper> > >
13 chrome!base::internal::Invoker<base::internal::BindState<void (*)(base::OnceCallback<base::File ()>, base::File *),base::OnceCallback<base::File ()>,base::File *>,void ()>::RunOnce
14 chrome!base::`anonymous namespace'::PostTaskAndReplyRelay::RunTaskAndPostReply
15 chrome!base::internal::Invoker<base::internal::BindState<void (*)(base::(anonymous namespace)::PostTaskAndReplyRelay),base::(anonymous namespace)::PostTaskAndReplyRelay>,void ()>::RunOnce
16 chrome!base::debug::TaskAnnotator::RunTask
17 chrome!base::internal::TaskTracker::RunOrSkipTask
18 chrome!base::internal::TaskTracker::RunAndPopNextTask
19 chrome!base::internal::SchedulerWorker::RunWorker
1a chrome!base::internal::SchedulerWorker::RunPooledWorker
1b chrome!base::`anonymous namespace'::ThreadFunc
1c KERNEL32!BaseThreadInitThunk
1d ntdll!RtlUserThreadStart

We also have a process monitor logs from that machine https://drive.google.com/open?id=1KPmPtJ2fVPl65WJIiEI_SPeCeiIGlDlE
But it looks like crash dumps are not always generated. 



jgruber@ - unfortunately crashes are sporadic and we do not have any specific pattern at this moment to reproduce


 

just adding another crash from this machine https://drive.google.com/open?id=1XUYWLGPBqgSaYsbl9lYaVASRB-HYsWdz , which seems to be v8
Full call stack: https://drive.google.com/open?id=132ssq7m_ttGe5GbUrEthlw2CK_wmmcAl7zSDh2oa0tU 

Just to confirm: these crashes are happening on multiple machines (so it's not simply bad RAM on a single machine)? There doesn't seem to be much in common between the jumplist update and cert verification crashes.

yes, these happening on multiple machines. Very first dumps I have attached are from different sources. Here is the example from another machine:
https://drive.google.com/open?id=1ANkUsxE2Cj5QCrPFRgK3PExMZU6GMBmC

It looks like we have two issues here.
1. Crashes due to  third-party software (we are working with customer to isolate) 
2. Crashes/hangs due to other reason. Also, sometimes no crash dump is generated.

A fresh crash report from customer, seems to be an issue with third-party software eecrlrev64.dll:
https://drive.google.com/open?id=153qkSRHvhv3Woq9lJS50xRlU3Kq2JqMq

0:014> kc
 # Call Site
00 bcrypt!BCryptDestroyHash
01 eecrlrev64
02 0x0

I can see multiple crash reports for eecrlrev64.dll, for example  Crash ID: crash/3a09a00c583498bf, crash/45b37324c2515c1c.
Also found this issue https://bugs.chromium.org/p/chromium/issues/detail?id=859072


Another report from today, looks to be related to v8: 
https://drive.google.com/open?id=1m6fQr7OSFwh7FMlwn6QxOyxJQdLKfOYt
Full call stack: https://drive.google.com/open?id=1Lq4rlwr-U83RYN9m2KB9V53TLaeKqirVulMKAgzTtIw

two more crash reports from yesterday:
https://drive.google.com/open?id=11lKpxlW2MnUu40bQK5dOGQTLBijfsKIT
https://drive.google.com/open?id=1ZqOv0lFcEzmAqJ_FxRc3bjznAnMpLLQ6

Call stack:
https://drive.google.com/open?id=14nrErJXTmrMnCHw-HbeHsD004NS_exPbriJYMbAtwrg

Debug log: 
https://drive.google.com/open?id=1EBgn0T1VuYYByqw31UfWdpIWDtv7KRud

Checking in, have we gotten any closer to a root cause?  The customer is looking for an update on a resolution to this issue..  It looks like vkasatkin@ has provided numerous logs and crash dumps.  

Let us know if you need anything to help resolve this P1 issue.

bigo@ - some of the updates can be seen on https://buganizer.corp.google.com/issues/112432627

Can we try this on beta again? The original description claims that this isn't happening on M69, but please confirm. 

Thanks for the update.  I've informed the customer about the possible fix in M69 and I will wait to hear back with any results.

Version M71 ChromeOS is unusable with Gmail
- But I'm hearing this problem happens on M69 as well 
- Changing labels to show impact (some more details in b/112432627)
- We have large enterprises reporting this issue now.

+kbleicher@ (Chrome OS M71 TPM), ptal comment #47.

If this is happening on M69 as well I'm not sure how it's a chrome OS issue or even a M71 blocker since it's not a regression...  

We need to bisect to identify the root cause and when it was introduced.  Still seems to be TBD in the bug.

Also as this is regressed in M69, it won't block M71 beta. Hence, removing "ReleaseBlock-Beta" label. 

M71 Stable promotion is coming VERY soon. Your bug is labelled as Stable  ReleaseBlock, pls make sure to land the fix and request a merge into the release branch ASAP. Thank you.

M71 Stable promotion is coming VERY soon. Your bug is labelled as Stable  ReleaseBlock, pls make sure to land the fix and request a merge into the release branch ASAP. Thank you.

This crash is not actually a crash bucket. The correlation between these crashes appears to be illusory. I looked at five crashes and their are four or five different causes. It is not meaningful to ask for a fix to this bug. We should break this report into multiple bugs, one for each signature.

The only one that I am particularly interested in is the first one that is a dupe of crbug.com/792289. Because this is a memory corruption crash it may be possible to identify it using Application Verifier. At the end of https://www.chromium.org/developers/how-tos/debugging-on-windows there is a section talking about how to install and use Application Verifier. If you do this then you can:
a) Launch Chrome
b) Configure Application Verifier for chrome.exe, with Locks, Leak, and Handles disabled. Click Save.
c) Open the gmail tab
d) If it crashes then please share the crash dump
e) Remove chrome.exe from Application Verifier to avoid making Chrome run slower and use more memory in general.

Note that this really only works with 64-bit Chrome.

Here's some more detailed analysis:

The first crash I looked at (https://drive.google.com/open?id=11lKpxlW2MnUu40bQK5dOGQTLBijfsKIT) looks like a duplicate of crbug.com/792289. Just as in that bug there is heap corruption that seems to be happening inside Microsoft's crypto libraries. This appears to be a bug in one of the crypto DLLs.

The next crash I examined (https://drive.google.com/open?id=1ZqOv0lFcEzmAqJ_FxRc3bjznAnMpLLQ6) shows no sign of being related, except that it looked like heap corruption. Maybe there are multiple bugs? Or maybe the heap corruption took a while to manifest.

https://drive.google.com/file/d/1ANkUsxE2Cj5QCrPFRgK3PExMZU6GMBmC/view is a crash in the FontCache. Maybe a corrupt font installed?

https://drive.google.com/file/d/153qkSRHvhv3Woq9lJS50xRlU3Kq2JqMq/view is probably a bug in the third-party module eecrlrev64.dll. There is nothing we can do about this.

https://drive.google.com/open?id=1m6fQr7OSFwh7FMlwn6QxOyxJQdLKfOYt is an out-of-memory crash. This is usually a problem with the particular web page.

M71 Stable promotion is coming VERY soon. Your bug is labelled as Stable  ReleaseBlock, pls make sure to land the fix and request a merge into the release branch ASAP. Thank you.

This is not a clearly identified bug - there are multiple unrelated bugs that are being bucketed into this bug. Therefore the RBS tag doesn't really make sense - I just removed it.

Hello!
This bug is receiving this notice because there has been no acknowledgment of its existence in quite a bit of time.
- If you are currently working on this bug, please provide an update.
- If you are currently affected by this bug, please update with your current symptoms and relevant logs.

If there has been no updates provided by EOD Thursday, 01/17/19 (5pm EST), this bug will be archived and can be re-opened at any time deemed necessary.

Thank you!

Due to lack of action this bug has been Archived. If work is still being done on this issue or you are still experiencing this issue please feel free to re-open with the appropriate information.