Since when this crashes are spiking?
--So far everything started since 12/10/2018

Is this across all the platforms?
--No as of this bug it's specific to Windows.

Is there a specific extension which is causing this crashes?
--Top extension with almost 50% contribution is "Chrome Media Router", need to check if we have any experiment enabled since this morning.


Please find the URL where the spike can be observed when compared to M70 to M71 stable below :
https://crash.corp.google.com/browse?q=product_name%3D%27Chrome%27+AND+expanded_custom_data.ChromeCrashProto.ptype%3D%27extension%27&compProp=product.Version&v1=70.0.3538.110&v2=71.0.3578.80#magicsignature:100,-magicsignature2:30,-stablesignature:30

Tagging as "RBS" for M71 for tracking.

Note: We're planning M71 stable respin release on Wednesday, RC cut tomorrow morning PT. 

This looks related to the parser. It is strange that it did not happen on Dev or Canary though. Seems like there were Beta crashers though.

Per pbommana@, all  crashes started between 8am - 9am PT on 12/10. And  All crashes are specific to Chrome 32bit on Stable, none of the spikes are observed on Chrome 64bit as of now. Also specific to Windows 7.

Pls note M71 version 71.0.3578.80 released to Beta on 12/03 and to Stable at 12/04 (25% Win & Mac, 10% Linux).

Related bug 913845 reported on M71 Beta and Stable .

Updating the comment from issue 913845 and marking the bug as duplicate :

For the below magic signature we are seeing sudden spike in crashes from December 10 on the latest Stable/ Beta #71.0.3578.80

v8::internal::String::SlowFlatten - crash/85d07c450e495977
v8::internal::Heap::ExternalStringTable::IterateAll - crash/10ff8bd50fa15dde
v8::internal::String::CalculateLineEnds - crash/5f6b81049ce021a5
v8::internal::parsing::ParseFunction - crash/a116ce45a46b73c9
v8::internal::HeapObject::CanBeRehashed - crash/71f0c6eedad1cab9
v8::internal::Deserializer<v8::internal::BuiltinDeserializerAllocator>::~Deserializer - crash/0b9967c47384a3cf
v8::internal::Deserializer<v8::internal::BuiltinDeserializerAllocator>::Synchronize - crash/3e8ed79e43cba507
blink::ParkableString::ToString - crash/b69ceff260ca21e6
mojo::core::NodeChannel::OnChannelMessage - crash/87d9922d280b280e
[Renderer hang] base::subtle::SpinLock::LockSlow - crash/d71a8676ab8b7690
content::BrowserAccessibility::NativeReleaseReference - crash/4893a3577ee93034
blink::StringResourceBase::GetWTFString - crash/0a5ae08ffde9d5c6

Note:
1. For most of the magic signatures, more crashes are seen on Windows 7
2. Unable to pinpoint the exact suspect, even from the experiments. Hence, requesting Stability Sheriff to look into the issue.
3. Adding the component Blink>JavaScript as most of the magic signatures are related to V8.
4. As we are seeing 4 and 5 digit crashes, adding the label RBS.

Issue 913845 has been merged into this issue.

+ brucedawson fyi

There's a reddit thread on this with some troubleshooting:

https://www.reddit.com/r/chrome/comments/a50b4g/help_aw_snap_on_every_page/

- One user confirmed that uninstalling AVG solved it.
- Another user claims that Avast had a bad VPS definition update and is working on it, but no source.

Could be a lead on a possible cause?

Please find the new magic Signatures from top#30 under renderer crashes listed below, but there are considerable number of crashes which are spiking and can be found from the link below  :

Link to crashes comparision : https://goto.google.com/axcee

[Renderer hang] base::subtle::SpinLock::LockSlow [crash reports 74467]
blink::ParkableString::ToString [crash reports 59522]
content::BrowserAccessibility::NativeReleaseReference[Crash reports 51124]
V8::internal::String::SlowFlatten[crash reports 38919]
[Out of Memory] _aligned_malloc[crash reports 38147]
v8::internal::String::CalculateLineEnds[crash reports 30060]
blink::StringResourceBase::GetWTFString[crash reports 9654]

Here's the source on Avast indicating this may be related to a push they made (third post in the thread):

https://forum.avast.com/index.php?topic=223803.0

On the ParkableString::ToString() crash:

Looking at a crash like https://crash.corp.google.com/browse?q=product_name%3D%27Chrome%27+AND+expanded_custom_data.ChromeCrashProto.ptype%3D%27renderer%27+AND+expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27blink%3A%3AParkableString%3A%3AToString%27+AND+expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27blink%3A%3AParkableString%3A%3AToString%27+AND+product.Version%3D%2771.0.3578.80%27&stbtiq=&reportid=&index=0

The crash is a memory access at 0x18, so very close to nullptr. It's then likely an access to a member variable of a class, where |this| is nullptr.

This likely comes from inlining the call in ParkableString::ToString() to ParkableStringImpl::ToString(). This call requires the impl_ member in ParkableString to be non-nullptr, as the code is:

String ParkableString::ToString() const {
  return impl_ ? impl_->ToString() : String();
}

So, impl_ is either valid, or garbage that is not nullptr:
1. impl_ is valid: Then mutex_ is as well (as a regular member variable), and no crash should happen in Lock(), unless it has been corrupted somehow.
2. impl_ is garbage and not nullptr: then mutex_ is garbage as well, which could trigger a crash.

Now, as impl_ is a scoped_refptr<> in ParkableString, that would imply that ParkableString has been corrupted as well, or that ParkableStringImpl's memory has been corrupted.

All these classes are allocated with PartitionAlloc.

TL;DR - somebody (Avast?) is patching our code, badly. The 0x5411a9b4 (0x5411a9b5) return address for blink::ParkableString::Lock is invalid. It is *possible* that something is corrupting the stack, but code modifications are (IMHO) far more likely. We can't do anything about this, except ask Avast to fix their patching code.


I grabbed the minidump for crash/aef57fd40bed25d2 which is the one I got by following the link in comment#17. It references address 0xC. It has this call stack:

00 ntdll!RtlpEnterCriticalSectionContended
01 ntdll!RtlEnterCriticalSection
02 chrome_child!WTF::MutexBase::lock
03 chrome_child!blink::ParkableString::Lock
04 chrome_child!blink::HTMLAnchorElement::~HTMLAnchorElement
05 chrome_child!blink::V8CodeCache::GetCompileOptions

and looking at blink::ParkableString::Lock shows that the code or stack must have been modified. The return address is right after a pop instruction, not a call.

Then I looked at crash/ebb73aa893e8c17e. Same nonsensical return address.
crash/02bd046d151b5e19 - same
crash/64998f659aec8abe - different nonsensical return address

Here is the disassembly and typical return address - it makes no sense:

chrome_child!blink::ParkableString::Lock:
091ea992 55              push    ebp
091ea993 89e5            mov     ebp,esp
091ea995 57              push    edi
091ea996 56              push    esi
091ea997 8b39            mov     edi,dword ptr [ecx]
091ea999 85ff            test    edi,edi
091ea99b 7417            je      chrome_child!blink::ParkableString::Lock+0x22 (091ea9b4)
091ea99d 8d7704          lea     esi,[edi+4]
091ea9a0 89f1            mov     ecx,esi
091ea9a2 e839f4dbfd      call    chrome_child!WTF::MutexBase::lock (06fa9de0)
091ea9a7 ff4720          inc     dword ptr [edi+20h]
091ea9aa 89f1            mov     ecx,esi
091ea9ac 5e              pop     esi
091ea9ad 5f              pop     edi
091ea9ae 5d              pop     ebp
091ea9af e940f4dbfd      jmp     chrome_child!WTF::MutexBase::unlock (06fa9df4)
091ea9b4 5e              pop     esi
091ea9b5 5f              pop     edi <-------- return address points to here!
091ea9b6 5d              pop     ebp
091ea9b7 c3              ret

Related Avast bug - https://bugs.chromium.org/p/chromium/issues/detail?id=913803.

Thannk you everyone. This is fixed by Avast,so will move forward with tomorrow's M71 stable release as plan.

So far the crash rates have dropped considerably on 71.0.3578.80, hence removing release blocker label.

Crash rates are back to normal since December 13.