280 crashes in 63.* builds: https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Android%27%20AND%20product.Version%20LIKE%20%2763%25%27%20AND%20crash.Reason%3D%27SIGABRT%27%20OMIT%20RECORD%20IF%20SUM(special_protos.android_info.logcat_errors%20LIKE%20%27%25heap%20corruption%20detected%25%27)%20%3D%200&sql_dialect=dremelsql&ignore_case=false&enable_rewrite=false&omit_field_name=&omit_field_value=&omit_field_opt=&stbtiq=&reportid=&index=0


What is interesting is that there are 49 reports that have "heap corruption detected" message, but they are not SIGABRTs. Looks like some of them are legitimate, because on some version of Android abort() was implemented as a write to 0xdeadbaad.

If we exclude crashes fixed by issue 766752, we're still looking at 220 reports for 63.*: https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Android%27%20AND%20product.Version%20LIKE%20%2763%25%27%20AND%20crash.Reason%3D%27SIGABRT%27%20OMIT%20RECORD%20IF%20SUM(special_protos.android_info.logcat_errors%20LIKE%20%27%25heap%20corruption%20detected%25%27)%20%3D%200%20OR%20SUM(CrashedStackTrace.StackFrame.FunctionName%20LIKE%20%27CookiesTreeModel%25%27)%20!%3D%200&sql_dialect=dremelsql&ignore_case=false&enable_rewrite=false&omit_field_name=&omit_field_value=&omit_field_opt=&stbtiq=&reportid=&index=0

+Folks who might find this interesting.

There are 85 reports in 63.* (excluding crashes fixed by issue 766752) that have BindState::Destroy: https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Android%27%20AND%20product.Version%20LIKE%20%2763%25%27%20AND%20crash.Reason%3D%27SIGABRT%27%20OMIT%20RECORD%20IF%20SUM(special_protos.android_info.logcat_errors%20LIKE%20%27%25heap%20corruption%20detected%25%27)%20%3D%200%20OR%20SUM(CrashedStackTrace.StackFrame.FunctionName%20LIKE%20%27CookiesTreeModel%25%27)%20!%3D%200%20OR%20SUM(CrashedStackTrace.StackFrame.FunctionName%20LIKE%20%27%25BindState%25%3A%3ADestroy%25%27)%20%3D%200&sql_dialect=dremelsql&ignore_case=false&enable_rewrite=false&omit_field_name=&omit_field_value=&omit_field_opt=#samplereports

I.e. in all those crashes BindState::Destroy() caused double free. Any idea how it can happen? As I understand either BindState owns something, or it doesn't delete it during destruction.

Some stacks:

<ref_counted.h:409, name omitted>
base::internal::BindState<void ((anonymous namespace)::AndroidAccessTokenFetcher::*)(GoogleServiceAuthError const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&, base::Time const&), base::WeakPtr<(anonymous namespace)::AndroidAccessTokenFetcher> >::Destroy(base::internal::BindStateBase const*)
base::internal::CallbackBase::~CallbackBase()
base::OnceCallback<void (GURL const&)>::Run(GURL const&) &&
base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*)
base::MessageLoop::RunTask(base::PendingTask*)
base::MessageLoop::DoWork()
Java_org_chromium_base_SystemMessageHandler_nativeDoRunLoopOnce


<ref_counted.h:409, name omitted>
base::internal::BindState<void ((anonymous namespace)::AndroidAccessTokenFetcher::*)(GoogleServiceAuthError const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&, base::Time const&), base::WeakPtr<(anonymous namespace)::AndroidAccessTokenFetcher> >::Destroy(base::internal::BindStateBase const*)
base::internal::CallbackBase::~CallbackBase()
base::OnceCallback<void (GURL const&)>::Run(GURL const&) &&
void base::internal::FunctorTraits<base::RepeatingCallback<void (scoped_refptr<content::AppCacheInfoCollection>)>, void>::Invoke<base::RepeatingCallback<void (scoped_refptr<content::AppCacheInfoCollection>)>, scoped_refptr<content::AppCacheInfoCollection> >(base::RepeatingCallback<void (scoped_refptr<content::AppCacheInfoCollection>)>&&, scoped_refptr<content::AppCacheInfoCollection>&&)
base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*)
base::MessageLoop::RunTask(base::PendingTask*)
base::MessageLoop::DoWork()
Java_org_chromium_base_SystemMessageHandler_nativeDoRunLoopOnce


<memory:1636, name omitted>
std::__ndk1::__list_imp<BrowsingDataLocalStorageHelper::LocalStorageInfo, std::__ndk1::allocator<BrowsingDataLocalStorageHelper::LocalStorageInfo> >::clear()
base::internal::BindState<base::RepeatingCallback<void (std::__ndk1::list<BrowsingDataLocalStorageHelper::LocalStorageInfo, std::__ndk1::allocator<BrowsingDataLocalStorageHelper::LocalStorageInfo> > const&)>, std::__ndk1::list<BrowsingDataLocalStorageHelper::LocalStorageInfo, std::__ndk1::allocator<BrowsingDataLocalStorageHelper::LocalStorageInfo> > >::~BindState()
base::internal::BindState<base::RepeatingCallback<void (std::__ndk1::list<BrowsingDataLocalStorageHelper::LocalStorageInfo, std::__ndk1::allocator<BrowsingDataLocalStorageHelper::LocalStorageInfo> > const&)>, std::__ndk1::list<BrowsingDataLocalStorageHelper::LocalStorageInfo, std::__ndk1::allocator<BrowsingDataLocalStorageHelper::LocalStorageInfo> > >::Destroy(base::internal::BindStateBase const*)
base::internal::CallbackBase::~CallbackBase()
base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*)
base::MessageLoop::RunTask(base::PendingTask*)
base::MessageLoop::DoWork()
Java_org_chromium_base_SystemMessageHandler_nativeDoRunLoopOnce


<ref_counted.h:381, name omitted>
base::internal::BindState<void (*)(base::WeakPtr<content::ServiceWorkerContextCore>, long long), base::WeakPtr<content::ServiceWorkerContextCore>, long long>::~BindState()
base::internal::BindState<void (*)(base::WeakPtr<content::ServiceWorkerContextCore>, long long), base::WeakPtr<content::ServiceWorkerContextCore>, long long>::Destroy(base::internal::BindStateBase const*)
base::internal::CallbackBase::~CallbackBase()
base::OnceCallback<void (GURL const&)>::Run(GURL const&) &&
base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*)
base::MessageLoop::RunTask(base::PendingTask*)
base::MessageLoop::DoWork()
Java_org_chromium_base_SystemMessageHandler_nativeDoRunLoopOnce


mojo::edk::Channel::Message::~Message()
mojo::edk::UserMessageImpl::~UserMessageImpl()
mojo::edk::UserMessageImpl::~UserMessageImpl()
mojo::edk::ports::UserMessageEvent::~UserMessageEvent()
mojo::edk::ports::UserMessageEvent::~UserMessageEvent()
mojo::edk::Core::DestroyMessage(unsigned int)
mojo::Message::~Message()
std::__ndk1::__vector_base<mojo::Message, std::__ndk1::allocator<mojo::Message> >::~__vector_base()
IPC::(anonymous namespace)::ChannelAssociatedGroupController::~ChannelAssociatedGroupController()
IPC::(anonymous namespace)::ChannelAssociatedGroupController::~ChannelAssociatedGroupController()
scoped_refptr<IPC::(anonymous namespace)::ChannelAssociatedGroupController>::~scoped_refptr()
base::internal::BindState<void (IPC::(anonymous namespace)::ChannelAssociatedGroupController::*)(mojo::Message), scoped_refptr<IPC::(anonymous namespace)::ChannelAssociatedGroupController>, base::internal::PassedWrapper<mojo::Message> >::Destroy(base::internal::BindStateBase const*)
base::internal::CallbackBase::~CallbackBase()
base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*)
base::MessageLoop::RunTask(base::PendingTask*)
base::MessageLoop::DoWork()
base::MessagePumpLibevent::Run(base::MessagePump::Delegate*)
<name omitted>
content::BrowserThreadImpl::IOThreadRun(base::RunLoop*)
content::BrowserThreadImpl::Run(base::RunLoop*)
base::Thread::ThreadMain()
base::(anonymous namespace)::ThreadFunc(void*)

We will miss this entirely in webview, because only google-breakpad microdump lines are captured by the crash forwarder, and minidumps don't get any logs attached at all. It might be possible to change the crash forwarder to capture those lines specifically, though.

See issue 766752 for a case where heap corruption caused problems in this kind of bindstate case.

We have 8.5K "heap corruption detected" crashes for 63.0.3239.111: https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Android%27%20AND%20product.Version%3D%2763.0.3239.111%27%20OMIT%20RECORD%20IF%20SUM(special_protos.android_info.logcat_errors%20LIKE%20%27%25heap%20corruption%20detected%25%27)%20%3D%200&sql_dialect=dremelsql&ignore_case=false&enable_rewrite=false&omit_field_name=&omit_field_value=&omit_field_opt=

Interestingly more than half the crashes are in GPU.