friendly ping

The attached trace file has 1.5GB allocation from notification database. The allocation stack trace is:

↳ƒbase::(anonymous namespace)::ThreadFunc(void*)
↳ƒbase::internal::SchedulerWorker::RunBackgroundPooledWorker()
↳ƒbase::internal::SchedulerWorker::RunWorker()
↳ƒbase::internal::TaskTracker::RunAndPopNextTask(scoped_refptr<base::internal::Sequence>, base::internal::CanScheduleSequenceObserver*)
↳ƒbase::internal::TaskTrackerPosix::RunOrSkipTask(base::internal::Task, base::internal::Sequence*, bool)
↳ƒbase::internal::TaskTracker::RunOrSkipTask(base::internal::Task, base::internal::Sequence*, bool)
↳ƒbase::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*)
↳ƒvoid base::internal::InvokeHelper<false, void>::MakeItSo<void (android_webview::AwLoginDelegate::*)(std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> > const&, std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> > const&), scoped_refptr<android_webview::AwLoginDelegate>, std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> >, std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> > >(void (android_webview::AwLoginDelegate::*&&)(std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> > const&, std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> > const&), scoped_refptr<android_webview::AwLoginDelegate>&&, std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> >&&, std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> >&&)
↳ƒvoid base::internal::Invoker<base::internal::BindState<void (content::PlatformNotificationContextImpl::*)(GURL const&, content::NotificationDatabaseData const&, base::RepeatingCallback<void (bool, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&)> const&), scoped_refptr<content::PlatformNotificationContextImpl>, GURL, content::NotificationDatabaseData, base::RepeatingCallback<void (bool, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&)> >, void ()>::RunImpl<void (content::PlatformNotificationContextImpl::* const&)(GURL const&, content::NotificationDatabaseData const&, base::RepeatingCallback<void (bool, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&)> const&), std::__ndk1::tuple<scoped_refptr<content::PlatformNotificationContextImpl>, GURL, content::NotificationDatabaseData, base::RepeatingCallback<void (bool, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&)> > const&, 0u, 1u, 2u, 3u>(void (content::PlatformNotificationContextImpl::* const&)(GURL const&, content::NotificationDatabaseData const&, base::RepeatingCallback<void (bool, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&)> const&), std::__ndk1::tuple<scoped_refptr<content::PlatformNotificationContextImpl>, GURL, content::NotificationDatabaseData, base::RepeatingCallback<void (bool, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&)> > const&, std::__ndk1::integer_sequence<unsigned int, 0u, 1u, 2u, 3u>)
↳ƒvoid base::internal::InvokeHelper<false, void>::MakeItSo<void (content::mojom::Frame_ExtractSmartClipData_ProxyToResponder::*)(std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> > const&, std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> > const&, gfx::Rect const&), std::__ndk1::unique_ptr<content::mojom::Frame_ExtractSmartClipData_ProxyToResponder, std::__ndk1::default_delete<content::mojom::Frame_ExtractSmartClipData_ProxyToResponder> >, std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> > const&, std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> > const&, gfx::Rect const&>(void (content::mojom::Frame_ExtractSmartClipData_ProxyToResponder::*&&)(std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> > const&, std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> > const&, gfx::Rect const&), std::__ndk1::unique_ptr<content::mojom::Frame_ExtractSmartClipData_ProxyToResponder, std::__ndk1::default_delete<content::mojom::Frame_ExtractSmartClipData_ProxyToResponder> >&&, std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> > const&, std::__ndk1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__ndk1::allocator<unsigned short> > const&, gfx::Rect const&)
↳ƒcontent::PlatformNotificationContextImpl::DoWriteNotificationData(GURL const&, content::NotificationDatabaseData const&, base::RepeatingCallback<void (bool, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&)> const&)
↳ƒcontent::NotificationDatabase::DeleteAllNotificationDataForOrigin(GURL const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&, std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >*)
↳ƒcontent::NotificationDatabase::DeleteAllNotificationDataInternal(GURL const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&, long long, std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >*)
↳ƒleveldb::(anonymous namespace)::DBIter::Next()
↳ƒleveldb::(anonymous namespace)::DBIter::FindNextUserEntry(bool, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >*)
↳ƒleveldb::(anonymous namespace)::MergingIterator::Next()
↳ƒleveldb::IteratorWrapper::Next()
↳ƒleveldb::(anonymous namespace)::TwoLevelIterator::Next()
↳ƒleveldb::IteratorWrapper::Next()
↳ƒleveldb::(anonymous namespace)::TwoLevelIterator::SkipEmptyDataBlocksForward()
↳ƒleveldb::(anonymous namespace)::TwoLevelIterator::InitDataBlock()
↳ƒleveldb::Table::BlockReader(void*, leveldb::ReadOptions const&, leveldb::Slice const&)
↳ƒleveldb::ReadBlock(leveldb::RandomAccessFile*, leveldb::ReadOptions const&, leveldb::BlockHandle const&, leveldb::BlockContents*)

Deleted: trace-88a4d5d1ced2a964.gz 38.0 KB

trace-88a4d5d1ced2a964.gz 38.0 KB Download

ssid@ Do you have access to this database? If so you you post a "ls -l" of the notification db directory? I think it should be "<profile-dir>/Platform Notifications"

Sorry for adding without context:
Out of Process Heap Profiling is a project that, for a very small set users, records heap allocations and sends back anonymized heap dumps. We've obtained a heap dump that suggests that there's a potential StackSamplingProfiler memory leak. We do not have repro steps. Small allocations [in size or frequency] were pruned from the heap dump before it was uploaded. We do not have access to any information about the users.

The old bug was filed using older system that did not have stack traces. The new profiling has stack traces from users.

If you think this is not leveldb bug then this should be assigned to notification team. If you could help out with scenarios where this could happen, it would be great.

The bug's description and details suggests 200MB of RAM use during open. I'm have not seen leveldb use this much RAM on open, but I wonder if the callback which is run by LazyInitialize is consuming the memory. May want to get somebody from the notification team to look into this.

Regarding the stack in #c3 above I can't say for sure, but if there was somehow a HUGE number of notification values stored for a given origin then DeleteAllNotificationDataInternal's WriteBatch (an in-memory transaction of sorts) could be consuming the memory. This leveldb limitation also affects Indexed DB, and there is a plan to fix this problem. Read more at https://docs.google.com/document/d/16_igCI15Gfzb6UYqeuJTmJPrzEtawz6Y1tVOKNtYgiU.

If this profiling test created and deleted a very large number of notification db entries, which is likely is, then it could very well be hitting these memory numbers.

Hi cmumford: To clarify - we are getting these memory dump reports from the wild, so this is a real bug.

Who works on notifications, can we route to them?

In the trace, we're seeing ~2000 PlatformNotificationData objects, and a total of ~2 million PlatformNotificationAction objects.

Yes this looks more like issue of notifications creating large number of objects since it looks like leveldb is WAI.
+notifications owners.

The attached trace in comment #3 has 7.2 million PlatformNotificationAction objects in 5000 PlatformNotificationData objects.

peter@, can you or someone on your team take a look at this?

Also include dimich@ as his team found some issue with leveldb recently.

This other stack trace suggests there is 72MB of data in std::vector<content::NotificationDatabaseData>, in another report:

Sorry for the stack trace being reverse from previous ones:

base::trace_event::CFIBacktraceAndroid::Unwind(void const**, unsigned int)
⬑ heap_profiling::AllocatorShimLogAlloc(heap_profiling::AllocatorType, void*, unsigned int, char const*)
⬑ heap_profiling::(anonymous namespace)::HookAlloc(base::allocator::AllocatorDispatch const*, unsigned int, void*)
⬑ operator new(unsigned int)
⬑ std::__ndk1::__split_buffer<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData>&>::__split_buffer(unsigned int, unsigned int, std::__ndk1::allocator<content::NotificationDatabaseData>&)
⬑ void std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> >::__push_back_slow_path<content::NotificationDatabaseData const&>(content::NotificationDatabaseData const&&&)
⬑ content::NotificationDatabase::ReadAllNotificationDataInternal(GURL const&, long long, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> >*) const
⬑ content::PlatformNotificationContextImpl::DoReadAllNotificationDataForServiceWorkerRegistration(GURL const&, long long, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)> const&, std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > >, bool)
⬑ void base::internal::FunctorTraits<void (content::PlatformNotificationContextImpl::*)(GURL const&, long long, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)> const&, std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > >, bool), void>::Invoke<void (content::PlatformNotificationContextImpl::*)(GURL const&, long long, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)> const&, std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > >, bool), scoped_refptr<content::PlatformNotificationContextImpl>, GURL, long long, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)>, std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > >, bool>(void (content::PlatformNotificationContextImpl::*)(GURL const&, long long, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)> const&, std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > >, bool), scoped_refptr<content::PlatformNotificationContextImpl>&&, GURL&&, long long&&, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)>&&, std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > >&&, bool&&)
⬑ void base::internal::InvokeHelper<false, void>::MakeItSo<void (content::PlatformNotificationContextImpl::*)(GURL const&, long long, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)> const&, std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > >, bool), scoped_refptr<content::PlatformNotificationContextImpl>, GURL, long long, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)>, std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > >, bool>(void (content::PlatformNotificationContextImpl::*&&)(GURL const&, long long, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)> const&, std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > >, bool), scoped_refptr<content::PlatformNotificationContextImpl>&&, GURL&&, long long&&, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)>&&, std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > >&&, bool&&)
⬑ void base::internal::Invoker<base::internal::BindState<void (content::PlatformNotificationContextImpl::*)(GURL const&, long long, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)> const&, std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > >, bool), scoped_refptr<content::PlatformNotificationContextImpl>, GURL, long long, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)>, base::internal::PassedWrapper<std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > > >, bool>, void ()>::RunImpl<void (content::PlatformNotificationContextImpl::* const&)(GURL const&, long long, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)> const&, std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > >, bool), std::__ndk1::tuple<scoped_refptr<content::PlatformNotificationContextImpl>, GURL, long long, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)>, base::internal::PassedWrapper<std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > > >, bool> const&, 0u, 1u, 2u, 3u, 4u, 5u>(void (content::PlatformNotificationContextImpl::* const&&&)(GURL const&, long long, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)> const&, std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > >, bool), std::__ndk1::tuple<scoped_refptr<content::PlatformNotificationContextImpl>, GURL, long long, base::RepeatingCallback<void (bool, std::__ndk1::vector<content::NotificationDatabaseData, std::__ndk1::allocator<content::NotificationDatabaseData> > const&)>, base::internal::PassedWrapper<std::__ndk1::unique_ptr<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > >, std::__ndk1::default_delete<std::__ndk1::set<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > > >, bool> const&&&, std::__ndk1::integer_sequence<unsigned int, 0u, 1u, 2u, 3u, 4u, 5u>)
⬑ void base::internal::InvokeHelper<false, void>::MakeItSo<void (chrome::mojom::ChromeRenderFrame_RequestThumbnailForContextNode_ProxyToResponder::* const&)(std::__ndk1::vector<unsigned char, std::__ndk1::allocator<unsigned char> > const&, gfx::Size const&), std::__ndk1::unique_ptr<chrome::mojom::ChromeRenderFrame_RequestThumbnailForContextNode_ProxyToResponder, std::__ndk1::default_delete<chrome::mojom::ChromeRenderFrame_RequestThumbnailForContextNode_ProxyToResponder> >, std::__ndk1::vector<unsigned char, std::__ndk1::allocator<unsigned char> > const&, gfx::Size const&>(void (chrome::mojom::ChromeRenderFrame_RequestThumbnailForContextNode_ProxyToResponder::* const&&&)(std::__ndk1::vector<unsigned char, std::__ndk1::allocator<unsigned char> > const&, gfx::Size const&), std::__ndk1::unique_ptr<chrome::mojom::ChromeRenderFrame_RequestThumbnailForContextNode_ProxyToResponder, std::__ndk1::default_delete<chrome::mojom::ChromeRenderFrame_RequestThumbnailForContextNode_ProxyToResponder> >&&, std::__ndk1::vector<unsigned char, std::__ndk1::allocator<unsigned char> > const&&&, gfx::Size const&&&)
⬑ base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*)
⬑ base::internal::TaskTracker::RunOrSkipTask(base::internal::Task, base::internal::Sequence*, bool)
⬑ base::internal::TaskTrackerPosix::RunOrSkipTask(base::internal::Task, base::internal::Sequence*, bool)
⬑ base::internal::TaskTracker::RunAndPopNextTask(scoped_refptr<base::internal::Sequence>, base::internal::CanScheduleSequenceObserver*)
⬑ base::internal::SchedulerWorker::RunWorker()
⬑ base::internal::SchedulerWorker::RunPooledWorker()
⬑ base::(anonymous namespace)::ThreadFunc(void*)

Re: #12. 

We found that if "reuse_logs" is enabled, the database that had entries added and then deleted still loads a log (up to 8Mb) into memory every time it is opened. The log contains add/delete records and DB is empty, but the log may be never compacted... https://chromium-review.googlesource.com/1026164

This case seems different because of the amount of content the DB seems to have...

I think there's a couple of different things in here -

  1. The stack in #c3 happens when deleting notifications. We're only storing one in memory there, so Chris' suspicion about a large WriteBatch sounds likely. None of the callbacks should be holding a large amount of data.
  2. The stack in #c13 happens when a developer calls ServiceWorkerRegistration.getNotifications(). This could be expensive if we've stored a large number of notifications as they're written to a vector.
  3. Notifications have at most 2 action buttons, so the ratios we're seeing in this bug (5000:7.2M) make no sense. I found a bug and sent a CL to address it[1].

Anita, you followed up with the Android team to figure out if we could have a race-free NotificationManager.getActiveNotifications() in the compat library, right? That'd allow us to synchronize more accurately.

Alternatively, could you talk with ssid@ to see if we can somehow identify which site(s) they're from, given that you're visiting MTV?

It seems reasonable to put some sort of limit on the number of notifications we store per origin. Could you give that a thought too?

Finally, yangsharon@ is working on moving away from having our own LevelDB instance for notifications, instead using the Service Worker database's user data mechanism.

[1] https://chromium-review.googlesource.com/c/chromium/src/+/1110231

Recent reports:
Id:                     Issue                           allocation size in MB
214f68a664e58bb7	NotificationDatabase		723M
4f18b4325a718ec0	NotificationDatabase		645M
9853eea34068ce85	NotificationDatabase		640M
4cf1fd9794ecfe9e	NotificationDatabase		618M

memlog/ID to see stack traces,
crash/ID to see trace file.

Issue 854706 has been merged into this issue.

I had a look at this with ssid@ earlier. 

All of the recent stacks we looked at had huge allocations of vectors of  NotificationActions, which makes the bug fix mentioned in [15] sound pretty promising. 

There is no way we can get which site(s) are sending the notifications from the memory reports. 

We could try to repro by sending a lot of notifications with actions and watch for memory growing in chrome://tracing. Or we could just wait for your fix and then see if it goes away.

This is top of the reports of large memory usage on Android, and is currently the 4th most commonly reported (these reports are taken when memory usage exceeds 200mb on Android).

I will follow up on the notification compat request but that will take a while to become available.. and didn't we try using getActiveNotifications on O+ but it had to be reverted due to some issue with webapks? ( Issue 800363  and  Issue 806982 ) So I'm not sure that will solve this issue.

Assigning back to peter for next steps.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9ea530770929075c5059a6475a3862d941186a54

commit 9ea530770929075c5059a6475a3862d941186a54
Author: Peter Beverloo <peter@chromium.org>
Date: Fri Jun 22 14:20:59 2018

Clear existing notification actions when deserializing the proto

Right now they're additive, which is a bug.

Bug: 800152
Change-Id: Ib1312dd921e18c17e872d4dea2c5962604cfecca
Reviewed-on: https://chromium-review.googlesource.com/1110231
Commit-Queue: Peter Beverloo <peter@chromium.org>
Reviewed-by: Anita Woodruff <awdf@chromium.org>
Cr-Commit-Position: refs/heads/master@{#569604}
[modify] https://crrev.com/9ea530770929075c5059a6475a3862d941186a54/content/browser/notifications/notification_database_data_conversions.cc
[modify] https://crrev.com/9ea530770929075c5059a6475a3862d941186a54/content/browser/notifications/notification_database_data_unittest.cc

Peter's fix rolled in 69.0.3472 [1] which has now reached Canary and Dev. 

Sid can you see if that made a difference yet or is it too early to tell?

[1] https://chromiumdash.appspot.com/commit/9ea530770929075c5059a6475a3862d941186a54

I am still seeing reports from with large allocations of NotificationDatabaseData. But, no longer seeing the Actions vector growing.
Report ID 79a9d60f61321381 has 330MB of notification data

 notification_data_vector - 140 MB
 PlatformNotificationData's strings - 115 MB
 PlatformNotificationData::data - 65 MB
 leveldb block cache - 8 MB