New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 709927 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Apr 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: iOS
Pri: 1
Type: Bug



Sign in to add a comment

Chrome_iOS: Crash Report - google::protobuf::internal::ArenaStringPtr::CreateInstanceNoArena

Project Member Reported by pmadalla@chromium.org, Apr 10 2017

Issue description

Product name: Chrome_iOS
Magic Signature: google::protobuf::internal::ArenaStringPtr::CreateInstanceNoArena

Current link:
https://crash.corp.google.com/browse?q=product.name%3D'Chrome_iOS'%20AND%20product.version%3D'57.0.2987.137'%20AND%20ReportID%3D'03492c9640000000'%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D'google%3A%3Aprotobuf%3A%3Ainternal%3A%3AArenaStringPtr%3A%3ACreateInstanceNoArena'&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#3

Search properties:
product.name: Chrome_iOS
product.version: 57.0.2987.137
reportid: 03492c9640000000

Metadata :
Product Name: Chrome_iOS
Product Version: 57.0.2987.137
Report ID: 03492c9640000000
Report Time: Tue, 04 Apr 2017 20:28:26 GMT
Uptime: 16904000 ms
Cumulative Uptime: 0 ms
User Email: 
OS Name: iOS
OS Version: 10.2.1 14D27
CPU Architecture: arm64
CPU Info: 

First seen in version: on 2017/04/01
Total number of occurrences so far in : 221
Average daily occurrences: 06

Stack Trace :
Stack Quality81%Show frame trust levels
0x00000001831a21e0	(libc++.1.dylib + 0x0003c1e0 )	std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
0x00000001000e7654	(Chrome -arenastring.h:291 )	google::protobuf::internal::ArenaStringPtr::CreateInstanceNoArena(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const*)
0x00000001000e7654	(Chrome -arenastring.h:291 )	google::protobuf::internal::ArenaStringPtr::CreateInstanceNoArena(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const*)
0x0000000100c5bd24	(Chrome -arenastring.h:211 )	syncer::DeviceInfoSyncService::CreateLocalData(syncer::DeviceInfo const*)
0x0000000100c5b420	(Chrome -device_info_sync_service.cc:288 )	syncer::DeviceInfoSyncService::SendLocalData(syncer::SyncChange::SyncChangeType)
0x00000001006cf10c	(Chrome -callback.h:85 )	base::Timer::RunScheduledTask()
0x0000000100682190	(Chrome -callback.h:68 )	base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*)
0x00000001006976f0	(Chrome -message_loop.cc:421 )	base::MessageLoop::RunTask(base::PendingTask*)
0x0000000100697938	(Chrome -message_loop.cc:430 )	base::MessageLoop::DeferOrRunPendingTask(base::PendingTask)
0x0000000100697df0	(Chrome -message_loop.cc:562 )	base::MessageLoop::DoDelayedWork(base::TimeTicks*)
0x00000001006e79ac	(Chrome -message_pump_mac.mm:306 )	base::MessagePumpCFRunLoopBase::RunWork()
0x00000001006e73d0	(Chrome -message_pump_mac.mm:278 )	base::MessagePumpCFRunLoopBase::RunWorkSource(void*)
0x000000018475eb58	(CoreFoundation + 0x000ddb58 )	__CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__
0x000000018475e4a0	(CoreFoundation + 0x000dd4a0 )	__CFRunLoopDoSources0
0x000000018475c0a0	(CoreFoundation + 0x000db0a0 )	__CFRunLoopRun
0x000000018468a2b4	(CoreFoundation + 0x000092b4 )	CFRunLoopRunSpecific
0x000000018613e194	(GraphicsServices + 0x0000c194 )	GSEventRunModal
0x000000018a6d17f8	(UIKit + 0x0007a7f8 )	-[UIApplication _run]
0x000000018a6cc530	(UIKit + 0x00075530 )	UIApplicationMain
0x00000001000aa20c	(Chrome -chrome_exe_main.mm:63 )	main
0x000000018366d5b4	(libdyld.dylib + 0x000045b4 )	start

 

Comment 1 by edchin@chromium.org, Apr 10 2017

Labels: ReleaseBlock-Stable M-58
Owner: zea@chromium.org
Status: Assigned (was: Untriaged)

Comment 2 by edchin@chromium.org, Apr 10 2017

Components: Services>Sync

Comment 3 by pkl@chromium.org, Apr 11 2017

Hi, Nicolas, Stack frame shows something related to sync. Not sure if this is actually your area. Please help re-dispatch if it isn't. Thanks!

Comment 4 by s...@chromium.org, Apr 11 2017

Cc: zea@chromium.org
Labels: Sync-V2
Owner: s...@chromium.org
Status: Started (was: Assigned)
Looks like we're waking up off a timer, and trying to write out local data from |local_device_info_provider_|. In the past we've had problems doing this if we're in the process of a shutdown. We theoretically guard against this by stopping the timer if StopSyncing() is called. Something is clearly going wrong though, will investigate.

Comment 5 by s...@chromium.org, Apr 11 2017

Labels: -Sync-V2

Comment 6 by s...@chromium.org, Apr 11 2017

arenastring.h seems to be a protobuf library that we're crashing in the middle of. We pass a bunch of const string& into it and then it converts to raw pointer, then de-refs again, and gives it to new ::std::string(), where we seem to crash. So my working theory is one of these strings is bad somehow.

We get all of these strings from the DeviceInfo object, with suspects beign
* guid
* client_name
* chrome_version
* sync_user_agent
* signin_scopd_device_id

Bizarrely, signin_scopd_device_id isn't held onto as a const by DeviceInfo, but just a normal std::string, see https://cs.chromium.org/chromium/src/components/sync/device_info/device_info.h?l=89

Looking at my data, it actually looks like the signin_scopd_device_id field has been blanked since ~M57.

Comment 7 by s...@chromium.org, Apr 11 2017

Looking at crashes, I see other arenastring stable signatures before M57, but none coming from DeviceInfoSyncService. This class didn't receive significant changes in M57, https://chromium.googlesource.com/chromium/src/+log/0dd660b9ba6e4699468c635874da841a25e135a7/components/sync/device_info/device_info_sync_service.cc , which makes me think it has to do with how the DeviceInfo[Provider] is exposing data.

Also interesting to note, these clients all have huge uptime, like hours. Which in my experience isn't very typical.

Comment 8 by s...@chromium.org, Apr 11 2017

Well, disregard #7, initial hunch in #4 seems to be correct. If you change the pulse interval at https://cs.chromium.org/chromium/src/components/sync/device_info/device_info_util.cc?l=19 to TimeDelta::FromMilliseconds(1), and then sign out, you immediately hit this crash. As we had encountered earlier, the shutdown is not being called in time. Going to add a simple nullptr check.
Project Member

Comment 9 by bugdroid1@chromium.org, Apr 11 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/babffd17e43935e30657dcfda02d962791adf460

commit babffd17e43935e30657dcfda02d962791adf460
Author: skym <skym@chromium.org>
Date: Tue Apr 11 22:47:21 2017

[Sync] Always check the device info provider for being null before using it.

During sign out, the DeviceInfoProviderImpl is reset before
DeviceInfoSyncService is told we're shutting down, as well as in a
separate task. This means if we get extremely unlucky, and a pulse is
happening at just the right time, we'll try to use the provider after
it's been destroyed. To fix this, we can just check that it's not
nullptr before using it. The other usages of the provider in
DeviceInfoSyncService have stronger guarantees from sync that we're not
in shutdown yet, and as such they DCHECK instead.

No unit tests were added as part of this change. It is non-trivial to
force an asyc pulse, and we currently do not have good seams to allow
this. Since this change needs to be merged back, I want this
modification to be as small as possible.

BUG= 709927 

Review-Url: https://codereview.chromium.org/2810873005
Cr-Commit-Position: refs/heads/master@{#463814}

[modify] https://crrev.com/babffd17e43935e30657dcfda02d962791adf460/components/sync/device_info/device_info_sync_service.cc

Comment 10 by s...@chromium.org, Apr 12 2017

Labels: Merge-Request-58
Adding merge request for M58. This is a race condition that's causing crashes to real users right now.

Was present in M57, we're seeing 10-30 crashes on iOS per day. Not sure why we haven't seen similar crashes on other platforms.
Project Member

Comment 11 by sheriffbot@chromium.org, Apr 12 2017

Labels: -Merge-Request-58 Merge-Review-58 Hotlist-Merge-Review
This bug requires manual review: Less than 9 days to go before AppStore submit on M58
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), bhthompson@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 12 by s...@chromium.org, Apr 12 2017

I forgot to mention in #10, the risk here is low. Basically just wrapping existing logic with a null check, so we don't UAF.
can you mark this issue as fixed?

Comment 14 by s...@chromium.org, Apr 13 2017

Status: Fixed (was: Started)

Comment 15 by s...@chromium.org, Apr 13 2017

Labels: Merge-Request-58
Labels: -Hotlist-Merge-Review -Merge-Request-58 -Merge-Review-58 Merge-Approved-58
Project Member

Comment 17 by bugdroid1@chromium.org, Apr 13 2017

Labels: -merge-approved-58 merge-merged-3029
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c9c3186627372f19063b9855a2176150567a5889

commit c9c3186627372f19063b9855a2176150567a5889
Author: Sky Malice <skym@chromium.org>
Date: Thu Apr 13 21:40:45 2017

[Sync] Always check the device info provider for being null before using it.

During sign out, the DeviceInfoProviderImpl is reset before
DeviceInfoSyncService is told we're shutting down, as well as in a
separate task. This means if we get extremely unlucky, and a pulse is
happening at just the right time, we'll try to use the provider after
it's been destroyed. To fix this, we can just check that it's not
nullptr before using it. The other usages of the provider in
DeviceInfoSyncService have stronger guarantees from sync that we're not
in shutdown yet, and as such they DCHECK instead.

No unit tests were added as part of this change. It is non-trivial to
force an asyc pulse, and we currently do not have good seams to allow
this. Since this change needs to be merged back, I want this
modification to be as small as possible.

BUG= 709927 

Review-Url: https://codereview.chromium.org/2810873005
Cr-Commit-Position: refs/heads/master@{#463814}
(cherry picked from commit babffd17e43935e30657dcfda02d962791adf460)

Review-Url: https://codereview.chromium.org/2818803003 .
Cr-Commit-Position: refs/branch-heads/3029@{#698}
Cr-Branched-From: 939b32ee5ba05c396eef3fd992822fcca9a2e262-refs/heads/master@{#454471}

[modify] https://crrev.com/c9c3186627372f19063b9855a2176150567a5889/components/sync/device_info/device_info_sync_service.cc

Sign in to add a comment