New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 794920 link

Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 2
Type: Bug



Sign in to add a comment

Data race in google_breakpad::NonAllocatingMap<40ul, 128ul, 200ul>::SetKeyValue

Project Member Reported by ClusterFuzz, Dec 14 2017

Issue description

Detailed report: https://clusterfuzz.com/testcase?key=6004094081957888

Fuzzer: ifratric-browserfuzzer-v3
Job Type: linux_tsan_chrome_mp
Platform Id: linux

Crash Type: Data race READ 1
Crash Address: 0x7bb4000005e8
Crash State:
  google_breakpad::NonAllocatingMap<40ul, 128ul, 200ul>::SetKeyValue
  crash_reporter::internal::CrashKeyStringImpl::Set
  discardable_memory::ClientDiscardableSharedMemoryManager::MemoryUsageChanged
  
Sanitizer: thread (TSAN)

Regressed: https://clusterfuzz.com/revisions?job=linux_tsan_chrome_mp&range=523781:523782

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=6004094081957888

Issue filed automatically.

See https://github.com/google/clusterfuzz-tools for more information.
 
Project Member

Comment 1 by ClusterFuzz, Dec 14 2017

Components: Internals>CrashReporting
Labels: Test-Predator-Auto-Components
Automatically applying components based on crash stacktrace and information from OWNERS files.

If this is incorrect, please apply the Test-Predator-Wrong-Components label.
Project Member

Comment 2 by ClusterFuzz, Dec 14 2017

Labels: Test-Predator-Auto-Owner
Owner: rsesek@chromium.org
Status: Assigned (was: Untriaged)
Automatically assigning owner based on suspected regression changelist https://chromium.googlesource.com/chromium/src/+/0232eb54cff5578b3a4365a58e5db788368c6251 (crash_keys: Convert keys in //components/discardable_memory to the new API.).

If this is incorrect, please remove the owner and apply the Test-Predator-Wrong-CLs label.

Comment 3 by rsesek@chromium.org, Dec 14 2017

This is a known issue with the Breakpad/Crashpad SimpleStringDictionary implementation. The global object in which the key/value pairs are stored is not thread safe. The solution is to transition to Crashpad Annotations (https://docs.google.com/document/d/1IeWeuvwhYqEK66V4-4TChU6UmdFbuEr5lgu-WvlLlOA/edit, note "Thread Safety" under "Design Problems"), which is happening on Mac and Windows in  issue 598854 . Indeed, the CL pinpointed by #2 actually fixes this data race on Mac/Win.

This specific issue is not new, either. The difference is that as part of  issue 598854 , crash keys no longer need to be pre-initialized in order to be used. That meant in tests, setting crash keys was actually a no-op, so the code path was never tested (and the data race never observed). Because crash keys auto-initialize now, setting a crash key actually has an effect in tests, so the data race was observed.

Tl;dr: Nothing to do here at the moment. This is fixed on Mac/Win, but it's dependent no Crashpad for Linux to replace Breakpad.

Comment 4 by rsesek@chromium.org, Dec 20 2017

Cc: pnangunoori@chromium.org rsesek@chromium.org
 Issue 796475  has been merged into this issue.
Project Member

Comment 5 by ClusterFuzz, Dec 21 2017

Labels: M-65 ReleaseBlock-Beta ClusterFuzz-Top-Crash
Testcase 6004094081957888 is a top crash on ClusterFuzz for linux platform. Please prioritize fixing this crash.

Marking this crash as a Beta release blocker.

If this is incorrect, please add ClusterFuzz-Wrong label and re-open the issue.

Comment 6 by rsesek@chromium.org, Dec 21 2017

Labels: -ReleaseBlock-Beta -M-65 ClusterFuzz-Wrong
Project Member

Comment 7 by ClusterFuzz, Dec 22 2017

ClusterFuzz has detected this issue as fixed in range 525864:525869.

Detailed report: https://clusterfuzz.com/testcase?key=6004094081957888

Fuzzer: ifratric-browserfuzzer-v3
Job Type: linux_tsan_chrome_mp
Platform Id: linux

Crash Type: Data race READ 1
Crash Address: 0x7bb4000005e8
Crash State:
  google_breakpad::NonAllocatingMap<40ul, 128ul, 200ul>::SetKeyValue
  crash_reporter::internal::CrashKeyStringImpl::Set
  discardable_memory::ClientDiscardableSharedMemoryManager::MemoryUsageChanged
  
Sanitizer: thread (TSAN)

Regressed: https://clusterfuzz.com/revisions?job=linux_tsan_chrome_mp&range=523781:523782
Fixed: https://clusterfuzz.com/revisions?job=linux_tsan_chrome_mp&range=525864:525869

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=6004094081957888

See https://github.com/google/clusterfuzz-tools for more information.

If you suspect that the result above is incorrect, try re-doing that job on the test case report page.

Comment 8 by rsesek@chromium.org, Dec 28 2017

Cc: kkaluri@chromium.org
 Issue 797830  has been merged into this issue.

Comment 9 by rsesek@chromium.org, Jan 24 2018

Cc: brajkumar@chromium.org
 Issue 805191  has been merged into this issue.
 Issue 820712  has been merged into this issue.
 Issue 598854  resolves the thread safety issues for anything using Crashpad, which currently is Windows and Mac. Linux still uses Breakpad, so this race is expected until Crashpad is ready there.
Project Member

Comment 13 by bugdroid1@chromium.org, Mar 20 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/55978c043b1c30b77739e2fcd3822eae438cfb13

commit 55978c043b1c30b77739e2fcd3822eae438cfb13
Author: Alexander Potapenko <glider@google.com>
Date: Tue Mar 20 15:47:02 2018

Suppress data races in crash_reporter::InitializeCrashKeys()

BUG=794920
TBR=rsesek@chromium.org

Change-Id: I3ea15492e4f08a0879fdd0c5476a09bcf5db6836
Reviewed-on: https://chromium-review.googlesource.com/970477
Reviewed-by: Alexander Potapenko <glider@chromium.org>
Reviewed-by: Robert Sesek <rsesek@chromium.org>
Commit-Queue: Alexander Potapenko <glider@chromium.org>
Cr-Commit-Position: refs/heads/master@{#544381}
[modify] https://crrev.com/55978c043b1c30b77739e2fcd3822eae438cfb13/build/sanitizers/tsan_suppressions.cc

Project Member

Comment 14 by bugdroid1@chromium.org, Mar 21 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/92c168a6fec7480404fc18c369246300cf271a78

commit 92c168a6fec7480404fc18c369246300cf271a78
Author: Alexander Potapenko <glider@google.com>
Date: Wed Mar 21 16:14:59 2018

Fix the TSan suppression for issue 794920

TSan suppressions can't be applied to memory allocation stacks in
the reports, only to memory access stacks.
Therefore suppress the access that occurs in base::debug::SetCrashKeyString()

BUG=794920
TBR=rsesek@chromium.org

Change-Id: I607bf4f13af69eade9a5c745d556a615f4c2d1c2
Reviewed-on: https://chromium-review.googlesource.com/972984
Reviewed-by: Alexander Potapenko <glider@chromium.org>
Commit-Queue: Alexander Potapenko <glider@chromium.org>
Cr-Commit-Position: refs/heads/master@{#544724}
[modify] https://crrev.com/92c168a6fec7480404fc18c369246300cf271a78/build/sanitizers/tsan_suppressions.cc

Project Member

Comment 15 by ClusterFuzz, Mar 30 2018

Labels: Fuzz-Blocker M-67 ReleaseBlock-Beta
This crash occurs very frequently on linux platform and is likely preventing the fuzzer ochang_domfuzzer from making much progress. Fixing this will allow more bugs to be found.

Marking this bug as a blocker for next Beta release.

If this is incorrect, please add ClusterFuzz-Wrong label and remove the ReleaseBlock-Beta label.
Cc: -pnangunoori@chromium.org -rsesek@chromium.org glider@chromium.org infe...@chromium.org
Labels: -ReleaseBlock-Beta -M-67
Re: #15: Is the suppression in #14 not working? Fixing this is not tractable at the moment, but Crashpad for Linux and Android is coming soon (probably EOQ2).
 Issue 827040  has been merged into this issue.
Cc: mfo...@chromium.org roc...@chromium.org rsesek@chromium.org
 Issue 829543  has been merged into this issue.
 Issue 833055  has been merged into this issue.
Yes suppression is not working. Probably need to add
crash_reporter::internal::CrashKeyStringImpl::Set ?

Cc: -mfo...@chromium.org
Project Member

Comment 22 by bugdroid1@chromium.org, Apr 24 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/78340952136237b8bf790be02f037fa5c814a38a

commit 78340952136237b8bf790be02f037fa5c814a38a
Author: Abhishek Arya <inferno@chromium.org>
Date: Tue Apr 24 15:59:45 2018

Add another TSan suppression for issue 794920.

TBR=glider@chromium.org

Bug: 794920
Change-Id: I46fd3e886acb2db047655c055f0b887971a8c09b
Reviewed-on: https://chromium-review.googlesource.com/1025968
Reviewed-by: Abhishek Arya <inferno@chromium.org>
Commit-Queue: Abhishek Arya <inferno@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553143}
[modify] https://crrev.com/78340952136237b8bf790be02f037fa5c814a38a/build/sanitizers/tsan_suppressions.cc

Cc: -roc...@chromium.org rockot@google.com
Project Member

Comment 24 by ClusterFuzz, Dec 1

ClusterFuzz has detected this issue as fixed in range 528993:528995.

Detailed report: https://clusterfuzz.com/testcase?key=6004094081957888

Fuzzer: ifratric-browserfuzzer-v3
Job Type: linux_tsan_chrome_mp
Platform Id: linux

Crash Type: Data race READ 1
Crash Address: 0x7bb4000005e8
Crash State:
  google_breakpad::NonAllocatingMap<40ul, 128ul, 200ul>::SetKeyValue
  crash_reporter::internal::CrashKeyStringImpl::Set
  discardable_memory::ClientDiscardableSharedMemoryManager::MemoryUsageChanged
  
Sanitizer: thread (TSAN)

Regressed: https://clusterfuzz.com/revisions?job=linux_tsan_chrome_mp&range=523781:523782
Fixed: https://clusterfuzz.com/revisions?job=linux_tsan_chrome_mp&range=528993:528995

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=6004094081957888

See https://github.com/google/clusterfuzz-tools for more information.

If you suspect that the result above is incorrect, try re-doing that job on the test case report page.

Sign in to add a comment