Investigate Memory Corruption in Histograms |
|||||
Issue descriptionVersion: Many OS: Windows (seems limited to versions of Windows) https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.component%3D%27src%2Fbase%2Fmetrics%27%20AND%20custom_data.ChromeCrashProto.malware_verdict%3Dfalse%20AND%20product.version%20like%20%275_._.%25%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27base%3A%3AHistogramSnapshotManager%3A%3APrepareSamples%27%20OMIT%20RECORD%20IF%20SUM(custom_data.ChromeCrashProto.modules.third_party_paths%20like%20%27%25kaspersky%25%27)%20%3E%200&ignore_case=false&enable_rewrite=false&omit_field_name=&omit_field_value=&omit_field_opt=#samplereports:5,productversion,magicsignature,-filepath,-author,-changelist,-magicsignature2,processuptime,osversion,experiments:100,3rdparty These are likely due to other issues overwriting memory but might be worth some investigation anyway.
,
Jul 13 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/38c5e05f22bc205fc270478546956e73b8e13407 commit 38c5e05f22bc205fc270478546956e73b8e13407 Author: bcwhite <bcwhite@chromium.org> Date: Wed Jul 13 03:58:09 2016 Fix capture of debug information when corruption is detected. BUG=600717 Review-Url: https://codereview.chromium.org/2148503002 Cr-Commit-Position: refs/heads/master@{#404992} [modify] https://crrev.com/38c5e05f22bc205fc270478546956e73b8e13407/base/metrics/histogram_snapshot_manager.cc
,
Jul 13 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/38c5e05f22bc205fc270478546956e73b8e13407 commit 38c5e05f22bc205fc270478546956e73b8e13407 Author: bcwhite <bcwhite@chromium.org> Date: Wed Jul 13 03:58:09 2016 Fix capture of debug information when corruption is detected. BUG=600717 Review-Url: https://codereview.chromium.org/2148503002 Cr-Commit-Position: refs/heads/master@{#404992} [modify] https://crrev.com/38c5e05f22bc205fc270478546956e73b8e13407/base/metrics/histogram_snapshot_manager.cc
,
Sep 23 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b7a682b6c61394faa2e2e6e8f31c96506079886e commit b7a682b6c61394faa2e2e6e8f31c96506079886e Author: bcwhite <bcwhite@chromium.org> Date: Fri Sep 23 14:02:04 2016 Try to fix gathering of ranges_ptr plus capture checksums. BUG=600717 Review-Url: https://codereview.chromium.org/2362113002 Cr-Commit-Position: refs/heads/master@{#420613} [modify] https://crrev.com/b7a682b6c61394faa2e2e6e8f31c96506079886e/base/metrics/histogram_snapshot_manager.cc
,
Nov 23 2016
https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.component%3D%27src%2Fbase%2Fmetrics%27%20AND%20custom_data.ChromeCrashProto.malware_verdict%3Dfalse%20AND%20product.version%20%3E%3D%20%2755.2870%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27base%3A%3AHistogramSnapshotManager%3A%3APrepareSamples%27%20AND%20custom_data.ChromeCrashProto.channel%3D%27canary%27%20OMIT%20RECORD%20IF%20SUM(custom_data.ChromeCrashProto.modules.third_party_paths%20like%20%27%25kaspersky%25%27)%20%3E%200&ignore_case=false&enable_rewrite=false&omit_field_name=&omit_field_value=&omit_field_opt=&stbtiq=&reportid=41465ed700000000&index=0#0 Found 16 bytes of memory corruption: 00000260`774c3d50 03 00 00 00 10 00 00 00-01 00 00 00 10 00 00 00 00000260`774c3d60 01 00 20 00 10 00 00 00-03 00 00 00 10 00 00 00 Or, as a series of 32-bit words: 00000260`774c3d50 00000003 00000010 00000001 00000010 00000260`774c3d60 00200001 00000010 00000003 00000010
,
Nov 24 2016
Seems that crash has overwolf module in the process - maybe it's to blame?
,
Dec 22 2016
https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.component%3D%27src%2Fbase%2Fmetrics%27%20AND%20custom_data.ChromeCrashProto.malware_verdict%3Dfalse%20AND%20product.version%20%3E%3D%20%2755.2870%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27base%3A%3AHistogramSnapshotManager%3A%3APrepareSamples%27%20AND%20custom_data.ChromeCrashProto.channel%3D%27canary%27%20OMIT%20RECORD%20IF%20SUM(custom_data.ChromeCrashProto.modules.third_party_paths%20like%20%27%25kaspersky%25%27)%20%3E%200&ignore_case=false&enable_rewrite=false&omit_field_name=&omit_field_value=&omit_field_opt=&stbtiq=&reportid=e82bb35880000000&index=0 Another crash, this time with a single 0xA8 byte written to the "ranges" array. No overwolf module listed.
,
Jan 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9d32d0bc1e2fbbad5ff6ba372ff386579d006cbc commit 9d32d0bc1e2fbbad5ff6ba372ff386579d006cbc Author: bcwhite <bcwhite@chromium.org> Date: Tue Jan 10 17:50:15 2017 Keep ranges_ptr variable for debug, not just the data it points to. BUG=600717 Review-Url: https://codereview.chromium.org/2628583002 Cr-Commit-Position: refs/heads/master@{#442627} [modify] https://crrev.com/9d32d0bc1e2fbbad5ff6ba372ff386579d006cbc/base/metrics/histogram_snapshot_manager.cc
,
Apr 24 2017
Went through another bunch of these. Many of the 64-bit crashes didn't save the memory areas reference at the point of crash even when the pointers were debug::Alias'd. I found corrupted map trees and bit flips. Other checksum problems weren't analyze-able because the minidump didn't contain the full array, including where the error was. The bad map trees were often nullptrs, which is odd for random errors though could be a store-trampler.
,
Apr 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1c8b1fb10b26ef237db82604f611bc4396c45f17 commit 1c8b1fb10b26ef237db82604f611bc4396c45f17 Author: bcwhite <bcwhite@chromium.org> Date: Mon Apr 24 20:58:35 2017 Add ThreadChecker to HistogramSnapshotManager. BUG=600717 Review-Url: https://codereview.chromium.org/2836993002 Cr-Commit-Position: refs/heads/master@{#466760} [modify] https://crrev.com/1c8b1fb10b26ef237db82604f611bc4396c45f17/base/metrics/histogram_snapshot_manager.cc [modify] https://crrev.com/1c8b1fb10b26ef237db82604f611bc4396c45f17/base/metrics/histogram_snapshot_manager.h
,
Apr 25 2017
As an idea... It would be possible for most Histogram types to be able to rebuild their "ranges" tables when corruption is detected. Better than crashing? I wonder, too, if those tables could be marked "read only" so that store-tramplers would crash at the code location of the trampling.
,
Apr 26 2017
Agree that those are good ideas. Still, I think ultimately memory stompers are likely caused by other code and I'd look towards tools like SyzyASAN to identifying those and having devs fix them. (Given the above, I don't thinke this is worth spending a lot of time on this quarter.)
,
May 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/947514553066c623a85712d05c3a01bd1bcbbffc commit 947514553066c623a85712d05c3a01bd1bcbbffc Author: bcwhite <bcwhite@chromium.org> Date: Tue May 09 04:01:17 2017 Add concurrency check to HistogramSnapshotManager. Using ThreadChecker causes problems when outside code is doing its own synchronization between multiple calling threads so remove that and add an atomic to do a run-time concurrency CHECK. This will likely be removed in the future once it's well assured that concurrent access is not the cause of the corrupted data structures. Also, make known_histograms_ member "const" as it should have been from the beginning. BUG= 719448 , 600717 Review-Url: https://codereview.chromium.org/2871663003 Cr-Commit-Position: refs/heads/master@{#470178} [modify] https://crrev.com/947514553066c623a85712d05c3a01bd1bcbbffc/base/metrics/histogram_snapshot_manager.cc [modify] https://crrev.com/947514553066c623a85712d05c3a01bd1bcbbffc/base/metrics/histogram_snapshot_manager.h
,
Sep 21 2017
Users experienced this crash on the following builds: Android Dev 63.0.3214.0 - 0.24 CPM, 8 reports, 8 clients (signature base::HistogramSnapshotManager::PrepareSamples) If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates. - Go/Fracas
,
Feb 26 2018
Users experienced this crash on the following builds: Linux Beta 65.0.3325.88 - 0.25 CPM, 1 reports, 1 clients (signature base::HistogramSnapshotManager::PrepareSamples) If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates. - Go/Fracas
,
Mar 30 2018
Users experienced this crash on the following builds: Win Canary 67.0.3383.0 - 0.08 CPM, 1 reports, 1 clients (signature base::HistogramSnapshotManager::PrepareSamples) If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates. - Go/Fracas |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by bugdroid1@chromium.org
, Apr 5 2016