New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 701162 link

Starred by 2 users

Issue metadata

Status: WontFix
Owner:
Last visit > 30 days ago
Closed: Mar 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

chrome crash reports still busted for certain types of crahses

Project Member Reported by diand...@chromium.org, Mar 13 2017

Issue description

As you can see from:

https://chromium-review.googlesource.com/c/442284/

...and the associated bug, crash reports were busted for a while in a pretty major way.  

That was fixed.  You can see the M57 pick landed Feb 16 at:

  https://chromium-review.googlesource.com/c/443308

...but we are still seeing a certain class of crashes that has no good crash reports.  Specifically see:

a767e034a0000000

When I look at that report I still see a stack quality of 18% and I see "Missing symbols: chrome".  The crash is from 9202.51.0 on 2017-03-08 

---

...but if I download this type of crash and symbolize manually and go into GDB then I get a wonderfully nice stack crawl.

(gdb) bt
#0  0xece0fb36 in ?? () from r/lib/libc.so.6
#1  0xece1ecd4 in raise () from r/lib/libc.so.6
#2  0xece1fc36 in abort () from r/lib/libc.so.6
#3  0xb771808e in base::debug::BreakDebugger() () at ../../../../../../../home/chrome-bot/chrome_root/src/base/debug/debugger_posix.cc:251
#4  0xb77248b2 in logging::LogMessage::~LogMessage() () at ../../../../../../../home/chrome-bot/chrome_root/src/base/logging.cc:759
#5  0xb7739630 in base::(anonymous namespace)::OnNoMemorySize(unsigned int) [clone .constprop.4] () at ../../../../../../../home/chrome-bot/chrome_root/src/base/process/memory_linux.cc:35
#6  0xb6296c82 in operator new(unsigned int, std::nothrow_t const&) () at ../../../../../../../home/chrome-bot/chrome_root/src/base/allocator/allocator_shim.cc:68
#7  0xb82b2476 in leveldb::ReadBlock(leveldb::RandomAccessFile*, leveldb::ReadOptions const&, leveldb::BlockHandle const&, leveldb::BlockContents*) ()
    at ../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/table/format.cc:124
#8  0xb82ad92a in leveldb::Table::BlockReader(void*, leveldb::ReadOptions const&, leveldb::Slice const&) ()
    at ../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/table/table.cc:189
#9  0xb82aecee in leveldb::(anonymous namespace)::TwoLevelIterator::InitDataBlock() () at ../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/table/two_level_iterator.cc:165
#10 0xb82aeef4 in leveldb::(anonymous namespace)::TwoLevelIterator::Seek(leveldb::Slice const&) ()
    at ../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/table/two_level_iterator.cc:93
#11 0xb82ad422 in leveldb::IteratorWrapper::Seek(leveldb::Slice const&) () at ../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/table/iterator_wrapper.h:47
#12 0xb82ad460 in leveldb::(anonymous namespace)::MergingIterator::Seek(leveldb::Slice const&) () at ../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/table/merger.cc:53
#13 0xb82a1ea0 in leveldb::(anonymous namespace)::DBIter::Seek(leveldb::Slice const&) () at ../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/db/db_iter.cc:280
#14 0xb6e2f934 in content::LevelDBIteratorImpl::Seek(base::BasicStringPiece<std::string> const&) ()
    at ../../../../../../../home/chrome-bot/chrome_root/src/content/browser/indexed_db/leveldb/leveldb_iterator_impl.cc:45
#15 0xb6e303ca in content::LevelDBTransaction::TransactionIterator::Seek(base::BasicStringPiece<std::string> const&) ()
    at ../../../../../../../home/chrome-bot/chrome_root/src/content/browser/indexed_db/leveldb/leveldb_transaction.cc:230
#16 0xb6e056c0 in content::IndexedDBBackingStore::Cursor::FirstSeek(leveldb::Status*) () at ../../../../../../../home/chrome-bot/chrome_root/src/content/browser/indexed_db/indexed_db_backing_store.cc:3182
#17 0xb6e09778 in content::IndexedDBBackingStore::OpenObjectStoreCursor(content::IndexedDBBackingStore::Transaction*, long long, long long, content::IndexedDBKeyRange const&, blink::WebIDBCursorDirection, leveldb::Status*) () at ../../../../../../../home/chrome-bot/chrome_root/src/content/browser/indexed_db/indexed_db_backing_store.cc:3969
#18 0xb6e1a138 in content::IndexedDBDatabase::OpenCursorOperation(std::unique_ptr<content::IndexedDBDatabase::OpenCursorOperationParams, std::default_delete<content::IndexedDBDatabase::OpenCursorOperationParams> >, content::IndexedDBTransaction*) () at ../../../../../../../home/chrome-bot/chrome_root/src/content/browser/indexed_db/indexed_db_database.cc:1536
#19 0xb6e1c5f8 in base::internal::Invoker<base::internal::BindState<leveldb::Status (content::IndexedDBDatabase::*)(std::unique_ptr<content::IndexedDBDatabase::OpenCursorOperationParams, std::default_delete<content::IndexedDBDatabase::OpenCursorOperationParams> >, content::IndexedDBTransaction*), scoped_refptr<content::IndexedDBDatabase>, base::internal::PassedWrapper<std::unique_ptr<content::IndexedDBDatabase::OpenCursorOperationParams, std::default_delete<content::IndexedDBDatabase::OpenCursorOperationParams> > > >, leveldb::Status (content::IndexedDBTransaction*)>::Run(base::internal::BindStateBase*, content::IndexedDBTransaction*&&) () at ../../../../../../../home/chrome-bot/chrome_root/src/base/bind_internal.h:214
#20 0xb6e2d200 in content::IndexedDBTransaction::ProcessTaskQueue() () at ../../../../../../../home/chrome-bot/chrome_root/src/base/callback.h:85
#21 0xb63fd84c in base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) () at ../../../../../../../home/chrome-bot/chrome_root/src/base/callback.h:68
#22 0xb63f2212 in base::MessageLoop::DoWork() () at ../../../../../../../home/chrome-bot/chrome_root/src/base/message_loop/message_loop.cc:421
#23 0xb63f2608 in base::MessagePumpDefault::Run(base::MessagePump::Delegate*) () at ../../../../../../../home/chrome-bot/chrome_root/src/base/message_loop/message_pump_default.cc:33
#24 0xb773e02a in base::RunLoop::Run() () at ../../../../../../../home/chrome-bot/chrome_root/src/base/run_loop.cc:37
#25 0xb7750c30 in base::Thread::ThreadMain() () at ../../../../../../../home/chrome-bot/chrome_root/src/base/threading/thread.cc:328
#26 0xb774ed5e in base::(anonymous namespace)::ThreadFunc(void*) () at ../../../../../../../home/chrome-bot/chrome_root/src/base/threading/platform_thread_posix.cc:71
#27 0xed4385fa in ?? () from r/lib/libpthread.so.0


---

What's wrong?
 

Comment 1 by vapier@chromium.org, Mar 14 2017

Cc: thestig@chromium.org
here's the build log for kevin for 9202.51.0:
https://luci-milo.appspot.com/buildbot/chromeos_release/kevin-release%20release-R57-9202.B/45

in the DebugSymbols stage, we see chrome was dumped correctly/fully:
03:26:58: INFO: Dumped /build/kevin/opt/google/chrome/chrome as chrome : 205ABEC3E744B4A37A33AD3FD957C2570

and uploaded correctly/fully:
03:45:27: INFO: Uploading symbol_file: chrome/205ABEC3E744B4A37A33AD3FD957C2570/chrome.sym
03:54:18: INFO: upload of  552457614 bytes took 0:08:51.104606

if we look at the crash report page, i don't see a debug/module id listed at all for chrome:
https://crash.corp.google.com/browse?q=reportid=%27a767e034a0000000%27#5
all the entries have the debug id field blank.

and indeed, when i grab that minidump and dump it, the debug field is empty:
$ minidump_dump upload_file_minidump-a767e034a0000000.dmp | less
...
module[0]
MDRawModule
...
  base_of_image                   = 0xb5e08000
  size_of_image                   = 0x509a000
  checksum                        = 0x0
  time_date_stamp                 = 0x0 1970-01-01 00:00:00
  module_name_rva                 = 0x80130
  version_info.signature          = 0x0
  version_info.struct_version     = 0x0
  version_info.file_version       = 0x0:0x0
  version_info.product_version    = 0x0:0x0
  version_info.file_flags_mask    = 0x0
  version_info.file_flags         = 0x0
  version_info.file_os            = 0x0
  version_info.file_type          = 0x0
  version_info.file_subtype       = 0x0
  version_info.file_date          = 0x0:0x0
  cv_record.data_size             = 0
  cv_record.rva                   = 0x0
  misc_record.data_size           = 0
  misc_record.rva                 = 0x0
  (code_file)                     = "/opt/google/chrome/chrome"
  (code_identifier)               = "id"
  (cv_record)                     = (null)
  (misc_record)                   = (null)
  (debug_file)                    = ""
  (debug_identifier)              = ""
  (version)                       = ""

the system crash-reporter doesn't handle core->minidumps w/Chrome ... it waits for Chrome itself to run things through breakpad to produce a minidump which it then hands directly to crash-reporter.  so if we've got a broken minidump here, it's because Chrome handed us a broken one.

Comment 2 by ivanpe@chromium.org, Mar 14 2017

This is likely an issue in the Breakpad client for Linux.  It only affects about 3% of the crash reports for this particular version of Chrome:

https://dashboards.corp.google.com/edit/google::_a0806b44_bb6a_4325_8e34_459b59c89c4f

Screenshot: https://screenshot.googleplex.com/tuRZ4XYG6ZN.png

Given the small percentage of affected reports and the fact that Breakpad will be replaced with a Crashpad client by the end of this year, I think we should not spend too much time on this issue.

Resolving as won't fix.

Comment 3 by ivanpe@chromium.org, Mar 14 2017

Status: WontFix (was: Untriaged)

Comment 4 by vapier@chromium.org, Mar 14 2017

> Breakpad will be replaced with a Crashpad client by the end of this year

for Linux ?  have a link ?

Comment 5 by ivanpe@chromium.org, Mar 14 2017

Here is a link to Crashpad 2016-1017 roadmap: https://docs.google.com/document/d/1ZjPuKe-2TPyfcy5S9WtXIJAMTb_koMY6IV9BOqkyF4A/edit#heading=h.mlfzhjfiwd62

Joshua Peraza and Mark Mentovai are currently working on the Crashpad client for Android.  The Linux client should follow soon after that.
@6: I think that's actually a separate issue and is the topic of:

- bug #665083: Chrome_ChromeOS: Crash Report - __divdi3, __clock_gettime, ...
- bug #702388: minidumps don't seem to handle stack overflows well

---

Actually, my analysis in bug #665083 (comment 106) indicates that this may account for some of the things claimed as "TCMalloc corruption", too.

Comment 8 by ivanpe@chromium.org, Mar 16 2017

Hi Zel, the 3% I was referring to are about the issue described in #1.

I can see that many of the top clusters in the above URL (#6) have broken stack traces.  I haven't looked at these in much detail yet, however according to bug #665083 the stack of the main thread seems busted which explains why Breakpad cannot walk the stack and symbolize its contents.  The stack traces for the rest of the treads looks good though.

Comment 9 by ivanpe@chromium.org, Mar 16 2017

Status: WontFix (was: Assigned)
Let's close this and discuss the other issues in their corresponding bugs.
Labels: Postmortem-Followup

Sign in to add a comment