I'm not familiar with the ChromeOS-specific scripts, but I know how to do it manually.

If you can pick a specific crash and point me to a system image from which I can extract the chrome and libc-2.19.so binary, I can give it a whirl and see if it works for me.

CCing more people who might know something about it

for now, let's assign to the build team to figure it out

i don't think the build team is responsible for this

however, the bug report doesn't contain details on what the actual problem is.  oshima@ didn't describe what he was trying to do, and the referenced crash reports don't exist anymore.

please post concrete steps as to what you're trying to do and what isn't working.

The steps are in the link that oshima@ posted in the original report:
- take _any_ chrome crash
- download the minidump
- dump2core it into a core dump
- open the core dump in gdb
- the stack trace is incomplete/funny

the FAQ he linked to contains a number of things.  you can't just say "try everything in that page" -- that's unreasonable, and trivial to make people waste time on.  if i grabbed a diff crash report and it happened to work, then that's not helpful.

if it's happening on any crash, can you link to a crash/ report now ?  preferably one we can attach here for posterity which means generating one using a test account.

I am trying to debug 
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_ChromeOS%27%20%0AAND%20CrashedStackTrace.StackFrame.FunctionName%20CONTAINS%20%27media%3A%3A%27%20%0AAND%20product.Version%3D%2753.0.2785.144%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27media%3A%3AVaapiWrapper%3A%3ABlitSurface%27%20AND%20ReportID%3D%2714a7b0b900000000%27&ignore_case=true&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#3 right now

I would basically point you to https://chromium.googlesource.com/chromium/src/+/master/docs/linux_minidump_to_core.md which is what I do. I've used the same steps successfully on ChromeOS, as seen in bug 620239.

The above is all I got. Hope that helps.

(cr) marcheu@marchesin ~/trunk/src/bug658895 $ /build/samus/usr/bin/minidump-2-core upload_file_minidump-14a7b0b900000000.dmp  > corefile 
(cr) marcheu@marchesin ~/trunk/src/bug658895 $ x86_64-cros-linux-gnu-gdb ./chrome
GNU gdb (Chromium OS 7.11.20160511 vanilla) 7.11
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=x86_64-cros-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://crbug.com/new>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./chrome...(no debugging symbols found)...done.
(gdb) core-file corefile 
[New LWP 1662]
[New LWP 1787]
[New LWP 1788]
[New LWP 1789]
[New LWP 17569]
warning: Could not load shared library symbols for 60 libraries, e.g. linux-vdso.so.1.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
Failed to read a valid object file image from memory.
Core was generated by `/opt/google/chrome/chrome --type=gpu-process --channel=1015.0.731876943 --mojo-'.
#0  0x00007f94498fe090 in ?? ()
[Current thread is 1 (LWP 1662)]
(gdb) bt
#0  0x00007f94498fe090 in ?? ()
#1  0x0000259a4d43bad0 in ?? ()
#2  0x0000000000000000 in ?? ()
(gdb)

the raw addresses themselves are WAI -- you didn't add symbol files for them.  both docs explain how that's done.

0x00007f94498fe090 is the correct starting address.  i suspect gdb needs the debugging symbols to be able to unwind the stack properly (via CFI), otherwise it's left with plain old sniffing which can be flaky.

You need to do the steps in: https://chromium.googlesource.com/chromium/src/+/master/docs/linux_minidump_to_core.md#Advanced-module-loading

For crash 14a7b0b900000000, you need to do the address calculation (and eventually add-symbol-file) for libdrm_intel.so.1.0.0, i965_drv_video.so, libva.so.1.3900.0, and chrome in that order, depending on how far up the stack you want to go.

we can prob set up some links to make the new --sobasedir flag easier to use.  alternatively, we could make it smarter to use standard /usr/lib/debug/.build-id/ since that's what all Linux systems have standardized on.

@14: I have never needed this in the past. But here is the output (I'm only loading the first one so show that it doesn't work):

(cr) marcheu@marchesin ~/trunk/src/bug658895 $ x86_64-cros-linux-gnu-gdb ./chrome
GNU gdb (Chromium OS 7.11.20160511 vanilla) 7.11
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=x86_64-cros-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://crbug.com/new>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./chrome...(no debugging symbols found)...done.
(gdb) symbol-file debug/usr/lib64/libdrm_intel.so.1.0.0.debug 
Reading symbols from debug/usr/lib64/libdrm_intel.so.1.0.0.debug...done.
(gdb) core-file corefile
[New LWP 1662]
[New LWP 1787]
[New LWP 1788]
[New LWP 1789]
[New LWP 17569]
Failed to read a valid object file image from memory.
Core was generated by `/opt/google/chrome/chrome --type=gpu-process --channel=1015.0.731876943 --mojo-'.
#0  0x00007f94498fe090 in ?? ()
[Current thread is 1 (LWP 1662)]
(gdb) bt
#0  0x00007f94498fe090 in ?? ()
#1  0x0000259a4d43bad0 in ?? ()
#2  0x0000000000000000 in ?? ()

Did gdb correctly figure out what address libdrm_intel.so.1.0.0 is loaded at? I think "info sharedlibrary" can tell you. For your crash, it should be loaded somewhere right after 7f94498f5000.

Hmm ok, if you do the relocation by hand it works. I'm 100% sure this used to work. What broke? Debugging like this is really painful.

I don't know. It's never worked on Linux AFAIK, so the rare times when I debugged on ChromeOS, I've always assumed it won't work and done it the slow way.

Also how does one load a shared library with this flow? We have separate debug symbols, so I can only load either code or symbols

Load the library. It should have a .gnu_debuglink for gdb to follow.

i'm fairly certain it's never worked automatically.  the Crash FAQ has long had these sections for how to load symbol files at the right addresses by hand.  but it's been one of those bugs that hasn't been enough of a pain point to dedicate resources to improving.

i think the debuglink logic is being defeated by the --sobasedir logic in breakpad's minidump-2-core.  it's pretty limited and doesn't allow for standard sysroot usage.

and indeed, once i get all the right symbols into gdb, i get a backtrace that lines up well with the crash server:

(gdb) bt
#0  drm_intel_gem_bo_bucket_for_size (size=0x80000, bufmgr_gem=0x259a4d43b800) at ../../libdrm-2.4.66/intel/intel_bufmgr_gem.c:410
#1  drm_intel_gem_bo_unreference_final (bo=bo@entry=0x259a4b319600, time=0x11c33) at ../../libdrm-2.4.66/intel/intel_bufmgr_gem.c:1363
#2  0x00007f94498fe269 in drm_intel_gem_bo_unreference (bo=0x259a4b319600) at ../../libdrm-2.4.66/intel/intel_bufmgr_gem.c:1405
#3  0x00007f9448fbbe5c in intel_batchbuffer_reset (batch=batch@entry=0x259a4b94b0a0, buffer_size=0x80000) at ../../libva-intel-driver-1.7.0/src/intel_batchbuffer.c:56
#4  0x00007f9448fbc0de in intel_batchbuffer_flush (batch=batch@entry=0x259a4b94b0a0) at ../../libva-intel-driver-1.7.0/src/intel_batchbuffer.c:148
#5  0x00007f9448fad442 in gen8_pp_object_walker (ctx=ctx@entry=0x259a45f92800, pp_context=pp_context@entry=0x259a45672220) at ../../libva-intel-driver-1.7.0/src/gen8_post_processing.c:1409
#6  0x00007f9448fad6c8 in gen8_pp_pipeline_setup (pp_context=0x259a45672220, ctx=0x259a45f92800) at ../../libva-intel-driver-1.7.0/src/gen8_post_processing.c:1427
#7  gen8_post_processing (ctx=0x259a45f92800, pp_context=0x259a45672220, src_surface=<optimized out>, src_rect=<optimized out>, dst_surface=<optimized out>, dst_rect=<optimized out>, pp_index=0xf, filter_param=0x0) at ../../libva-intel-driver-1.7.0/src/gen8_post_processing.c:1455
#8  0x00007f9448fa9f2f in i965_post_processing_internal (filter_param=0x0, pp_index=<optimized out>, dst_rect=0x7fff38f522b0, dst_surface=0x7fff38f522d0, src_rect=0x7fff38f522a0, src_surface=0x7fff38f522c0, pp_context=0x259a45672220, ctx=0x259a45f92800) at ../../libva-intel-driver-1.7.0/src/i965_post_processing.c:4774
#9  i965_proc_picture_fast (proc_state=0x259a4e1b2038, proc_state=0x259a4e1b2038, proc_context=0x259a45672200, ctx=0x259a45f92800) at ../../libva-intel-driver-1.7.0/src/i965_post_processing.c:5975
#10 i965_proc_picture (ctx=ctx@entry=0x259a45f92800, profile=profile@entry=VAProfileNone, codec_state=codec_state@entry=0x259a4e1b2038, hw_context=0x259a45672200) at ../../libva-intel-driver-1.7.0/src/i965_post_processing.c:6001
#11 0x00007f9448f7bd0a in gen75_vpp_fmt_cvt (hw_context=0x259a5075a640, codec_state=0x259a4e1b2038, profile=VAProfileNone, ctx=0x259a45f92800) at ../../libva-intel-driver-1.7.0/src/gen75_picture_process.c:61
#12 gen75_proc_picture (ctx=0x259a45f92800, profile=VAProfileNone, codec_state=0x259a4e1b2038, hw_context=0x259a5075a640) at ../../libva-intel-driver-1.7.0/src/gen75_picture_process.c:308
#13 0x00007f9448a853ef in vaEndPicture (dpy=0x259a4e0f6f00, context=0x2000000) at ../../libva-1.7.0/va/va.c:1228

i've posted these CLs to make minidump-2-core have a saner (imo) default:
https://chromium-review.googlesource.com/402908
https://chromium-review.googlesource.com/402909

which means now you can download the rootfs and the debug tarball and do something like:
$ minidump-2-core upload_file_minidump-14a7b0b900000000.dmp > core
$ mkdir rootfs
$ sudo losetup -P /dev/loop7 chromeos_8530.93.0_auron-paine_recovery_stable-channel_paine-mp-v2.bin
$ sudo mount -o ro /dev/loop7p3 rootfs
$ tar xf debug.tgz
$ x86_64-cros-linux-gnu-gdb -q -ex 'set debug-file-directory ./debug/' -ex 'set sysroot ./rootfs/' \
  ./rootfs/opt/google/chrome/chrome ./core
<symbols be found>

i'm going to drop the labels as it doesn't seem like a regression, and it's possible with the tools today to get the right behavior.

The following revision refers to this bug:
  https://chromium.googlesource.com/breakpad/breakpad/+/ed7dcced1945864c94dcfd358a64e23c812fd0dc

commit ed7dcced1945864c94dcfd358a64e23c812fd0dc
Author: Mike Frysinger <vapier@chromium.org>
Date: Tue Oct 25 06:36:38 2016

minidump-2-core: rewrite argument processing

This uses the same general framework as other minidump tools by using
getopt to parse command line options, and then passing the parsed state
around as a struct rather than via globals.

This does change the --sobasedir flag to -S because we don't support
getopt_long anywhere in the tree.  Unfortunate, but better to match
all the other breakpad tools which only accept short options.

BUG= chromium:598947 

Change-Id: I473081a29a8e3ef07a370848343f1a9e6681fd4e
Reviewed-on: https://chromium-review.googlesource.com/402908
Reviewed-by: Mark Mentovai <mark@chromium.org>

[modify] https://crrev.com/ed7dcced1945864c94dcfd358a64e23c812fd0dc/src/tools/linux/md2core/minidump-2-core.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/breakpad/breakpad/+/54b524be13cfaf70b99788998fc2c9fca3084d01

commit 54b524be13cfaf70b99788998fc2c9fca3084d01
Author: Mike Frysinger <vapier@chromium.org>
Date: Wed Oct 26 05:14:11 2016

minidump-2-core: add an -o flag for controlling core output

Always writing to stdout makes it hard to debug, and hard to use in
some script environments.  Add an explicit -o flag to make it easier.

BUG= chromium:598947 

Change-Id: I79667d033c8bdc8412d3a44fe3557d65f704968f
Reviewed-on: https://chromium-review.googlesource.com/403988
Reviewed-by: Mark Mentovai <mark@chromium.org>

[modify] https://crrev.com/54b524be13cfaf70b99788998fc2c9fca3084d01/src/tools/linux/md2core/minidump-2-core.cc

i've finalized the last CL which adds much better control over the filenames that show up in the core file.  i'll close this out once it's been merged.

The following revision refers to this bug:
  https://chromium.googlesource.com/breakpad/breakpad/+/7df62a96eb58474bd1393e4486de0a44c15fb88c

commit 7df62a96eb58474bd1393e4486de0a44c15fb88c
Author: Mike Frysinger <vapier@chromium.org>
Date: Tue Oct 25 07:01:00 2016

minidump-2-core: add more control over filenames

The code has been rewriting the location of the shared lib lookup
completely which breaks normal sysroot usage with gdb.  Split out
the behavior into dedicated flags so people can opt into it.  You
can see examples of -i/-f in the usage() text.

We also change the -S behavior so that it's no longer enabled by
default -- if people want /var/lib/breakpad/, they can pass the -S
flag explicitly.

BUG= chromium:598947 

Change-Id: Ic81726c27b4ad6c271c70696f2ac62798f07ccfb
Reviewed-on: https://chromium-review.googlesource.com/402909
Reviewed-by: Mark Mentovai <mark@chromium.org>

[modify] https://crrev.com/7df62a96eb58474bd1393e4486de0a44c15fb88c/src/tools/linux/md2core/minidump-2-core.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/breakpad/breakpad/+/117aa2510771ebfdf254dd5f4f88562a70afb59e

commit 117aa2510771ebfdf254dd5f4f88562a70afb59e
Author: Mike Frysinger <vapier@chromium.org>
Date: Fri Mar 24 16:21:48 2017

minidump_dump: add proper cli processing

In preparation for adding more flexibility to this tool, add a
proper parser for the command line flags.  This uses the style
as seen in other breakpad tools.

BUG= chromium:598947 

Change-Id: I95495e6ca7093be34d0d426f98a6c22880ff24a3
Reviewed-on: https://chromium-review.googlesource.com/457019
Reviewed-by: Mark Mentovai <mark@chromium.org>

[modify] https://crrev.com/117aa2510771ebfdf254dd5f4f88562a70afb59e/src/processor/minidump_dump.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/breakpad/breakpad/+/e1b3620ec763cf3cd116570556cc3d7ca1771ad2

commit e1b3620ec763cf3cd116570556cc3d7ca1771ad2
Author: Mike Frysinger <vapier@chromium.org>
Date: Fri Mar 24 16:22:21 2017

minidump_dump: dump stack memory like hexdump

The current stack output is one line byte string which is not easy for
humans to parse.  Extend the print mode to support a hexdump-like view
and switch to that by default.  Now we get something like:
Stack
00000000  20 67 7b 53 94 7f 00 00  01 00 00 00 00 00 00 00  | g{S...........|
00000010  00 70 c4 44 9a 25 00 00  08 65 7a 53 94 7f 00 00  |.p.D.%...ezS...|

BUG= chromium:598947 

Change-Id: I868e1cf4faa435a14c5f1c35f94a5db4a49b6a6d
Reviewed-on: https://chromium-review.googlesource.com/404008
Reviewed-by: Mark Mentovai <mark@chromium.org>

[modify] https://crrev.com/e1b3620ec763cf3cd116570556cc3d7ca1771ad2/src/processor/minidump_dump.cc
[modify] https://crrev.com/e1b3620ec763cf3cd116570556cc3d7ca1771ad2/src/google_breakpad/processor/minidump.h
[modify] https://crrev.com/e1b3620ec763cf3cd116570556cc3d7ca1771ad2/src/processor/minidump.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/breakpad/breakpad/+/7a8374fc4c199ecf8f355a536fe26fccfe0faaf0

commit 7a8374fc4c199ecf8f355a536fe26fccfe0faaf0
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat Mar 25 01:37:18 2017

minidump_dump: fix up init paths

Fix some build & test failures in the previous minidump_dump code.

BUG= chromium:598947 

Change-Id: Ia8fce453265167368de96747a8a92af930e78245
Reviewed-on: https://chromium-review.googlesource.com/458881
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/7a8374fc4c199ecf8f355a536fe26fccfe0faaf0/src/processor/minidump_dump.cc
[modify] https://crrev.com/7a8374fc4c199ecf8f355a536fe26fccfe0faaf0/src/processor/minidump.cc