Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Issue 255163 chrome crashed in autotest and produced .dmp but no stack crawl
Starred by 1 user Project Member Reported by diand...@chromium.org, Jun 27 2013 Back to list
Status: Duplicate
Owner: sbasi@chromium.org
Closed: Jul 2015
Cc: vapier@chromium.org
OS: ----
Pri: 2
Type: Bug


Sign in to add a comment
You can see a failed test in stout canary here:

https://chromegw.corp.google.com/i/chromeos/builders/stout%20canary/builds/1498


If you open up:

http://cautotest/tko/retrieve_logs.cgi?job=/results/3534426-chromeos-test/

...you can dig down to:
-> desktopui_FlashSanityCheck
 -> sysinfo
  -> var
   -> spool
    -> crash

There's a .dmp file but the .dmp.txt file is 0 bytes.  Doh!
 
Labels: -OS-Chrome Hardware-Lab
Owner: sbasi@chromium.org
Is this a bug in the symbolizing server? Lab folks can you take a look?
BTW: is there some easy way to symbolize this dump?  I don't really want to go through all of the steps in <http://www.chromium.org/developers/decoding-crash-dumps>
Comment 3 by vapier@chromium.org, Jun 27 2013
you could manually upload the .dmp to the crash server.  do you have the .meta file too ?  maybe put it on a device in /var/spool/crash/ and then run `crash_sender` ?
FWIW: beeps@ helped me get this crash symbolized nicely.

curl -F minidump=@chrome.20130627.122423.28234.dmp http://chromeos-crash1.cros:8080/symbolicate_dump?archive_url=gs://chromeos-image-archive/stout-release/R30-4330.0.0/

...so that means that there was nothing wrong with the minidump--just with the process.

---

That showed me a known crash bug #234382:

Operating system: Linux
                  0.0.0 Linux 3.4.0 #1 SMP Thu Jun 27 10:56:20 PDT 2013 x86_64
CPU: amd64
     family 6 model 58 stepping 9
     2 CPUs

Crash reason:  SIGABRT
Crash address: 0x0

Thread 0 (crashed)
 0  libc-2.15.so!raise [raise.c : 64 + 0x10]
    rbx = 0x00007fff9e0b4310   r12 = 0x00007fff9e0b3ea0
    r13 = 0x0000000000000000   r14 = 0x0000000000000001
    r15 = 0x0000000000000000   rip = 0x00007fc5180b9d95
    rsp = 0x00007fff9e0b3d38   rbp = 0x00007fc518c48ee0
    Found by: given as instruction pointer in context
 1  libc-2.15.so!abort [abort.c : 91 + 0x9]
    rbx = 0x00007fff9e0b4310   r12 = 0x00007fff9e0b3ea0
    r13 = 0x0000000000000000   r14 = 0x0000000000000001
    r15 = 0x0000000000000000   rip = 0x00007fc5180bb218
    rsp = 0x00007fff9e0b3d40   rbp = 0x00007fc518c48ee0
    Found by: call frame info
 2  chrome!base::debug::BreakDebugger() [debugger_posix.cc : 247 + 0x4]
    rbx = 0x00007fff9e0b4310   r12 = 0x00007fff9e0b3ea0
    r13 = 0x0000000000000000   r14 = 0x0000000000000001
    r15 = 0x0000000000000000   rip = 0x00007fc51a9d39c9
    rsp = 0x00007fff9e0b3e70   rbp = 0x00007fc518c48ee0
    Found by: call frame info
 3  chrome!logging::LogMessage::~LogMessage() [logging.cc : 649 + 0x4]
    rbx = 0x00007fff9e0b4310   r12 = 0x00007fff9e0b3ea0
    r13 = 0x0000000000000000   r14 = 0x0000000000000001
    r15 = 0x0000000000000000   rip = 0x00007fc51d989f15
    rsp = 0x00007fff9e0b3e80   rbp = 0x00007fc518c48ee0
    Found by: call frame info
 4  chrome!chromeos::CrosDBusServiceImpl::OnOwnership(std::string const&, bool) [cros_dbus_service.cc : 82 + 0x7]
    rbx = 0x00007fff9e0b4318   r12 = 0x00007fc517e26418
    r13 = 0x00007fff9e0b4670   r14 = 0x0000000000000001
    r15 = 0x0000000000000000   rip = 0x00007fc51dc787f2
    rsp = 0x00007fff9e0b4310   rbp = 0x00007fc517e26410
    Found by: call frame info
 5  chrome!base::internal::Invoker<2, base::internal::BindState<base::Callback<void (std::string const&, bool)>, void (std::string const&, bool), void (std::string, bool)>, void (std::string const&, bool)>::Run(base::internal::BindStateBase*) [callback.h : 479 + 0x1]
    rbx = 0x00007fc517e263f0   r12 = 0x00007fc517e26418
    r13 = 0x00007fff9e0b4670   r14 = 0x0000000000000001
    r15 = 0x0000000000000000   rip = 0x00007fc51af3118b
    rsp = 0x00007fff9e0b44c0   rbp = 0x00007fc517e26410
    Found by: call frame info
 6  chrome!base::MessageLoop::RunTask(base::PendingTask const&) [callback.h : 396 + 0x6]
    rbx = 0x00007fff9e0b4660   r12 = 0x00007fc520916465
    r13 = 0x00007fff9e0b4670   r14 = 0x0000000000000001
    r15 = 0x0000000000000000   rip = 0x00007fc51a9ecf38
    rsp = 0x00007fff9e0b4500   rbp = 0x00007fc517dc9800
    Found by: call frame info
 7  chrome!base::MessageLoop::DeferOrRunPendingTask(base::PendingTask const&) [message_loop.cc : 496 + 0x7]
    rbx = 0x00007fc517e28ca0   r12 = 0x00007fff9e0b4678
    r13 = 0x00007fff9e0b4670   r14 = 0x0000000000000001
    r15 = 0x0000000000000000   rip = 0x00007fc51a9ee368
    rsp = 0x00007fff9e0b4630   rbp = 0x00007fc517dc9800
    Found by: call frame info
 8  chrome!base::MessageLoop::DoWork() [message_loop.cc : 688 + 0x4]
    rbx = 0x00007fc517e28ca0   r12 = 0x00007fff9e0b4678
    r13 = 0x00007fff9e0b4670   r14 = 0x0000000000000001
    r15 = 0x0000000000000000   rip = 0x00007fc51a9f0b6a
    rsp = 0x00007fff9e0b4660   rbp = 0x00007fc517dc9800
    Found by: call frame info
 9  chrome!base::MessagePumpGlib::RunWithDispatcher(base::MessagePump::Delegate*, base::MessagePumpDispatcher*) [message_pump_glib.cc : 202 + 0x8]
    rbx = 0x00007fc517d8ce00   r12 = 0x00007fc517d8ce20
    r13 = 0x0000000000000000   r14 = 0x0000000000000001
    r15 = 0x0000000000000000   rip = 0x00007fc51dc72b58
    rsp = 0x00007fff9e0b46f0   rbp = 0x0000000000000001
    Found by: call frame info
10  chrome!base::RunLoop::Run() [run_loop.cc : 45 + 0x4]
    rbx = 0x00007fff9e0b47e0   r12 = 0x0000000000000000
    r13 = 0x00007fc517d8cd38   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007fc51d990cdd
    rsp = 0x00007fff9e0b4750   rbp = 0x0000000000000000
    Found by: call frame info
11  chrome!ChromeBrowserMainParts::MainMessageLoopRun(int*) [chrome_browser_main.cc : 1620 + 0xc]
    rbx = 0x000028633e514000   r12 = 0x0000000000000000
    r13 = 0x00007fc517d8cd38   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007fc51dc72942
    rsp = 0x00007fff9e0b4760   rbp = 0x0000000000000000
    Found by: call frame info
12  chrome!content::BrowserMainLoop::RunMainMessageLoopParts() [browser_main_loop.cc : 628 + 0x9]
    rbx = 0x00007fc517d8cd20   r12 = 0x00007fc517d7ba40
    r13 = 0x00007fc517d85fa0   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007fc51dc72781
    rsp = 0x00007fff9e0b4850   rbp = 0x00000000ffffffff
    Found by: call frame info
13  chrome!content::BrowserMainRunnerImpl::Run() [browser_main_runner.cc : 118 + 0x8]
    rbx = 0x00007fc517d7ba40   r12 = 0x00007fc517d7ba40
    r13 = 0x00007fc517d85fa0   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007fc51dc7273d
    rsp = 0x00007fff9e0b4860   rbp = 0x00000000ffffffff
    Found by: call frame info
14  chrome!content::BrowserMain(content::MainFunctionParams const&) [browser_main.cc : 22 + 0x8]
    rbx = 0x00007fc517d7ba40   r12 = 0x00007fc517d7ba40
    r13 = 0x00007fc517d85fa0   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007fc51d97c2a9
    rsp = 0x00007fff9e0b4870   rbp = 0x00000000ffffffff
    Found by: call frame info
15  chrome!content::ContentMainRunnerImpl::Run() [content_main_runner.cc : 754 + 0x4]
    rbx = 0x00007fc518c48ee0   r12 = 0x00007fc517d7ba40
    r13 = 0x00007fc517d85fa0   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007fc51d97bfe1
    rsp = 0x00007fff9e0b4890   rbp = 0x00007fc517d85f50
    Found by: call frame info
16  chrome!content::ContentMain(int, char const**, content::ContentMainDelegate*) [content_main.cc : 35 + 0x8]
    rbx = 0x00007fc517d85fa0   r12 = 0x00007fff9e0b4a78
    r13 = 0x00007fff9e0b4930   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007fc51d9716c1
    rsp = 0x00007fff9e0b4900   rbp = 0x00000000ffffffff
    Found by: call frame info
17  chrome!ChromeMain [chrome_main.cc : 32 + 0xc]
    rbx = 0x0000000000000028   r12 = 0x00007fc51a49c8b0
    r13 = 0x00007fff9e0b4a70   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007fc51d971488
    rsp = 0x00007fff9e0b4930   rbp = 0x00007fff9e0b4a78
    Found by: call frame info
18  libc-2.15.so!__libc_start_main [libc-start.c : 234 + 0x19]
    rbx = 0x0000000000000000   r12 = 0x00007fc51a49c8b0
    r13 = 0x00007fff9e0b4a70   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007fc5180a6465
    rsp = 0x00007fff9e0b49a0   rbp = 0x0000000000000000
    Found by: call frame info
19  chrome + 0x9618d8
    rbx = 0x0000000000000000   r12 = 0x00007fc51a49c8b0
    r13 = 0x00007fff9e0b4a70   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007fc51a49c8d9
    rsp = 0x00007fff9e0b4a60   rbp = 0x0000000000000000
    Found by: call frame info

...
...
...
OK.  I just went to the logs of the failed job, to try and figure out
why the symbolication failed.  The log data is below.  The failure
about minidump_stackwalk is a failure message I've seen an awful
lot when perusing logs.  I've never known why it happens.  Perhaps
we should figure it out?

Who owns this part of the system?

Error text is below:
====
06/27 12:25:25.013 INFO |site_crash:0026| symbol_dir: /usr/local/autotest/server/../../../lib/debug
06/27 12:25:25.013 DEBUG|base_utils:0081| Running 'minidump_stackwalk "/usr/local/autotest/results/3534426-chromeos-test/chromeos2-row4-rack5-host3/desktopui_FlashSanityCheck/sysinfo/var/spool/crash/chrome.20130627.122423.28234.dmp" "/usr/local/autotest/server/../../../lib/debug" > "/usr/local/autotest/results/3534426-chromeos-test/chromeos2-row4-rack5-host3/desktopui_FlashSanityCheck/sysinfo/var/spool/crash/chrome.20130627.122423.28234.dmp.txt"'
06/27 12:25:25.019 WARNI|site_crash:0084| Failed to generate stack trace locally for dump /usr/local/autotest/results/3534426-chromeos-test/chromeos2-row4-rack5-host3/desktopui_FlashSanityCheck/sysinfo/var/spool/crash/chrome.20130627.122423.28234.dmp (rc=127):
CmdError('minidump_stackwalk "/usr/local/autotest/results/3534426-chromeos-test/chromeos2-row4-rack5-host3/desktopui_FlashSanityCheck/sysinfo/var/spool/crash/chrome.20130627.122423.28234.dmp" "/usr/local/autotest/server/../../../lib/debug" > "/usr/local/autotest/results/3534426-chromeos-test/chromeos2-row4-rack5-host3/desktopui_FlashSanityCheck/sysinfo/var/spool/crash/chrome.20130627.122423.28234.dmp.txt"', * Command: 
    minidump_stackwalk "/usr/local/autotest/results/3534426-chromeos-test/chr
    omeos2-row4-rack5-host3/desktopui_FlashSanityCheck/sysinfo/var/spool/crash
    /chrome.20130627.122423.28234.dmp"
    "/usr/local/autotest/server/../../../lib/debug" >
    "/usr/local/autotest/results/3534426-chromeos-test/chromeos2-row4-rack5-ho
    st3/desktopui_FlashSanityCheck/sysinfo/var/spool/crash/chrome.20130627.122
    423.28234.dmp.txt"
Exit status: 127
Duration: 0.00244879722595

stderr:
/bin/bash: minidump_stackwalk: command not found, 'Command returned non-zero exit status')

Looking at the source, it seems that this message isn't entirely unexpected;
when this occurs, the code shifts to trying to symbolicate on the devserver.
The logs show *that* operation failed with a 500 status:

====
06/27 12:25:25.067 INFO |connection:0190| Starting new HTTP connection (1): 172.17.40.24
06/27 12:25:25.724 DEBUG|connection:0267| "POST /symbolicate_dump?archive_url=gs://chromeos-image-archive/stout-release/R30-4330.0.0 ('HTTP/1.1',)" 500 931
06/27 12:25:25.751 WARNI|site_crash:0094| Failed to generate stack trace on devserver for dump /usr/local/autotest/results/3534426-chromeos-test/chromeos2-row4-rack5-host3/desktopui_FlashSanityCheck/sysinfo/var/spool/crash/chrome.20130627.122423.28234.dmp:
DevServerException('\n\n\n    \n    500 Internal Server Error\n    \n    #powered_by {\n        margin-top: 20px;\n        border-top: 2px solid black;\n        font-style: italic;\n    }\n\n    #traceback {\n        color: red;\n    }\n    \n\n    \n        500 Internal Server Error\n        The server encountered an unexpected condition which prevented it from fulfilling the request.\n        Traceback (most recent call last):\n  File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 656, in respond\n    response.body = self.handler()\n  File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 188, in __call__\n    self.body = self.oldhandler(*args, **kwargs)\n  File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 34, in __call__\n    return self.callable(*self.args, **self.kwargs)\n  File "/home/chromeos-test/chromiumos/src/platform/dev/devserver.py", line 587, in symbolicate_dump\n    if self.stage(archive_url=archive_url, artifacts=\'symbols\') != \'Success\':\n  File "/home/chromeos-test/chromiumos/src/platform/dev/devserver.py", line 478, in stage\n    artifacts.split(\',\'))\n  File "/home/chromeos-test/chromiumos/src/platform/dev/downloader.py", line 106, in Download\n    self._DownloadArtifactsSerially(required_artifacts, no_wait=True)\n  File "/home/chromeos-test/chromiumos/src/platform/dev/downloader.py", line 117, in _DownloadArtifactsSerially\n    artifact.Process(no_wait)\n  File "/home/chromeos-test/chromiumos/src/platform/dev/build_artifact.py", line 172, in Process\n    self._WaitForArtifactToExist(timeout)\n  File "/home/chromeos-test/chromiumos/src/platform/dev/build_artifact.py", line 117, in _WaitForArtifactToExist\n    self.name)\nArtifactDownloadError: Could not find debug.tgz in Google Storage\n\n    \n    Powered by CherryPy 3.2.2\n    \n    \n\n',)

Reformatting the log message, you get this:

====
    500 Internal Server Error
    
    #powered_by {
        margin-top: 20px;
        border-top: 2px solid black;
        font-style: italic;
    }

    #traceback {
        color: red;
    }
    

    
        500 Internal Server Error
        The server encountered an unexpected condition which prevented it from fulfilling the request.
        Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/home/chromeos-test/chromiumos/src/platform/dev/devserver.py", line 587, in symbolicate_dump
    if self.stage(archive_url=archive_url, artifacts='symbols') != 'Success':
  File "/home/chromeos-test/chromiumos/src/platform/dev/devserver.py", line 478, in stage
    artifacts.split(','))
  File "/home/chromeos-test/chromiumos/src/platform/dev/downloader.py", line 106, in Download
    self._DownloadArtifactsSerially(required_artifacts, no_wait=True)
  File "/home/chromeos-test/chromiumos/src/platform/dev/downloader.py", line 117, in _DownloadArtifactsSerially
    artifact.Process(no_wait)
  File "/home/chromeos-test/chromiumos/src/platform/dev/build_artifact.py", line 172, in Process
    self._WaitForArtifactToExist(timeout)
  File "/home/chromeos-test/chromiumos/src/platform/dev/build_artifact.py", line 117, in _WaitForArtifactToExist
    self.name)
ArtifactDownloadError: Could not find debug.tgz in Google Storage

    
    Powered by CherryPy 3.2.2

Status: Assigned
minidump_stackwalk is installed inside the chroot and the target board at /usr/bin/.  what context is that code running in ?
Re comment #10:  I believe this code is running on the autotest master, or one
of the drones.  The environment is a more-or-less standard Goobuntu installation:
It's outside of the chroot, and indeed, there's no chroot even installed on the
system.

Arguably, the code in question should probably just be purged.  However, that's
orthogonal to this bug, since the minidump_stackwalk failure has been happening
for a very long time, and isn't contributing to this problem, other than as a red
herring.

Mergedinto: 515742
Status: Duplicate
This is old. Keeping it alive to not lose the comments.
Sign in to add a comment