Breakpad cannot handle version of Dwarf frame info generated by LLVM |
||||||||||||||||
Issue descriptionDaisy images built with the latest LLVM consistently fail the logging_UserCrash in bvt-cq.
,
May 25 2016
It appears that LLVM creates a slightly different stack than expected...the recursive function being searched for is there, but the test is looking for it in frame 15, and it appears to actually be in frames 14 & 16:
Thread 0 (crashed)
0 crasher_nobreakpad!recbomb [bomb.cc : 9 + 0x8]
r0 = 0x00000016 r1 = 0x00000000 r2 = 0x00000001 r3 = 0x00000000
r4 = 0xbe810b30 r5 = 0x00000000 r6 = 0x00000000 r7 = 0xbe810b18
r8 = 0x00000000 r9 = 0x00000000 r10 = 0xb6fb9fcc r12 = 0x00000000
fp = 0xbe810a20 sp = 0xbe810a18 lr = 0xb6fb8774 pc = 0xb6fb8758
Found by: given as instruction pointer in context
1 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a20 pc = 0xb6fb9fcc
Found by: stack scanning
2 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a28 pc = 0xb6fb8774
Found by: stack scanning
3 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a30 pc = 0xb6fb9fcc
Found by: stack scanning
4 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a38 pc = 0xb6fb8774
Found by: stack scanning
5 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a40 pc = 0xb6fb9fcc
Found by: stack scanning
6 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a48 pc = 0xb6fb8774
Found by: stack scanning
7 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a50 pc = 0xb6fb9fcc
Found by: stack scanning
8 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a58 pc = 0xb6fb8774
Found by: stack scanning
9 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a60 pc = 0xb6fb9fcc
Found by: stack scanning
10 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a68 pc = 0xb6fb8774
Found by: stack scanning
11 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a70 pc = 0xb6fb9fcc
Found by: stack scanning
12 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a78 pc = 0xb6fb8774
Found by: stack scanning
13 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a80 pc = 0xb6fb9fcc
Found by: stack scanning
14 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a88 pc = 0xb6fb8774
Found by: stack scanning
15 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a90 pc = 0xb6fb9fcc
Found by: stack scanning
16 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a98 pc = 0xb6fb8774
Found by: stack scanning
17 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810aa0 pc = 0xb6fb9fcc
Found by: stack scanning
18 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810aa8 pc = 0xb6fb8774
Found by: stack scanning
19 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810ab0 pc = 0xb6fb9fcc
Found by: stack scanning
20 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810ab8 pc = 0xb6fb8774
Found by: stack scanning
21 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810ac0 pc = 0xb6fb9fcc
Found by: stack scanning
22 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810ac8 pc = 0xb6fb8774
...
and
05/18 14:37:07.618 WARNI| test:0606| Autotest caught exception when running test:
Traceback (most recent call last):
File "/usr/local/autotest/common_lib/test.py", line 600, in _exec
_call_test_function(self.execute, *p_args, **p_dargs)
File "/usr/local/autotest/common_lib/test.py", line 804, in _call_test_function
return func(*args, **dargs)
File "/usr/local/autotest/common_lib/test.py", line 461, in execute
dargs)
File "/usr/local/autotest/common_lib/test.py", line 347, in _call_run_once_with_retry
postprocess_profiled_run, args, dargs)
File "/usr/local/autotest/common_lib/test.py", line 376, in _call_run_once
self.run_once(*args, **dargs)
File "/usr/local/autotest/tests/logging_UserCrash/logging_UserCrash.py", line 710, in run_once
initialize_crash_reporter=True)
File "/usr/local/autotest/cros/crash_test.py", line 696, in run_crash_tests
getattr(self, '_test_' + test_name)()
File "/usr/local/autotest/tests/logging_UserCrash/logging_UserCrash.py", line 445, in _test_chronos_crasher
self._check_crashing_process('chronos')
File "/usr/local/autotest/tests/logging_UserCrash/logging_UserCrash.py", line 425, in _check_crashing_process
from_crash_reporter=True)
File "/usr/local/autotest/tests/logging_UserCrash/logging_UserCrash.py", line 285, in _check_minidump_stackwalk
self._verify_stack(stack, basename, from_crash_reporter)
File "/usr/local/autotest/tests/logging_UserCrash/logging_UserCrash.py", line 161, in _verify_stack
raise error.TestFail('Did not show recursion line on stack')
TestFail: Did not show recursion line on stack
And from line 161 in logging_UserCrash.py:
# Should identify recursion line which is on the stack
# for 15 levels
if not self._is_frame_in_stack(15, basename, 'recbomb',
'bomb.cc', 12, stack):
raise error.TestFail('Did not show recursion line on stack')
,
May 25 2016
On further examination of the test source code, I think the real issue is that it is not expecting all the "crasher_nobreakpad!_fini" frames that are in the stack.
,
May 25 2016
,
May 25 2016
Full stack dump:
Thread 0 (crashed)
0 crasher_nobreakpad!recbomb [bomb.cc : 9 + 0x8]
r0 = 0x00000016 r1 = 0x00000000 r2 = 0x00000001 r3 = 0x00000000
r4 = 0xbe810b30 r5 = 0x00000000 r6 = 0x00000000 r7 = 0xbe810b18
r8 = 0x00000000 r9 = 0x00000000 r10 = 0xb6fb9fcc r12 = 0x00000000
fp = 0xbe810a20 sp = 0xbe810a18 lr = 0xb6fb8774 pc = 0xb6fb8758
Found by: given as instruction pointer in context
1 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a20 pc = 0xb6fb9fcc
Found by: stack scanning
2 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a28 pc = 0xb6fb8774
Found by: stack scanning
3 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a30 pc = 0xb6fb9fcc
Found by: stack scanning
4 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a38 pc = 0xb6fb8774
Found by: stack scanning
5 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a40 pc = 0xb6fb9fcc
Found by: stack scanning
6 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a48 pc = 0xb6fb8774
Found by: stack scanning
7 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a50 pc = 0xb6fb9fcc
Found by: stack scanning
8 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a58 pc = 0xb6fb8774
Found by: stack scanning
9 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a60 pc = 0xb6fb9fcc
Found by: stack scanning
10 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a68 pc = 0xb6fb8774
Found by: stack scanning
11 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a70 pc = 0xb6fb9fcc
Found by: stack scanning
12 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a78 pc = 0xb6fb8774
Found by: stack scanning
13 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a80 pc = 0xb6fb9fcc
Found by: stack scanning
14 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a88 pc = 0xb6fb8774
Found by: stack scanning
15 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810a90 pc = 0xb6fb9fcc
Found by: stack scanning
16 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810a98 pc = 0xb6fb8774
Found by: stack scanning
17 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810aa0 pc = 0xb6fb9fcc
Found by: stack scanning
18 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810aa8 pc = 0xb6fb8774
Found by: stack scanning
19 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810ab0 pc = 0xb6fb9fcc
Found by: stack scanning
20 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810ab8 pc = 0xb6fb8774
Found by: stack scanning
21 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810ac0 pc = 0xb6fb9fcc
Found by: stack scanning
22 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810ac8 pc = 0xb6fb8774
Found by: stack scanning
23 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810ad0 pc = 0xb6fb9fcc
Found by: stack scanning
24 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810ad8 pc = 0xb6fb8774
Found by: stack scanning
25 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810ae0 pc = 0xb6fb9fcc
Found by: stack scanning
26 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810ae8 pc = 0xb6fb8774
Found by: stack scanning
27 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810af0 pc = 0xb6fb9fcc
Found by: stack scanning
28 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x2]
sp = 0xbe810af8 pc = 0xb6fb8774
Found by: stack scanning
29 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810b00 pc = 0xb6fb9fcc
Found by: stack scanning
30 crasher_nobreakpad!main [crasher.cc : 20 + 0x6]
sp = 0xbe810b08 pc = 0xb6fb866c
Found by: stack scanning
31 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810b10 pc = 0xb6fb9fcc
Found by: stack scanning
32 libc-2.19.so + 0x16307
sp = 0xbe810b18 pc = 0xb6d6c309
Found by: stack scanning
33 crasher_nobreakpad!DefeatTailOptimizationForBomb [crasher.cc : 15 + 0x6]
sp = 0xbe810b1c pc = 0xb6fb8658
Found by: stack scanning
34 crasher_nobreakpad!DefeatTailOptimizationForCrasher [bomb.cc : 16 + 0x7]
sp = 0xbe810b38 pc = 0xb6fb878d
Found by: stack scanning
35 crasher_nobreakpad!_fini + 0x7f2
sp = 0xbe810b4c pc = 0xb6fb9fcc
Found by: stack scanning
,
May 25 2016
For comparison, I built the test by hand with the clang++ compiler in my chroot (version 3.9), then copied it onto a daisy Chromebook and ran it under gdb (it automatically crashes). Then I did a backtrace at the crash: Program received signal SIGSEGV, Segmentation fault. 0x2a000800 in recbomb(int) () (gdb) bt #0 0x2a000800 in recbomb(int) () #1 0x2a000818 in recbomb(int) () #2 0x2a000818 in recbomb(int) () #3 0x2a000818 in recbomb(int) () #4 0x2a000818 in recbomb(int) () #5 0x2a000818 in recbomb(int) () #6 0x2a000818 in recbomb(int) () #7 0x2a000818 in recbomb(int) () #8 0x2a000818 in recbomb(int) () #9 0x2a000818 in recbomb(int) () #10 0x2a000818 in recbomb(int) () #11 0x2a000818 in recbomb(int) () #12 0x2a000818 in recbomb(int) () #13 0x2a000818 in recbomb(int) () #14 0x2a000818 in recbomb(int) () #15 0x2a000818 in recbomb(int) () #16 0x2a0006ac in main () (gdb) :-( That appears to be what the test program is expecting. So LLVM seems to be doing the right thing when I build/run this by hand, but the test is getting confused by something. I'm not sure if this needs to be a P1 or not at this point.
,
May 26 2016
this is also failing for Peppy (at least for the testing done today) https://uberchromegw.corp.google.com/i/chromeos/builders/llvm-toolchain-group/builds/75
,
May 27 2016
Update: There appear to be two separate issues. The first issues is that the breakpad symbol file for the llvm-compiled version of the test is missing "STACK CFI INIT" entries for most of the functions (it only has entries for __libc_csu_init and __libc_csu_fin). The breakpad symbol file for the gcc-compiled version of the test has symbols for all of the functions: main, DefeatTailOptimizationForBomb, PrepareBelow, recbomb, DefeatTailOptimizationForCrasher, __libc_csu_init, and __libc_csu_fini. This could either be a bug in the dump_syms utility that generates the breakpad sym file from the crashed binary, or it could be a bug in LLVM (not generating the necessary information). I'm still tracking this down.
The second issue is that the stack scanning portion of the minidump_stackwalker utility does not work properly on the LLVM-compiled version of the test case. To verify that this was llvm-specific, I edited the breakpad sym file for the gcc-generated case, and removed the "CFI STACK INIT" entries for all of the functions except __libc_csu_init and __libc_csu_fini (the two that the llvm version has). Then I ran minidump stackwalker using the new sym file, to force it to do stack scanning, and it still wrked properly (no extra frames):
Thread 0 (crashed)
0 crasher_nobreakpad!recbomb [bomb.cc : 9 + 0x0]
r0 = 0x00000001 r1 = 0x00000000 r2 = 0x00000000 r3 = 0x00000016
r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40 r7 = 0xbef33378
r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc r12 = 0x00000000
fp = 0x00000000 sp = 0xbef33378 lr = 0xb6fbe805 pc = 0xb6fbe81c
Found by: given as instruction pointer in context
1 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef33388 pc = 0xb6fbe805
Found by: stack scanning
2 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef33398 pc = 0xb6fbe805
Found by: stack scanning
3 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef333a8 pc = 0xb6fbe805
Found by: stack scanning
4 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef333b8 pc = 0xb6fbe805
Found by: stack scanning
5 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef333c8 pc = 0xb6fbe805
Found by: stack scanning
6 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef333d8 pc = 0xb6fbe805
Found by: stack scanning
7 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef333e8 pc = 0xb6fbe805
Found by: stack scanning
8 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef333f8 pc = 0xb6fbe805
Found by: stack scanning
9 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef33408 pc = 0xb6fbe805
Found by: stack scanning
10 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef33418 pc = 0xb6fbe805
Found by: stack scanning
11 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef33428 pc = 0xb6fbe805
Found by: stack scanning
12 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef33438 pc = 0xb6fbe805
Found by: stack scanning
13 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef33448 pc = 0xb6fbe805
Found by: stack scanning
14 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef33458 pc = 0xb6fbe805
Found by: stack scanning
15 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
sp = 0xbef33468 pc = 0xb6fbe805
Found by: stack scanning
16 crasher_nobreakpad!main [crasher.cc : 20 + 0x5]
sp = 0xbef33478 pc = 0xb6fbe5e3
Found by: stack scanning
17 libc-2.19.so + 0x16347
sp = 0xbef33498 pc = 0xb6d71349
Found by: stack scanning
18 crasher_nobreakpad!_init + 0x97
sp = 0xbef3349c pc = 0xb6fbe57d
Found by: stack scanning
19 crasher_nobreakpad!DefeatTailOptimizationForCrasher [bomb.cc : 17 + 0x7]
sp = 0xbef334b8 pc = 0xb6fbe839
Found by: stack scanning
20 crasher_nobreakpad!_fini + 0x736
sp = 0xbef334cc pc = 0xb6fbffcc
Found by: stack scanning
Just for comparison, with the full gcc version of the breakpad sym file, minidump_stackwalk generates:
Thread 0 (crashed)
0 crasher_nobreakpad!recbomb [bomb.cc : 9 + 0x0]
r0 = 0x00000001 r1 = 0x00000000 r2 = 0x00000000 r3 = 0x00000016
r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40 r7 = 0xbef33378
r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc r12 = 0x00000000
fp = 0x00000000 sp = 0xbef33378 lr = 0xb6fbe805 pc = 0xb6fbe81c
Found by: given as instruction pointer in context
1 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef33388 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef33388 pc = 0xb6fbe805
Found by: call frame info
2 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef33398 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef33398 pc = 0xb6fbe805
Found by: call frame info
3 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef333a8 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef333a8 pc = 0xb6fbe805
Found by: call frame info
4 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef333b8 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef333b8 pc = 0xb6fbe805
Found by: call frame info
5 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef333c8 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef333c8 pc = 0xb6fbe805
Found by: call frame info
6 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef333d8 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef333d8 pc = 0xb6fbe805
Found by: call frame info
7 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef333e8 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef333e8 pc = 0xb6fbe805
Found by: call frame info
8 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef333f8 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef333f8 pc = 0xb6fbe805
Found by: call frame info
9 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef33408 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef33408 pc = 0xb6fbe805
Found by: call frame info
10 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef33418 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef33418 pc = 0xb6fbe805
Found by: call frame info
11 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef33428 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef33428 pc = 0xb6fbe805
Found by: call frame info
12 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef33438 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef33438 pc = 0xb6fbe805
Found by: call frame info
13 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef33448 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef33448 pc = 0xb6fbe805
Found by: call frame info
14 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef33458 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef33458 pc = 0xb6fbe805
Found by: call frame info
15 crasher_nobreakpad!recbomb [bomb.cc : 12 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef33468 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef33468 pc = 0xb6fbe805
Found by: call frame info
16 crasher_nobreakpad!main [crasher.cc : 20 + 0x5]
r3 = 0x00000000 r4 = 0xb6e4dc80 r5 = 0x00000001 r6 = 0xb6e4db40
r7 = 0xbef33480 r8 = 0xbef335f4 r9 = 0x00000000 r10 = 0xb6fbffcc
fp = 0x00000000 sp = 0xbef33478 pc = 0xb6fbe5e3
Found by: call frame info
17 libc-2.19.so + 0x16347
r4 = 0xbef334b0 r5 = 0x00000000 r6 = 0x00000000 r7 = 0xbef33498
r8 = 0x00000000 r9 = 0x00000000 r10 = 0xb6fbffcc fp = 0x00000000
sp = 0xbef33498 pc = 0xb6d71349
Found by: call frame info
18 crasher_nobreakpad!_init + 0x97
sp = 0xbef3349c pc = 0xb6fbe57d
Found by: stack scanning
19 crasher_nobreakpad!DefeatTailOptimizationForCrasher [bomb.cc : 17 + 0x7]
sp = 0xbef334b8 pc = 0xb6fbe839
Found by: stack scanning
20 crasher_nobreakpad!_fini + 0x736
sp = 0xbef334cc pc = 0xb6fbffcc
Found by: stack scanning
I am attaching the two sym files to this issue.
,
May 28 2016
What are the compiler command lines? Have you run objdump or readelf?
,
May 31 2016
The compiler command line (as represented by a Python list) is: /usr/bin/clang++ -Qunused-arguments -Wno-deprecated-declarations -fstack-protector-strong -pie -fno-omit-frame-pointer -fPIE -D_FORTIFY_SOURCE=2 -O2 -O2 -O2 -pipe -march=armv7-a -mtune=cortex-a15 -mfpu=neon -mfloat-abi=hard -g -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -c crasher.cc -o crasher.o -B/usr/libexec/gcc/armv7a-cros-linux-gnueabi -target armv7a-cros-linux-gnueabi -mfloat-abi=hard Yes I have run readelf. I will attach the results of --debug-dump=frames and --debug-dump=info (output caught in two separate files).
,
May 31 2016
I have found/identified the main source of the problem: LLVM is generating CIE/FDE frame info using Version 4. The dwarf2reader.cc file does not handle version 4 -- it explicitly says it only handles versions 1 & 3 (excerpt from file dwarf2reader.cc, lines 2253 - 2262):
// If we don't recognize the version, we can't parse any more fields of the
// CIE. For DWARF CFI, we handle versions 1 through 3 (there was never a
// version 2 of CFI data). For .eh_frame, we handle versions 1 and 3 as well;
// the difference between those versions seems to be the same as for
// .debug_frame.
if (cie->version < 1 || cie->version > 3) {
reporter_->UnrecognizedVersion(cie->offset, cie->version);
return false;
}
This needs to be fixed ASAP, as the toolchain team is planning on migrating to LLVM as the main ChromeOS compiler very soon.
I am going to rename this bug (old name was "Daisy image built with LLVM fails logging_UserCrash (in bvt-cq)") and assign it to ivanpe to distribute as he sees fit.
,
May 31 2016
can't we build with -gdwarf-3 in the meantime ?
,
Jun 1 2016
Crashpad is the upcoming replacement for Breakpad, but it won’t have a processor component until at least Q4. It will natively understand DWARF, including v4. For the time being debugging information should be generated using -gdwarf-3.
,
Jun 2 2016
Re-opening this and assigning it to myself -- we need to update the LLVM builds to pass -gdwarf-3 to debug builds until crashpad is released.
,
Jun 2 2016
shouldn't that be in the compiler wrapper ? bad symbolization with tests will also happen with all the release programs built with clang ...
,
Jun 2 2016
Yes, the fix needs to go into the compiler wrapper and that is where I am planning on fixing it.
,
Jun 7 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/5e93af8f3b58937aac4541aabf241a127833db2f commit 5e93af8f3b58937aac4541aabf241a127833db2f Author: Caroline Tice <cmtice@google.com> Date: Fri Jun 03 13:55:53 2016 [GCC] Update compiler wrappers to pass -gdwarf-3 to clang. Breakpad cannot handle dwarf V4 .debug_frames info, which is what clang generates by default. This updates the compiler wrapper script to pass '-gdwarf-3' to clang (when '-g' is specified), so clang will generate dwarf V3, which breakpad can understand. BUG= chromium:614788 TEST=Tested in my chroot. Change-Id: I5792d681a843761b7e39b4e864cae1b01d8d24af Reviewed-on: https://chromium-review.googlesource.com/349422 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/5e93af8f3b58937aac4541aabf241a127833db2f/sys-devel/gcc/files/sysroot_wrapper.hardened [rename] https://crrev.com/5e93af8f3b58937aac4541aabf241a127833db2f/sys-devel/gcc/gcc-4.9.2-r127.ebuild
,
Jun 7 2016
,
Jul 1 2016
,
Aug 3 2016
has a sep/new bug been filed for breakpad ? i don't see one in the tracker: https://bugs.chromium.org/p/google-breakpad/issues can you file one please ?
,
Aug 3 2016
I spoke to the breakpad people and they are not planning on fixing this, because breakpad is deprecated, and crashpad, the upcoming replacement, supposedly handles this just fine. So there's no point in filing a bug against breakpad.
,
Aug 5 2016
iiuc, CrOS/Linux isn't on the "soon" roadmap for crashpad
,
Aug 9 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/a81d5ade0b043208e06ad71a38bcf9c348a1a52f commit a81d5ade0b043208e06ad71a38bcf9c348a1a52f Author: Caroline Tice <cmtice@google.com> Date: Tue Aug 02 21:37:32 2016 [LLVM] Force debug_frames to be Dwarf version 1. This is a temporary workaround. Currently LLVM generates Dwarf version 4 for all of its debug information, including its Debug Frames (CIE) information. Breakpad cannot handle Dwarf Version 4 (see https://bugs.chromium.org/p/chromium/issues/detail?id=614788). We tried reducing all of the Dwarf information to version 3, but that causes problems with Fission (the splitting of debug information into separate files), which requires the Dwarf DIEs to be emitted at version 4. This patch works around that issue by forcing the Debug Frames version of Dwarf to be version 1, while leaving the rest of the Dwarf data at version 4. When the replacement for Breakpad comes out we will no longer need this patch, as that is supposed to be able to handle version 4 frames information. BUG= chromium:614788 TEST=Built clang in chroot; compiled test program with it; debug info & frames had correct versions. Change-Id: Iffe2c3acd030641b3c75890c9b287df94cf59693 Reviewed-on: https://chromium-review.googlesource.com/365441 Commit-Ready: Caroline Tice <cmtice@chromium.org> Tested-by: Caroline Tice <cmtice@chromium.org> Reviewed-by: Caroline Tice <cmtice@chromium.org> [rename] https://crrev.com/a81d5ade0b043208e06ad71a38bcf9c348a1a52f/sys-devel/gcc/gcc-4.9.2-r134.ebuild [modify] https://crrev.com/a81d5ade0b043208e06ad71a38bcf9c348a1a52f/sys-devel/gcc/files/sysroot_wrapper.hardened [add] https://crrev.com/a81d5ade0b043208e06ad71a38bcf9c348a1a52f/sys-devel/llvm/files/llvm-3.9-dwarf-version.patch [rename] https://crrev.com/a81d5ade0b043208e06ad71a38bcf9c348a1a52f/sys-devel/llvm/llvm-3.9_pre265926-r7.ebuild
,
Aug 19 2016
,
Aug 29 2016
,
Oct 7 2016
,
Nov 2 2016
Caroline, I think you’ll be pleased to learn that https://chromium-review.googlesource.com/c/406012/ should enable us to deal with DWARF 4 CIE.
,
Nov 2 2016
i've pushed that into CrOS master now. thanks!
,
Nov 2 2016
vapier@, just making sure I understand your comment: Are you saying you've pushed a revert of https://chromium-review.googlesource.com/c/406012 ? (I was going to create one, but if you've already pushed it I don't want to duplicate the work...)
,
Nov 2 2016
i've only updated our copy of breakpad in CrOS. i haven't made any toolchain changes since it'd require doing a good amount of validation first, and it seems like it'd be better for the toolchain team to handle that part :).
,
Nov 19 2016
,
Jan 21 2017
,
Mar 4 2017
,
Mar 17 2017
|
||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||
Comment 1 by cmt...@chromium.org
, May 25 2016