After llvm r289925 (http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20161212/413488.html), 3 telemetry_perf_unittests started failing:

core.stacktrace_unittest.TabStackTraceTest.testCrashSymbols
core.stacktrace_unittest.TabStackTraceTest.testBadBreakpadFileIgnored
core.stacktrace_unittest.TabStackTraceTest.testCrashMinimalSymbols
 

https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/357983

I'm pasting one of the errors at the end, it's kind of long.

As far as I can tell, the stack trace comes from:

https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/desktop_browser_backend.py?q=GetStackTrace+file:tele&sq=package:chromium&dr=CSs&l=533 which seems to call minidump_stackwalk and generate_breakpad_symbols

It calls https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/desktop_browser_backend.py?q=GetStackTrace+file:tele&sq=package:chromium&dr=CSs&l=430 which either gets a crashpad or breakpad minidump which calls crashpad_database_util to get the minidump, and it sets BREAKPAD_DUMP_LOCATION to some temp folder (and I guess crashpad uses that env var?)

The test somehow navigates chrome to about:crash to cause an actual crash, but I couldn't figure out how (just `out/gn/chrome about:crash` doesn't do that).


I reverted the clang-side change for now, but I'm on the hook on getting upstream a repro case, and if this is just a bug in breakpad/crashpad/whatever does the symbolizing here, maybe we want to fix that instead. Upstream bug is at https://llvm.org/bugs/show_bug.cgi?id=31407

+dyen to say if I got the telemetry bits roughly right and to clue me in how to manually do what telemetry does.

+mark for breakpad / crashpad.


Traceback (most recent call last):
File "/b/swarm_slave/w/irXcfLuW/third_party/catapult/telemetry/telemetry/testing/browser_test_case.py", line 41, in WrappedMethod
method(self)
File "/b/swarm_slave/w/irXcfLuW/tools/perf/core/stacktrace_unittest.py", line 39, in testCrashMinimalSymbols
self.assertIn('OnNavigate', '\n'.join(c.exception.stack_trace))
AssertionError: 'OnNavigate' not found in 'Operating system: Linux\n 0.0.0 Linux 3.13.0-46-generic #79~precise1-Ubuntu SMP Tue Mar 10 20:25:33 UTC 2015 x86_64\nCPU: amd64\n family 6 model 62 stepping 4\n 8 CPUs\n\nGPU: UNKNOWN\n\nCrash reason: SIGSEGV\nCrash address: 0x0\nProcess uptime: not available\n\nThread 0 (crashed)\n 0 chrome + 0x5db563d\n rax = 0x54f7c57e66dc1b00 rdx = 0x000000000000031a\n rcx = 0x0000000000000002 rbx = 0x00007fffa5eea3b0\n rsi = 0x0000000000000000 rdi = 0x00007f6d3099c800\n rbp = 0x00007fffa5eeaaa8 rsp = 0x00007fffa5eea370\n r8 = 0x00007f6d26e59980 r9 = 0x00007f6d26e59980\n r10 = 0x0000000000000000 r11 = 0x0000000000000000\n r12 = 0x00007fffa5eeab78 r13 = 0x00007fffa5eea3a8\n r14 = 0x000035290d3cfc00 r15 = 0x00007fffa5eeaaa8\n rip = 0x00007f6d2cc4a63d\n Found by: given as instruction pointer in context\n 1 librt-2.15.so + 0x415d\n rbp = 0x00007fffa5eeaaa8 rsp = 0x00007fffa5eea390\n rip = 0x00007f6d2664b15d\n Found by: stack scanning\n\nThread 1\n 0 libc-2.15.so + 0xf3a03\n rax = 0x0000000000000001 rdx = 0x0000000000000020\n rcx = 0xffffffffffffffff rbx = 0x000035290d376c00\n rsi = 0x000035290d376c00 rdi = 0x0000000000000003\n rbp = 0x0000000000000000 rsp = 0x00007f6d1a787440\n r8 = 0x000035290d32bb80 r9 = 0x0000000000000002\n r10 = 0x00000000ffffffff r11 = 0x0000000000000293\n r12 = 0x000035290d324100 r13 = 0x000035290d3246f8\n r14 = 0x000035290d3a4be0 r15 = 0x000035290d324100\n rip = 0x00007f6d206bca03\n Found by: given as instruction pointer in context\n\nThread 2\n 0 libc-2.15.so + 0xf3a03\n rax = 0xfffffffffffffffc rdx = 0x0000000000000020\n rcx = 0xffffffffffffffff rbx = 0x000035290d376a80\n rsi = 0x000035290d376a80 rdi = 0x0000000000000054\n rbp = 0x0000000000000000 rsp = 0x00007f6d19f86440\n r8 = 0x000035290d32bc60 r9 = 0x0000000000000003\n r10 = 0x00000000ffffffff r11 = 0x0000000000000293\n r12 = 0x000035290d323a00 r13 = 0x000035290d323ff8\n r14 = 0x000035290d3a4bc0 r15 = 0x000035290d323a00\n rip = 0x00007f6d206bca03\n Found by: given as instruction pointer in context\n\nThread 3\n 0 libpthread-2.15.so + 0xc0fe\n rax = 0xfffffffffffffdfc rdx = 0x0000000000000001\n rcx = 0xffffffffffffffff rbx = 0x0000000000007bc7\n rsi = 0x0000000000000089 rdi = 0x00007f6d19785314\n rbp = 0x7fffffffffffffff rsp = 0x00007f6d19785080\n r8 = 0x00007f6d197852e8 r9 = 0x00000000ffffffff\n r10 = 0x00007f6d197850f8 r11 = 0x0000000000000206\n r12 = 0x0000000000000001 r13 = 0x00007f6d197850f8\n r14 = 0xffffffffffffff92 r15 = 0x0000000000000000\n rip = 0x00007f6d26a5f0fe\n Found by: given as instruction pointer in context\n 1 chrome!<name omitted> + 0x1c\n rbp = 0x7fffffffffffffff rsp = 0x00007f6d19785280\n rip = 0x00007f6d27a2fa4c\n Found by: stack scanning\n\nThread 4\n 0 libpthread-2.15.so + 0xbd84\n rax = 0xfffffffffffffe00 rdx = 0x0000000000000001\n rcx = 0xffffffffffffffff rbx = 0x00007f6d18f84170\n rsi = 0x0000000000000080 rdi = 0x00007f6d18f84174\n rbp = 0x7fffffffffffffff rsp = 0x00007f6d18f83f30\n r8 = 0x0000000000000000 r9 = 0x0000000000000000\n r10 = 0x0000000000000000 r11 = 0x0000000000000246\n r12 = 0x00007f6d18f84128 r13 = 0x00007f6d18f84200\n r14 = 0x000035290d463710 r15 = 0x7fffffffffffffff\n rip = 0x00007f6d26a5ed84\n Found by: given as instruction pointer in context\n 1 chrome!RemoveRange + 0x128\n rbp = 0x7fffffffffffffff rsp = 0x00007f6d18f83f80\n rip = 0x00007f6d27a31ff8\n Found by: stack scanning\n 2 librt-2.15.so + 0x415d\n rbp = 0x7fffffffffffffff rsp = 0x00007f6d18f83f90\n rip = 0x00007f6d2664b15d\n Found by: stack scanning\n 3 chrome!FetchFromCentralCache + 0x11e\n rbp = 0x7fffffffffffffff rsp = 0x00007f6d18f84030\n rip = 0x00007f6d27a3730e\n Found by: stack scanning\n 4 chrome!FetchFromCentralCache + 0x11e\n rbp = 0x7fffffffffffffff rsp = 0x00007f6d18f84040\n rip = 0x00007f6d27a3730e\n Found by: stack scanning\n 5 chrome!<name omitted> + 0x1c\n rbp = 0x7fffffffffffffff rsp = 0x00007f6d18f840e0\n rip = 0x00007f6d27a2fa4c\n Found by: stack scanning\n\nThread 5\n 0 libpthread-2.15.so + 0xbd84\n rax = 0xfffffffffffffe00 rdx = 0x0000000000000006\n rcx = 0xffffffffffffffff rbx = 0x000035290d31be80\n rsi = 0x0000000000000080 rdi = 0x000035290d31be84\n rbp = 0x000035290d45ad14 rsp = 0x00007f6d187839f0\n r8 = 0x0000000000000000 r9 = 0x0000000000000001\n r10 = 0x0000000000000000 r11 = 0x0000000000000246\n r12 = 0x000035290d31be80 r13 = 0x000035290d3da978\n r14 = 0x000035290d45ad14 r15 = 0x000035290d31bd78\n rip = 0x00007f6d26a5ed84\n Found by: given as instruction pointer in context\n 1 chrome!do_free_with_callback + 0x308\n rsp = 0x00007f6d18783ad0 rip = 0x00007f6d27a3a348\n Found by: stack scanning\n 2 chrome!do_free_with_callback + 0x308\n rsp = 0x00007f6d18783b10 rip = 0x00007f6d27a3a348\n Found by: stack scanning\n\nThread 6\n 0 libpthread-2.15.so + 0xbd84\n rax = 0xfffffffffffffe00 rdx = 0x0000000000000008\n rcx = 0xffffffffffffffff rbx = 0x000035290d31be80\n rsi = 0x0000000000000080 rdi = 0x000035290d31be84\n rbp = 0x000035290d45ad44 rsp = 0x00007f6d17d829f0\n r8 = 0x0000000000000000 r9 = 0x0000000000000002\n r10 = 0x0000000000000000 r11 = 0x0000000000000246\n r12 = 0x000035290d31be80 r13 = 0x000035290d3da9e8\n r14 = 0x000035290d45ad44 r15 = 0x000035290d31bd78\n rip = 0x00007f6d26a5ed84\n Found by: given as instruction pointer in context\n 1 chrome!do_free_with_callback + 0x308\n rsp = 0x00007f6d17d82ad0 rip = 0x00007f6d27a3a348\n Found by: stack scanning\n 2 chrome!do_free_with_callback + 0x308\n rsp = 0x00007f6d17d82b10 rip = 0x00007f6d27a3a348\n Found by: stack scanning\n\nThread 7\n 0 libpthread-2.15.so + 0xbd84\n rax = 0xfffffffffffffe00 rdx = 0x000000000000000a\n rcx = 0xffffffffffffffff rbx = 0x000035290d31be80\n rsi = 0x0000000000000080 rdi = 0x000035290d31be84\n rbp = 0x000035290d45ad64 rsp = 0x00007f6d175819f0\n r8 = 0x0000000000000000 r9 = 0x0000000000000003\n r10 = 0x0000000000000000 r11 = 0x0000000000000246\n r12 = 0x000035290d31be80 r13 = 0x000035290d3daa58\n r14 = 0x000035290d45ad64 r15 = 0x000035290d31bd78\n rip = 0x00007f6d26a5ed84\n Found by: given as instruction pointer in context\n 1 chrome!do_free_with_callback + 0x308\n rsp = 0x00007f6d17581ad0 rip = 0x00007f6d27a3a348\n Found by: stack scanning\n 2 chrome!do_free_with_callback + 0x308\n rsp = 0x00007f6d17581b10 rip = 0x00007f6d27a3a348\n Found by: stack scanning\n\nThread 8\n 0 libpthread-2.15.so + 0xbd84\n rax = 0xfffffffffffffe00 rdx = 0x000000000000000c\n rcx = 0xffffffffffffffff rbx = 0x000035290d31be80\n rsi = 0x0000000000000080 rdi = 0x000035290d31be84\n rbp = 0x000035290d45ad74 rsp = 0x00007f6d16d809f0\n r8 = 0x0000000000000000 r9 = 0x0000000000000004\n r10 = 0x0000000000000000 r11 = 0x0000000000000246\n r12 = 0x000035290d31be80 r13 = 0x000035290d3daac8\n r14 = 0x000035290d45ad74 r15 = 0x000035290d31bd78\n rip = 0x00007f6d26a5ed84\n Found by: given as instruction pointer in context\n 1 chrome!do_free_with_callback + 0x308\n rsp = 0x00007f6d16d80ad0 rip = 0x00007f6d27a3a348\n Found by: stack scanning\n 2 chrome!do_free_with_callback + 0x308\n rsp = 0x00007f6d16d80b10 rip = 0x00007f6d27a3a348\n Found by: stack scanning\n\nThread 9\n 0 libpthread-2.15.so + 0xbd84\n rax = 0xfffffffffffffe00 rdx = 0x0000000000000001\n rcx = 0xffffffffffffffff rbx = 0x000035290d31bec0\n rsi = 0x0000000000000080 rdi = 0x000035290d31bec4\n rbp = 0x000035290d45ada2 rsp = 0x00007f6d1657f9f0\n r8 = 0x0000000000000000 r9 = 0x0000000000000000\n r10 = 0x0000000000000000 r11 = 0x0000000000000246\n r12 = 0x000035290d31bec0 r13 = 0x000035290d3dab38\n r14 = 0x000035290d45ada2 r15 = 0x000035290d31bd78\n rip = 0x00007f6d26a5ed84\n Found by: given as instruction pointer in context\n 1 chrome!do_free_with_callback + 0x308\n rsp = 0x00007f6d1657fad0 rip = 0x00007f6d27a3a348\n Found by: stack scanning\n 2 chrome!do_free_with_callback + 0x308\n rsp = 0x00007f6d1657fb10 rip = 0x00007f6d27a3a348\n Found by: stack scanning\n\nThread 10\n 0 libpthread-2.15.so + 0xc0fe\n rax = 0xfffffffffffffdfc rdx = 0x0000000000000009\n rcx = 0xffffffffffffffff rbx = 0x0000000000007e1f\n rsi = 0x0000000000000089 rdi = 0x000035290d32caa4\n rbp = 0x00007f6d30901880 rsp = 0x00007f6d15d7e7d0\n r8 = 0x000035290d32ca78 r9 = 0x00000000ffffffff\n r10 = 0x00007f6d15d7e848 r11 = 0x0000000000000202\n r12 = 0x0000000000000009 r13 = 0x00007f6d15d7e848\n r14 = 0xffffffffffffff92 r15 = 0x0000000000000000\n rip = 0x00007f6d26a5f0fe\n Found by: given as instruction pointer in context\n 1 chrome!ReleaseToCentralCache + 0x12c\n rbp = 0x00007f6d30901880 rsp = 0x00007f6d15d7e900\n rip = 0x00007f6d27a371dc\n Found by: stack scanning\n 2 chrome!ListTooLong + 0x17\n rbp = 0x00007f6d30901880 rsp = 0x00007f6d15d7e970\n rip = 0x00007f6d27a37377\n Found by: stack scanning\n 3 chrome!do_free_with_callback + 0x334\n rbp = 0x00007f6d30901880 rsp = 0x00007f6d15d7e990\n rip = 0x00007f6d27a3a374\n Found by: stack scanning\n 4 chrome!erase + 0xfe\n rbp = 0x00007f6d30901880 rsp = 0x00007f6d15d7ea10\n rip = 0x00007f6d27d7af4e\n Found by: stack scanning\n\nLoaded modules:\n0x7f6d1555f000 - 0x7f6d1557efff .org.chromium.Chromium.xJksXX (deleted) ???\n0x7f6d17d84000 - 0x7f6d17f83fff .org.chromium.Chromium.FioDgb (deleted) ???\n0x7f6d1a789000 - 0x7f6d1a7a8fff .org.chromium.Chromium.sSxl3B (deleted) ???\n0x7f6d1a7a9000 - 0x7f6d1b818fff resources.pak ???\n0x7f6d1b819000 - 0x7f6d1b851fff en-US.pak ???\n0x7f6d1b852000 - 0x7f6d1b8d0fff chrome_100_percent.pak ???\n0x7f6d1b8d1000 - 0x7f6d1b923fff natives_blob.bin ???\n0x7f6d1b924000 - 0x7f6d1c2cdfff icudtl.dat ???\n0x7f6d1c2ce000 - 0x7f6d1c73cfff locale-archive ???\n0x7f6d1c73d000 - 0x7f6d1c73ffff libgpg-error.so.0.8.0 ???\n0x7f6d1c941000 - 0x7f6d1c943fff libkeyutils.so.1.4 ???\n0x7f6d1cb45000 - 0x7f6d1cb55fff libp11-kit.so.0.0.0 ???\n0x7f6d1cd57000 - 0x7f6d1cdd1fff libgcrypt.so.11.7.0 ???\n0x7f6d1cfd6000 - 0x7f6d1cfe5fff libtasn1.so.3.1.12 ???\n0x7f6d1d1e7000 - 0x7f6d1d1edfff libkrb5support.so.0.1 ???\n0x7f6d1d3ef000 - 0x7f6d1d3f1fff libcom_err.so.2.1 ???\n0x7f6d1d5f3000 - 0x7f6d1d617fff libk5crypto.so.3.1 ???\n0x7f6d1d81b000 - 0x7f6d1d8defff libkrb5.so.3.3 ???\n0x7f6d1dae9000 - 0x7f6d1daeafff libXinerama.so.1.0.0 ???\n0x7f6d1dcec000 - 0x7f6d1dd03fff libresolv-2.15.so ???\n0x7f6d1df08000 - 0x7f6d1df24fff libselinux.so.1 ???\n0x7f6d1e127000 - 0x7f6d1e14bfff libdbus-glib-1.so.2.2.2 ???\n0x7f6d1e34d000 - 0x7f6d1e35cfff libavahi-client.so.3.2.9 ???\n0x7f6d1e55e000 - 0x7f6d1e568fff libavahi-common.so.3.5.3 ???\n0x7f6d1e76a000 - 0x7f6d1e81efff libgnutls.so.26.21.8 ???\n0x7f6d1ea26000 - 0x7f6d1ea61fff libgssapi_krb5.so.2.2 ???\n0x7f6d1ec65000 - 0x7f6d1ec67fff libplds4.so ???\n0x7f6d1ee69000 - 0x7f6d1ee6cfff libplc4.so ???\n0x7f6d1f06e000 - 0x7f6d1f072fff libXdmcp.so.6.0.0 ???\n0x7f6d1f274000 - 0x7f6d1f275fff libXau.so.6.0.0 ???\n0x7f6d1f477000 - 0x7f6d1f4b2fff libpcre.so.3.12.1 ???\n0x7f6d1f6b4000 - 0x7f6d1f6bafff libffi.so.6.0.0 ???\n0x7f6d1f8bc000 - 0x7f6d1f8d1fff libz.so.1.2.3.4 ???\n0x7f6d1fad3000 - 0x7f6d1fadafff libxcb-render.so.0.0.0 ???\n0x7f6d1fcdd000 - 0x7f6d1fcdefff libxcb-shm.so.0.0.0 ???\n0x7f6d1fee0000 - 0x7f6d1ff05fff libpng12.so.0.46.0 ???\n0x7f6d20108000 - 0x7f6d20197fff libpixman-1.so.0.30.2 ???\n0x7f6d2039f000 - 0x7f6d203c7fff libpangoft2-1.0.so.0.3000.0 ???\n0x7f6d205c9000 - 0x7f6d2077cfff libc-2.15.so ??? (WARNING: No symbols, libc-2.15.so, FC1C16FC0D250E85B08A7B76BAD5D2020)\n0x7f6d20987000 - 0x7f6d2099bfff libgcc_s.so.1 ???\n0x7f6d20b9d000 - 0x7f6d20c7efff libstdc++.so.6.0.16 ???\n0x7f6d20e9d000 - 0x7f6d20ebafff libgdk_pixbuf-2.0.so.0.2600.1 ???\n0x7f6d210bd000 - 0x7f6d21169fff libgdk-x11-2.0.so.0.2400.10 ???\n0x7f6d2136f000 - 0x7f6d2179cfff libgtk-x11-2.0.so.0.2400.10 ???\n0x7f6d219aa000 - 0x7f6d219c8fff libatk-1.0.so.0.20409.1 ???\n0x7f6d21bcc000 - 0x7f6d21cb1fff libasound.so.2.0.0 ???\n0x7f6d21eb9000 - 0x7f6d21ebffff libXrandr.so.2.2.0 ???\n0x7f6d220c1000 - 0x7f6d220c2fff libXss.so.1.0.0 ???\n0x7f6d222c5000 - 0x7f6d2235bfff libfreetype.so.6.8.0 ???\n0x7f6d22562000 - 0x7f6d226a9fff libgio-2.0.so.0.3200.4 ???\n0x7f6d228b1000 - 0x7f6d228ddfff libgconf-2.so.4.1.5 ???\n0x7f6d22adf000 - 0x7f6d22b20fff libdbus-1.so.3.5.8 ???\n0x7f6d22d23000 - 0x7f6d22d56fff libfontconfig.so.1.4.4 ???\n0x7f6d22f59000 - 0x7f6d22f7ffff libexpat.so.1.5.2 ???\n0x7f6d23183000 - 0x7f6d2327dfff libm-2.15.so ???\n0x7f6d2347f000 - 0x7f6d234cefff libcups.so.2 ???\n0x7f6d236d3000 - 0x7f6d2370cfff libnspr4.so ???\n0x7f6d23912000 - 0x7f6d23930fff libsmime3.so ???\n0x7f6d23b34000 - 0x7f6d23b55fff libnssutil3.so ???\n0x7f6d23d5d000 - 0x7f6d23e46fff libnss3.so ???\n0x7f6d24050000 - 0x7f6d24052fff libgmodule-2.0.so.0.3200.4 ???\n0x7f6d24254000 - 0x7f6d24258fff libXtst.so.6.1.0 ???\n0x7f6d2445a000 - 0x7f6d24462fff libXrender.so.1.3.0 ???\n0x7f6d24664000 - 0x7f6d24672fff libXi.so.6.1.0 ???\n0x7f6d24874000 - 0x7f6d24878fff libXfixes.so.3.1.0 ???\n0x7f6d24a7a000 - 0x7f6d24a89fff libXext.so.6.4.0 ???\n0x7f6d24c8b000 - 0x7f6d24c8cfff libXdamage.so.1.1.0 ???\n0x7f6d24e8e000 - 0x7f6d24e96fff libXcursor.so.1.0.2 ???\n0x7f6d25098000 - 0x7f6d25099fff libXcomposite.so.1.0.0 ???\n0x7f6d2529b000 - 0x7f6d252b7fff libxcb.so.1.1.0 ???\n0x7f6d254b9000 - 0x7f6d254b9fff libX11-xcb.so.1.0.0 ???\n0x7f6d256bb000 - 0x7f6d257eafff libX11.so.6.3.0 ???\n0x7f6d259f0000 - 0x7f6d25ae1fff libglib-2.0.so.0.3200.4 ???\n0x7f6d25ce5000 - 0x7f6d25d31fff libgobject-2.0.so.0.3200.4 ???\n0x7f6d25f34000 - 0x7f6d25fecfff libcairo.so.2.11000.2 ???\n0x7f6d261f2000 - 0x7f6d26237fff libpango-1.0.so.0.3000.0 ???\n0x7f6d2643b000 - 0x7f6d26445fff libpangocairo-1.0.so.0.3000.0 ???\n0x7f6d26647000 - 0x7f6d2664dfff librt-2.15.so ??? (WARNING: No symbols, librt-2.15.so, 1298198CB916146CA1C82A4C884D1FA70)\n0x7f6d2684f000 - 0x7f6d26850fff libdl-2.15.so ???\n0x7f6d26a53000 - 0x7f6d26a6afff libpthread-2.15.so ??? (WARNING: No symbols, libpthread-2.15.so, 8BF9179EF7863C72A1DC42235AEAB33B0)\n0x7f6d26c70000 - 0x7f6d26c91fff ld-2.15.so ???\n0x7f6d26caf000 - 0x7f6d26e57fff snapshot_blob.bin ???\n0x7f6d26e89000 - 0x7f6d26e8ffff gconv-modules.cache ???\n0x7f6d26e95000 - 0x7f6d303f6fff chrome ??? (main) (WARNING: Corrupt symbols, chrome, 1A8ECE9F3396DE54D4501AAFB73B08180)\n0x7fffa5fe5000 - 0x7fffa5fe6fff linux-gate.so ???'