GN config:
goma_dir = "/b/build/slave/cache/cipd/goma"
is_chrome_branded = true
is_debug = false
is_official_build = true
use_goma = true

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2daffdccc85520a52ddfaaf8739b3d08517ff1a9

commit 2daffdccc85520a52ddfaaf8739b3d08517ff1a9
Author: dpranke <dpranke@chromium.org>
Date: Wed Jun 22 18:31:16 2016

Flip 'Google Chrome Mac' builder on chromium.chrome back to GYP.

Looks like dump_syms is broken ...

TBR=rsesek@chromium.org, mark@chromium.org
NOTRY=true
BUG= 622406 

Review-Url: https://codereview.chromium.org/2083383004
Cr-Commit-Position: refs/heads/master@{#401362}

[modify] https://crrev.com/2daffdccc85520a52ddfaaf8739b3d08517ff1a9/tools/mb/mb_config.pyl

I wonder if this is because of -g in GN vs -gdwarf-2 in GYP and the old version of Xcode (see  issue 622467 ).

There should be a crash report on the bot in ~/Library/Logs/DiagnosticReports. Its stack backtrace may be helpful.

Yup, I should have pulled it when I was on the bot yesterday.

Process:         dump_syms [32881]
Path:            /b/*/dump_syms
Identifier:      dump_syms
Version:         ???
Code Type:       X86-64 (Native)
Parent Process:  Python [32877]
Responsible:     Python [240]
User ID:         500

Date/Time:       2016-06-22 12:01:27.818 -0700
OS Version:      Mac OS X 10.9.5 (13F1808)
Report Version:  11
Anonymous UUID:  2392A8DB-1973-372A-0A01-27FABDB27DAC


Crashed Thread:  0  Dispatch queue: com.apple.main-thread

Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000

Application Specific Information:
abort() called

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib              0x00007fff95101866 __pthread_kill + 10
1   libsystem_pthread.dylib             0x00007fff933d735c pthread_kill + 92
2   libsystem_c.dylib                   0x00007fff92dbeb2e abort + 125
3   dump_syms                           0x0000000104beb323 abort_message + 195
4   dump_syms                           0x0000000104beb3ee default_terminate_handler() + 190
5   dump_syms                           0x0000000104c2e8d8 std::__terminate(void (*)()) + 8
6   dump_syms                           0x0000000104c2df5b __cxa_throw + 171
7   dump_syms                           0x0000000104bd8b37 std::__1::__vector_base_common<true>::__throw_out_of_range() const + 71
8   dump_syms                           0x0000000104af1aa2 std::__1::vector<dwarf2reader::CompilationUnit::Abbrev, std::__1::allocator<dwarf2reader::CompilationUnit::Abbrev> >::at(unsigned long) + 82 (vector:1517)
9   dump_syms                           0x0000000104aee598 dwarf2reader::CompilationUnit::ProcessDIEs() + 920 (dwarf2reader.cc:587)
10  dump_syms                           0x0000000104aee17f dwarf2reader::CompilationUnit::Start() + 5231 (dwarf2reader.cc:353)
11  dump_syms                           0x0000000104b51e68 google_breakpad::DumpSymbols::ReadDwarf(google_breakpad::Module*, google_breakpad::mach_o::Reader const&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, google_breakpad::mach_o::Section, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, google_breakpad::mach_o::Section> > > const&, bool) const + 2200 (dump_syms.cc:446)
12  dump_syms                           0x0000000104b5309d google_breakpad::DumpSymbols::LoadCommandDumper::SegmentCommand(google_breakpad::mach_o::Segment const&) + 2189 (dump_syms.cc:562)
13  dump_syms                           0x0000000104b5d300 google_breakpad::mach_o::Reader::WalkLoadCommands(google_breakpad::mach_o::Reader::LoadCommandHandler*) const + 1264 (macho_reader.cc:376)
14  dump_syms                           0x0000000104b538c0 google_breakpad::DumpSymbols::ReadSymbolData(google_breakpad::Module**) + 320 (dump_syms.cc:615)
15  dump_syms                           0x0000000104b8121e Start(Options const&) + 1614 (dump_syms_tool.cc:161)
16  dump_syms                           0x0000000104b808fb main + 59 (dump_syms_tool.cc:261)
17  dump_syms                           0x0000000104ae4314 start + 52

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000000000000  rbx: 0x00007fff79869310  rcx: 0x00007fff5b1191f8  rdx: 0x0000000000000000
  rdi: 0x0000000000000507  rsi: 0x0000000000000006  rbp: 0x00007fff5b119220  rsp: 0x00007fff5b1191f8
   r8: 0x0000000000000040   r9: 0x00007fff5b119200  r10: 0x0000000008000000  r11: 0x0000000000000206
  r12: 0x00007ffea0789970  r13: 0x434c4e47432b2b00  r14: 0x0000000000000006  r15: 0x0000000104c71ac0
  rip: 0x00007fff95101866  rfl: 0x0000000000000206  cr2: 0x0000000104c30770
  
Logical CPU:     0
Error Code:      0x02000148
Trap Number:     133

That’s the std::vector<>::at() that I thought would be throwing.

dump_syms’ dwarf2reader::CompilationUnit::ReadAbbrevs() assumes that the DWARF abbreviation table is numbered starting at 1 and giving the next integer to each successive abbreviation. It doesn’t contemplate sparseness or out-of-order storage in the abbreviation table. As far as I can tell, DWARF 2 (and 3 and 4) doesn’t actually impose this requirement. There’s an assert() that checks the bogus assumption, but we only ever run dump_syms built in Release mode, which lacks an effective assertion because NDEBUG is set. Then, along comes a DIE referencing an abbreviation, and it’ll wind up pulling out either the wrong one (if it was within the size of the table) or throwing an exception.

Do we have the .dSYM that makes this go bad? It’d be so easy to change this from a vector to a map, but I want to make sure that it works first (and I want help choosing between map and hash_map, which may be significant in a .dSYM as large as Chrome’s).

llvm agrees with me (and not with dump_syms) and handles this case:

http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/DebugInfo/DWARF/DWARFDebugAbbrev.cpp?revision=227586&view=markup#l25

36        if (PrevAbbrCode + 1 != AbbrDecl.getCode()) {
37          // Codes are not consecutive, can't do O(1) lookups.
38          FirstAbbrCode = UINT32_MAX;
39        }

(to be read in conjunction with getAbbreviationDeclaration() in that same file.)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/adfae52965b56ca7a17fbd60149f78be801691f1

commit adfae52965b56ca7a17fbd60149f78be801691f1
Author: rsesek <rsesek@chromium.org>
Date: Fri Jun 24 01:56:39 2016

[Mac/GN] Use -gdwarf-2 instead of -g2 in //build/config/compiler:symbols.

This matches GYP.

BUG= 622889 , 622406 
R=dpranke@chromium.org

Review-Url: https://codereview.chromium.org/2098613003
Cr-Commit-Position: refs/heads/master@{#401781}

[modify] https://crrev.com/adfae52965b56ca7a17fbd60149f78be801691f1/build/config/compiler/BUILD.gn

We don't know this is fixed yet.

Still happening, unfortunately. I was able to grab the Framework, dSYM, and dump_syms from the bot, though, and running locally I can repro the crash with those build artifacts.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/68ce3ab21aa4ed68e31716f7ef74e3db8fecd4b3

commit 68ce3ab21aa4ed68e31716f7ef74e3db8fecd4b3
Author: dpranke <dpranke@chromium.org>
Date: Fri Jun 24 21:10:40 2016

Flip 'Google Chrome Mac' on 'chromium.chrome' back to GYP.

Looks like we're still seeing issues w/ dump_syms.

R=rsesek@chromium.org
NOTRY=true
BUG= 618468 ,  622406 

Review-Url: https://codereview.chromium.org/2097963002
Cr-Commit-Position: refs/heads/master@{#401970}

[modify] https://crrev.com/68ce3ab21aa4ed68e31716f7ef74e3db8fecd4b3/tools/mb/mb_config.pyl

If you’ve got the dSYM (and I guess Framework) readily accessible somewhere, I can work on the dump_syms side unless you’ve already started.

Uploaded the files here: https://drive.google.com/a/chromium.org/folderview?id=0B4kReKV-OI17UjhCMFlfYTRmMEE&usp=sharing

I downloaded Xcode 5.1.1 and modified the linker_driver.py to use its dsymutil and I was able to get dump_syms to crash (albeit in a different way):

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_ARITHMETIC (SIGFPE)
Exception Codes:       EXC_I386_DIV (divide by zero)

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   dump_syms                     	0x000000010d465bb0 dwarf2reader::LineInfo::ProcessOneOpcode(dwarf2reader::ByteReader*, dwarf2reader::LineInfoHandler*, dwarf2reader::LineInfoHeader const&, unsigned char const*, dwarf2reader::LineStateMachine*, unsigned long*, unsigned long, bool*) + 192 (dwarf2reader.cc:1025)
1   dump_syms                     	0x000000010d465878 dwarf2reader::LineInfo::ReadLines() + 248 (dwarf2reader.cc:1221)
2   dump_syms                     	0x000000010d465052 dwarf2reader::LineInfo::Start() + 34 (dwarf2reader.cc:919)
3   dump_syms                     	0x000000010d4c45ee google_breakpad::DumpSymbols::DumperLineToModule::ReadProgram(unsigned char const*, unsigned long long, google_breakpad::Module*, std::__1::vector<google_breakpad::Module::Line, std::__1::allocator<google_breakpad::Module::Line> >*) + 142 (dump_syms.cc:324)
4   dump_syms                     	0x000000010d4a7c6c google_breakpad::DwarfCUToModule::ReadSourceLines(unsigned long long) + 2108 (dwarf_cu_to_module.cc:827)
5   dump_syms                     	0x000000010d4a960b google_breakpad::DwarfCUToModule::Finish() + 123 (dwarf_cu_to_module.cc:1044)
6   dump_syms                     	0x000000010d455311 dwarf2reader::DIEDispatcher::EndDIE(unsigned long long) + 241 (dwarf2diehandler.cc:127)
7   dump_syms                     	0x000000010d45e4ac dwarf2reader::CompilationUnit::ProcessDIEs() + 892 (dwarf2reader.cc:587)
8   dump_syms                     	0x000000010d45e0af dwarf2reader::CompilationUnit::Start() + 5231 (dwarf2reader.cc:354)
9   dump_syms                     	0x000000010d4c1c28 google_breakpad::DumpSymbols::ReadDwarf(google_breakpad::Module*, google_breakpad::mach_o::Reader const&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, google_breakpad::mach_o::Section, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, google_breakpad::mach_o::Section> > > const&, bool) const + 2200 (dump_syms.cc:446)
10  dump_syms                     	0x000000010d4c2e5d google_breakpad::DumpSymbols::LoadCommandDumper::SegmentCommand(google_breakpad::mach_o::Segment const&) + 2189 (dump_syms.cc:562)
11  dump_syms                     	0x000000010d4cd0e0 google_breakpad::mach_o::Reader::WalkLoadCommands(google_breakpad::mach_o::Reader::LoadCommandHandler*) const + 1264 (macho_reader.cc:376)
12  dump_syms                     	0x000000010d4c3680 google_breakpad::DumpSymbols::ReadSymbolData(google_breakpad::Module**) + 320 (dump_syms.cc:615)
13  dump_syms                     	0x000000010d4f10ce Start(Options const&) + 1614 (dump_syms_tool.cc:161)
14  dump_syms                     	0x000000010d4f07ab main + 59 (dump_syms_tool.cc:261)
15  dump_syms                     	0x000000010d453f44 start + 52

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000000000009  rbx: 0x0000000000000009  rcx: 0x0000000000000000  rdx: 0x0000000000000000
  rdi: 0x00007fff527a9808  rsi: 0x00000001221ef309  rbp: 0x00007fff527a96f0  rsp: 0x00007fff527a9530
   r8: 0x00007fff527a9778   r9: 0x00007fff527a9740  r10: 0xffffffffffffffff  r11: 0x0000000000000012
  r12: 0x0000000000000000  r13: 0x0000000000000000  r14: 0x0000000000000000  r15: 0x0000000000000000
  rip: 0x000000010d465bb0  rfl: 0x0000000000010216  cr2: 0x00007fa818564000
  
Logical CPU:     6
Error Code:      0x00000000
Trap Number:     0

I then downloaded Xcode 6.1.1 and used its dsymutil and did not get a crash. So I think that's probably our best option at this point.

It turns out that while I didn't get a crash, none of the FUNC records have names (all <name omitted>), so upgrading to 6.1.1 / 6.2 does not help.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/68422c30c7bbb5ef0253e24103b09ebe858a569b

commit 68422c30c7bbb5ef0253e24103b09ebe858a569b
Author: rsesek <rsesek@chromium.org>
Date: Tue Jun 28 02:05:26 2016

[Mac/GN] Compile with -fno-standalone-debug if enable_dsyms=true.

This works around dump_syms not being able to process dSYMs produced by
dsymutil from Xcodes prior to version 7.

BUG= 622406 
R=mark@chromium.org,dpranke@chromium.org

Review-Url: https://codereview.chromium.org/2106443003
Cr-Commit-Position: refs/heads/master@{#402375}

[modify] https://crrev.com/68422c30c7bbb5ef0253e24103b09ebe858a569b/build/config/compiler/BUILD.gn

dump_syms debuggery over to bug breakpad:701.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dde87e139b67dcdd6dcb598b35f56aca2314bd47

commit dde87e139b67dcdd6dcb598b35f56aca2314bd47
Author: dpranke <dpranke@chromium.org>
Date: Wed Jun 29 02:17:30 2016

Flip Google Chrome Mac, mac trunk to GN for testing.

I will revert this CL once we have a build or two for data.

R=rsesek@chromium.org
BUG= 622406 

Review-Url: https://codereview.chromium.org/2104223002
Cr-Commit-Position: refs/heads/master@{#402662}

[modify] https://crrev.com/dde87e139b67dcdd6dcb598b35f56aca2314bd47/tools/mb/mb_config.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ae51ae23a9a17a5c693d60bb69db9e51d8390e24

commit ae51ae23a9a17a5c693d60bb69db9e51d8390e24
Author: dpranke <dpranke@chromium.org>
Date: Wed Jun 29 04:37:47 2016

Revert of Flip Google Chrome Mac, mac trunk to GN for testing. (patchset #1 id:1 of https://codereview.chromium.org/2104223002/ )

Reason for revert:
The remoting_me2me_host fix hasn't landed, and so the compile fails before we really get to test the dump_syms fix.

I'll try again tomorrow ...

Original issue's description:
> Flip Google Chrome Mac, mac trunk to GN for testing.
>
> I will revert this CL once we have a build or two for data.
>
> R=rsesek@chromium.org
> BUG= 622406 
>
> Committed: https://crrev.com/dde87e139b67dcdd6dcb598b35f56aca2314bd47
> Cr-Commit-Position: refs/heads/master@{#402662}

TBR=rsesek@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 622406 

Review-Url: https://codereview.chromium.org/2110723002
Cr-Commit-Position: refs/heads/master@{#402718}

[modify] https://crrev.com/ae51ae23a9a17a5c693d60bb69db9e51d8390e24/tools/mb/mb_config.pyl

From the trial run last night, chromium.chrome succeeded, including the dump_syms step. So I think we can call this fixed. (Though Mark is still interested in why the old dsymutil is doing such a bad job of producing DWARF).

https://build.chromium.org/p/chromium.chrome/builders/Google%20Chrome%20Mac/builds/11565
https://build.chromium.org/p/chromium.chrome/builders/Google%20Chrome%20Mac/builds/11566
https://build.chromium.org/p/chromium.chrome/builders/Google%20Chrome%20Mac/builds/11567
https://build.chromium.org/p/chromium.chrome/builders/Google%20Chrome%20Mac/builds/11568

The official.desktop.continuous failed because of a remoting issue.

We can discuss bad DWARF on breakpad:701. I think that we can be done here, provided that dump_syms actually runs to completion and produces a usable Breakpad symbol file (should have enough FUNCs, minimal <name omitted>, etc.)