The problem wasn't strictly the binary size, but the debug symbols size that exceeded 4GB. Does this issue suggest splitting the binary into multiple libraries?

We mainly want to see if debug fission https://gcc.gnu.org/wiki/DebugFission
is a feasible solution for the build.

Splitting chrome.debug into multiple files sounds like a good win for many things. I am a little worried about all the places "chrome.debug" might be hardwired, but looking for a while it might only be in afdo.py
https://code.google.com/p/chromium/codesearch#search/&q=%22chrome.debug%22%20-%22chrome.debugger%22&sq=package:chromium&type=cs

Please, make sure simple chrome workflow will still have symbols by default.

PS:
I was very surprised that my recent commands [deploy_chrome --nostrip] and [--strip-flags '-w -K "!*WebCore*"' ] do not do what I expect. No more stack traces!

It turned out that chrome_sdk now sets CFLAGS and CXXFLAGS to include "--split-dwarf" by default. I don't know who has included this in environment, but this is surely destructive. :(

We should collect more workflows like the one above relying on symbols and make sure they are not broken.

#5. split-debug has been enabled since 2014:

https://chromium-review.googlesource.com/#/c/237260/

So, what you are seeing is not recent behavior.

So, can you provide more details of what is what you were seing and what is that you are not seeing now and when did it start. 

I spent whole day first trying to find some point in the past when Debug build had symbols, and then, looking for the point where it broke.
(And actually it all started because xdai@ came to me and asked why it all did not work.)

And now I cannot reproduce the original case I had earlier. Now it seems to me that debug build has the symbols.
So probably, the problem was with outdated SDK and newer ckeckout.

For the release builds I do not see any debug symbols.
But I am not sure when its disappeared. Probably it really happened a year ago, and I was building debug builds all the time when I needed stack traces.

So now my request looks like this: it would be nice to have control on symbol information via deploy_chrome options even for release builds.

Is there any document on how to symbolize the crash report? I want to test whether debug fission will break it or not. Currently it breaks gdb...

Re #9: http://dev.chromium.org/developers/crash-reports

Another newbie question, how can I generate a crash dump locally?
I tried to crash the browser using 
chrome://inducebrowsercrashforrealz

but I could not find the dumps under 
/var/log/chrome/Crash Reports

BTW, why do you need crash dump?

Probably it would be simpler to debug on a device?

http://www.chromium.org/chromium-os/how-tos-and-troubleshooting/building-chromium-browser#TOC-Remote-GDB

I can debug Chrome built with debug fission with some hack on gdb and the Chrome ebuild. I want to see whether it affects symbolizing the core dumps.

Currently google-breakpad does not support debug fission. It could not generate
full debug symbol from binaries built with debug fission.

Can you ping the breakpad team on crash-users and figure out if they are planning to support it? It was only once mentioned last year.

I did not remember this, but
the CL https://chromium-review.googlesource.com/#/c/237260/

contains some comments about the interaction between breakpad and -gsplit-dwarf (fission). At that time it was know there was an issue there but the chrome generated with simple chrome is never used in the images we release using the release builders. The release builders use the chrome ebuild workflow. 

So, the solution for alemate@ is to disable fission in the simple chrome builds where he want to test crashdumps. 
Yunlian, could you please figure out the setting in CXXFLAGS to disable fission and document it in the simple chrome workflow page for other people needing this? 

Now, this also means that unless we fix breakpad, we cannot use fission for the release builds... Yunlian, can you try to find out if the breakpad team has something new about this (as suggested in #18)?

I have a local patch that adds debug fission support to breakpad. I also has a local patch to make gdb support .dwp in the debug dir.
I will try to upstream these patches first and then backport them to ChromeOS.

added Zel to the cc for this bug.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/portage-stable/+/bb91541c629d3cf821372f04baceadc73fd80474

commit bb91541c629d3cf821372f04baceadc73fd80474
Author: Yunlian Jiang <yunlian@google.com>
Date: Mon Apr 25 16:32:33 2016

ccache: upgrade to 3.2.4

ccache does not support debug fission until 3.2.3.
So we need to upgrade it for debug fission.

BUG= chromium:599633 
TEST=cbuildbot chromiumos-sdk falco-release

Change-Id: I81d1b68dc6f58c2b04932ff20f5782ca5977ddfd
Reviewed-on: https://chromium-review.googlesource.com/340368
Commit-Ready: Yunlian Jiang <yunlian@chromium.org>
Tested-by: Yunlian Jiang <yunlian@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[delete] https://crrev.com/9a4c9afc6a95c31a802023b7ad9a0286bf039deb/dev-util/ccache/files/ccache-3.1.10-real-temp-files.patch
[modify] https://crrev.com/bb91541c629d3cf821372f04baceadc73fd80474/dev-util/ccache/Manifest
[rename] https://crrev.com/bb91541c629d3cf821372f04baceadc73fd80474/dev-util/ccache/files/ccache-config-3
[rename] https://crrev.com/bb91541c629d3cf821372f04baceadc73fd80474/dev-util/ccache/ccache-3.2.4.ebuild
[delete] https://crrev.com/9a4c9afc6a95c31a802023b7ad9a0286bf039deb/dev-util/ccache/files/ccache-3.1.7-no-perl.patch

The patch is almost there. Testing..

I compared the symbol file generated by breakpad with/without debug fission.
The files are almost the same. Some differences are:
when build with debug fission, there are 55378 function symbols in the symbol files.

Because of the linker option --icf=all. If several functions are identical, in the symbol file, it will only choose one function to represent all the identical functions. Some times, with/without debug fission, it chooses different function to represent a function group. But they are all valid.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/bce665f5788ed80207d72333b2e2ffdc586af725

commit bce665f5788ed80207d72333b2e2ffdc586af725
Author: Yunlian Jiang <yunlian@google.com>
Date: Wed May 04 18:33:01 2016

chromeos-chrome: use debug fission for arm and x86.

It uses debug fission for arm and x86 to save space.

BUG= chromium:599633 
TEST=chrome builds with debug fission. It generates symbol files
     with updated breakpad.

Change-Id: Iddc55b10f0371369f5d51acd68695d717db3a117
Reviewed-on: https://chromium-review.googlesource.com/342471
Commit-Ready: Yunlian Jiang <yunlian@chromium.org>
Tested-by: Yunlian Jiang <yunlian@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Luis Lozano <llozano@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/bce665f5788ed80207d72333b2e2ffdc586af725/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

We need to verify this on the crash server with Chrome 52.0.2734.0.

No symbols whatsover with 2734 including 64 bit:
https://crash.corp.google.com/browse?stbtiq=chrome_chromeos%20%2052.0.2734.0

Even though symbols on the build before were fine:
https://crash.corp.google.com/browse?stbtiq=chrome_chromeos%20%2052.0.2733.0

I am afraid we need to revert this ASAP. Or do you have a change in flight to update the code on the crash server? Where is that part?

Stevenjb, could the lack of symbols on 64 bit be due to one of your changes? This change should be 32 bit only.

I can't think of what it would, but it certainly isn't impossible.

Do any of these look suspicious:
https://chromium-review.googlesource.com/#/c/344422/
https://chromium-review.googlesource.com/#/c/343930/
https://chromium-review.googlesource.com/#/c/344458/

As was pointed out in a chat, there are already crashes with such bad traces much earlier. We should track them in a new issue and wait for 32 bit crashes here.

https://crash.corp.google.com/browse?stbtiq=chrome_chromeos%20%2052.0.2727.0%20start_thread

Scary stuff.

yes, I dont see a regression here.. crashes that dont start in Chrome dont have CHrome symbols. that was happening before. Not sure if that is expected

Lower priority?

We have 20 crashes right now, *all* without symbols, all from 64 bit. There is not a single crash on 32 bit. I am willing to lower priority once we see *any* crashes on 32 bit (and file a new issue for 64 bit).

There are some 32 bit symbol crashes. They do look a little funny. Lets drop Pri until we get more data.

by using the following query on the crash server:

product.name='Chrome_ChromeOS' AND product.version='52.0.2739.0' AND cpu.architecture='arm'

I see 18 crashes for 32-bit. They come from 3 different boards: daisy, veyron_minnie-cheets and veyron_jerry.

the daisy ones look ok. They have symbols, class names and the signature seems to be the same as for crashes in previous versions.

the ones for veyron_minnie-cheets all look bad, but that is because the symbols were never uploaded for them due to issue:610902

the ones for veyron_jerry also look bad, but that signature has never been symbolized. ie: previous chrome versions also don't get symbolized. Use the following query: product.name='Chrome_ChromeOS' AND custom_data.ChromeCrashProto.magic_signature_1.name='9a39ced5_6046d414_5e2e28e5_277cf502_27719c35'

So, so far, I don't see any regressions caused by the -gsplit-debug feature. 

Is there any remaining tasks for this issue?
I see many bad veyron_minnie reports (e.g. https://crash.corp.google.com/browse?q=reportid=%27cd5e6fa900000000%27) while samus reports for the same Magic Signature are looking good (https://crash.corp.google.com/browse?q=reportid=%27f176b30100000000%27).

Symbol files were uploaded in the example above.
Probably dump_syms failed to read DWARF from the dwp file correctly, and resorted to generate imperfect symbol files from export symbols.

I think the symbol file upload is not because of debug fission. We tested that with debug fission, it still generate the symbol files correctly. 

BTW, the examples shown in #38 look fine.

The samus example is good.
The veyron_minnie example is bad because there is no valid code path to generate the stack trace displayed there.

This seems like a server-side issue as running minidump_stackwalk locally against the uploaded veyron_minnie minidump generates a fine stack trace.
Probably the server-side code is outdated?

how does the stack trace look when you get it locally?

With the veyron_minnie minidump, locally I can get the same stack trace as the samus report on the crash report.

FYI I've filed b/30860471 for this.