The bug here is that some targets don't link, right? pdbs being larger is generally expected.

https://build.chromium.org/p/chromium.fyi/builders/ClangToTWin(dbg) manages to link everything. Is this only with is_win_fastlink? (The bot doesn't use that.)

Yep, the bug is that some target don't link and the reason for that is that the PDBs are too big. 

There's also a few targets that are really close from the 4GB limit, including chrome.dll.pdb.

It only happen when using is_win_fastlink=true and the VS2017 linker.

I didn't knew that the PDB are expected to be bigger with Clang, a 2x size increase seems like a big issue.

Oh, it works with the 2015 linker? That'd be surprising!

clang doesn't talk to mspdbsrv. The pdb sizes should be comparable to pdb sizes you get with msvc when you set use_goma=true (which enables /Z7 instead of /Zi and makes msvc also not talk to mspdbsrv). If it's 2x the size of MSVC's pdbs when using use_goma=true, then that'd be unexpected.

Here's the size of browser_tests.exe.pdb for various configurations, with these GN args:
is_debug = true
is_component_build = true
enable_nacl = false
target_cpu = ""x86""
remove_webcore_debug_symbols = true
symbol_level = 2
is_win_fastlink = true
use_goma = true
goma_dir = ""C:\src\goma""
use_jumbo_build = true"


1) is_clang = true, using the VS2015 linker: 1,812 MB
2) is_clang = true, using the VS2017 linker: reaches 4096 MB and then the linker crashes
3) is_clang = false, using the VS2017 linker: 2,680 MB


The important flag to repro this seems to be |is_win_fastlink| . So yeah, it's 2x the size of MSVC's pdbs when using use_goma=true.

That's unexpected then. Thanks for the report!

As mentioned above I still have the builds, so I can share some build artifacts (PDBs, objs...) or do more investigation if needed.

As of today I can't build chrome.dll.pdb with symbols anymore on a non-component build, it crossed the 4GB mark.

args.gn:
is_debug = false
is_component_build = false
is_win_fastlink = true
enable_nacl = false
enable_google_now = false
symbol_level = 2
target_cpu = "x86"
use_goma = true
goma_dir="c:\src\goma-win64"

comment 7: msvc2017, yes?

Yes, MSVC2017. This used to build last week.

Component build with the same arguments as above is at 3680MBs, so it's also pretty close to the breaking point.

Thanks, I'll take a look tomorrow. As a workaround, I suppose you can use 2015 for now :-/

If anyone happens to have the linker error message in a cmd window somewhere, maybe you could paste it here so that people searching for the error message can find this bug.

Linker message:

LINK : fatal error LNK1201: error writing to program database 'D:\src\chromium\src\out\Profile\chrome_child.dll.pdb'; check for insufficient disk space, invalid path, or insufficient privilege
ninja: build stopped: subcommand failed.

Zach was looking into /debug:fastlink support recently, so he's a good person to look into this.

That's interesting that VS 2017 is generating PDBs that are so much bigger. They claimed significant speedup in fastlink linking with VC++ 2017 but I didn't realize that that came with a size cost.

There are three extra settings that may help with this. They aren't solutions because we don't want to be required to use them, but they may be worthwhile as a stop-gap measure:

target_cpu = "x86" - x86 builds are smaller and maybe the PDBs will be smaller as well?
remove_webcore_debug_symbols = true - this turns off debug information for some translation units
use_jumbo_build = true - this combines multiple translation units into one which minimizes duplication

If push comes to shove we could start requiring use_jumbo_build for some configurations, and we could start using this setting more aggressively in order to get more benefits. We clearly don't want to start requiring that developers use x86 builds or remove_webcore_debug_symbols.

I did some tests with a 32-bit debug component build with my recommended options and I see (with the VC++ 2017 linker) that the clang PDBs average 1.8x larger, with chrome.dll.pdb hitting 3.8 GB. This is synced to R494597 from August 15th. My settings are:

is_component_build = true
is_debug = true
target_cpu = "x86"
enable_nacl = false
use_goma = true
is_win_fastlink = true
symbol_level = 2
remove_webcore_debug_symbols = true
use_jumbo_build = true

and my largest PDBs for both builds are:

>dir out\debug_component_clang\*.pdb /os | tail -15
08/16/2017  03:50 PM        95,973,376 ui_views_mus_lib.dll.pdb
08/16/2017  03:49 PM       119,156,736 gpu.dll.pdb
08/16/2017  03:50 PM       121,163,776 service.dll.pdb
08/16/2017  03:50 PM       123,047,936 aura.dll.pdb
08/16/2017  03:50 PM       152,236,032 ui.service.exe.pdb
08/16/2017  03:58 PM       166,916,096 ppapi_proxy.dll.pdb
08/16/2017  04:01 PM       169,783,296 headless.dll.pdb
08/16/2017  03:49 PM       193,613,824 cc.dll.pdb
08/16/2017  03:50 PM       195,514,368 views.dll.pdb
08/16/2017  03:47 PM       233,263,104 libGLESv2.dll.pdb
08/16/2017  03:49 PM       437,006,336 net.dll.pdb
08/16/2017  04:01 PM     2,685,251,584 content.dll.pdb
08/16/2017  04:03 PM     3,804,278,784 chrome.dll.pdb
             200 File(s) 10,511,048,704 bytes

>dir out\debug_component\*.pdb /os | tail -15
08/16/2017  04:07 PM        66,965,504 ppapi_proxy.dll.pdb
08/16/2017  04:08 PM        68,104,192 ui_views_mus_lib.dll.pdb
08/16/2017  04:08 PM        76,484,608 blink_core.dll.pdb
08/16/2017  04:07 PM        79,704,064 aura.dll.pdb
08/16/2017  04:07 PM        81,498,112 service.dll.pdb
08/16/2017  04:10 PM        87,642,112 headless.dll.pdb
08/16/2017  04:07 PM        98,471,936 blink_platform.dll.pdb
08/16/2017  04:07 PM       100,143,104 ui.service.exe.pdb
08/16/2017  04:07 PM       105,836,544 views.dll.pdb
08/16/2017  04:06 PM       117,125,120 libGLESv2.dll.pdb
08/16/2017  04:07 PM       158,117,888 net.dll.pdb
08/16/2017  04:10 PM     1,181,159,424 content.dll.pdb
08/16/2017  04:13 PM     1,975,037,952 chrome.dll.pdb
             200 File(s)  5,745,647,616 bytes

Woah. clang and fastlink and VS 2017 really don't get along, but clang and fastlink and VS 2015 get along just fine.

>dir out\debug_component_clang_2015\*.pdb /os | tail -15
08/16/2017  04:57 PM        56,709,120 ppapi_proxy.dll.pdb
08/16/2017  04:51 PM        56,938,496 service.dll.pdb
08/16/2017  04:50 PM        59,183,104 gpu.dll.pdb
08/16/2017  04:51 PM        65,581,056 views.dll.pdb
08/16/2017  04:51 PM        65,769,472 cc.dll.pdb
08/16/2017  04:51 PM        65,892,352 ui.service.exe.pdb
08/16/2017  05:02 PM        75,108,352 headless.dll.pdb
08/16/2017  05:00 PM        75,788,288 blink_core.dll.pdb
08/16/2017  04:58 PM        84,504,576 blink_platform.dll.pdb
08/16/2017  04:49 PM        86,585,344 libGLESv2.dll.pdb
08/16/2017  04:50 PM       164,032,512 net.dll.pdb
08/16/2017  05:02 PM       930,729,984 content.dll.pdb
08/16/2017  05:08 PM     1,362,333,696 chrome.dll.pdb
             200 File(s)  4,394,229,760 bytes

Unless I've done something terribly wrong in my science this means that the VS 2017 linker is making the PDBs 2.4x larger!

There is no args.gn variable to indicate the VS compiler version so I set GYP_MSVS_VERSION=2015 and then ran "gclient runhooks" before doing the 2015 build in a clean directory with the same gn args. I had previously set GYP_MSVS_VERSION=2017 for the clang and VS builds in the previous comment.

And, for completeness, a full VS 2015 build (no clang).

>dir out\debug_component_2015\*.pdb /os | tail -15
08/16/2017  05:23 PM        43,061,248 views.dll.pdb
08/16/2017  05:21 PM        45,707,264 service.dll.pdb
08/16/2017  05:21 PM        46,338,048 gpu.dll.pdb
08/16/2017  05:20 PM        49,057,792 skia.dll.pdb
08/16/2017  05:23 PM        51,499,008 ui.service.exe.pdb
08/16/2017  05:19 PM        54,931,456 libGLESv2.dll.pdb
08/16/2017  05:24 PM        62,459,904 blink_modules.dll.pdb
08/16/2017  05:26 PM        64,974,848 headless.dll.pdb
08/16/2017  05:24 PM        79,933,440 blink_core.dll.pdb
08/16/2017  05:23 PM       103,698,432 blink_platform.dll.pdb
08/16/2017  05:21 PM       110,948,352 net.dll.pdb
08/16/2017  05:26 PM       619,016,192 content.dll.pdb
08/16/2017  05:36 PM       932,712,448 chrome.dll.pdb
             200 File(s)  3,376,218,112 bytes

So:
If using VC++ then switching from 2015 to 2017 increases PDB sizes by 70%
If using clang then switching from 2015 to 2017 increases PDB sizes by 140%
If using the VC++ 2015 toolchain then switching to the clang compiler increases PDB sizes by 30%
If using the VC++ 2017 toolchain then switching to the clang compiler increases PDB sizes by 80%
Going from VC++ 2015 to VC++ 2017 plus clang increases PDB sizes by 210% (1.7 * 1.8 = 3.06, 2.4 * 1.3 = 3.12)

These results don't necessarily apply to other configurations, and are believed to be primarily triggered by fastlink.

Here's some numbers I took against two copies of llvm-tblgen.pdb, one where the object files were built with clang-cl and the other where the object files were built with cl.

https://docs.google.com/spreadsheets/d/1lm8HR_OUQodUo3217ixUUS-vTewJnBx57k8kW5cd-ok/edit#gid=192877472

Most of the difference comes from certain object files, which incidentally all appear to be of the form .*Emitter.cpp.  I'm sure you could glance at the file and know in a second what kind of strange pattern they all share, but whatever they are doing seems to be mostly responsible for the bloat (18MB/23MB of the size regression comes entirely from those files)

Why doesn't this happen when linking with VS 2015?  And what's ending up in the PDB that shouldn't be?  That's a question for tomorrow.

(BTW, you'll want to look at "Sheet 2" in that doc)

I'm now tracking this officially on the LLVM side here: https://bugs.llvm.org/show_bug.cgi?id=34239

TL;DR - We write a different symbol than they do for local variables and function parameters, and I strongly suspect this to be the culprit.

I filed a VC++ bug for this issue - it mostly just links back to this:

https://developercommunity.visualstudio.com/content/problem/101930/pdbs-created-with-debugfastlink-and-clang-cl-are-h.html

I'm not convinced this is necessarily a VC bug, but doesn't hurt to see what they say I guess.

I've checked in some changes that close about 50% of the size gap between us and MSVC.  Details are in the LLVM tracking bug [https://bugs.llvm.org/show_bug.cgi?id=34239].  Similar improvements can shave off another 50% or more, but they will take a little more investigation to nail down.  In the meantime, hopefully the current set of improvements at least unblock us.

Marking this fixed.  It's going to take me a couple of hours to finish a chrome build to get numbers for chrome.dll, but I'm seeing reductions of 50-60% in PDB size across the board after LLVM revision r312583, and in all cases /DEBUG:FASTLINK PDB from clang-cl + link is now smaller than from cl + link

Just to follow up with some numbers, I finally got a build of chrome completed (took literally all night since I couldn't use goma).  Without the patch, I had:

08/16/2017  01:49 PM     1,079,554,048 content.dll.pdb
08/16/2017  12:56 PM     1,365,020,672 chrome.dll.pdb

with the patch, against the same source tree, I had:

09/05/2017  08:08 PM       528,838,656 content.dll.pdb
09/06/2017  08:43 AM       777,490,432 chrome.dll.pdb

It also turns out this is a regression introduced several weeks ago.  We've added PDBs to the perf bots that track binary size so we should find this if it happens again in the future.

Thanks for the quick fix! 

Could you make sure that you link to this crbug when you'll roll Clang deps? (if it hasn't already been done). I'd like to switch back to using Clang for my local builds.

This is fixed upstream, but it's not in Chromium yet. Let's reopen this until we roll clang, which should happen soon.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/01f121e242233f5fbbe2669d5d10df6fc0682949

commit 01f121e242233f5fbbe2669d5d10df6fc0682949
Author: Hans Wennborg <hans@chromium.org>
Date: Fri Sep 08 16:45:43 2017

Roll Clang 310694-2:312679

And remove the local patch.

Bug:  760615 , 755611 
Change-Id: Id131f345bdc7d30bbec00e22c562f2662e8bf7ed
Reviewed-on: https://chromium-review.googlesource.com/656062
Reviewed-by: Zachary Turner <zturner@chromium.org>
Commit-Queue: Hans Wennborg <hans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#500608}
[delete] https://crrev.com/71a5f211c38e0f318896ea1492e5dbd82f176ec1/tools/clang/scripts/InstructionCombining.cpp
[modify] https://crrev.com/01f121e242233f5fbbe2669d5d10df6fc0682949/tools/clang/scripts/update.py

I just did some tests with the latest clang, VS 2017, and the build settings listed in comment #13. The biggest PDBs from building the chrome target are:

clang with VS 2017 linker
>dir *.pdb /os | tail -15
09/08/2017  10:48 AM        50,376,704 service.dll.pdb
09/08/2017  10:54 AM        50,401,280 ppapi_proxy.dll.pdb
09/08/2017  10:48 AM        52,211,712 cc.dll.pdb
09/08/2017  10:56 AM        53,268,480 blink_modules.dll.pdb
09/08/2017  10:48 AM        58,109,952 ui.service.exe.pdb
09/08/2017  10:49 AM        62,722,048 views.dll.pdb
09/08/2017  10:46 AM        68,456,448 libGLESv2.dll.pdb
09/08/2017  10:56 AM        74,649,600 blink_core.dll.pdb
09/08/2017  10:55 AM        80,285,696 blink_platform.dll.pdb
09/08/2017  10:57 AM        84,611,072 headless.dll.pdb
09/08/2017  10:48 AM       140,447,744 net.dll.pdb
09/08/2017  10:57 AM       889,622,528 content.dll.pdb
09/08/2017  11:00 AM     1,208,422,400 chrome.dll.pdb
             202 File(s)  3,945,578,496 bytes

That's 17% larger than VC++ 2015 (compiler and linker), so a modest increase in size, but that increase comes from changes to the VC++ 2017 linker, not the clang compiler. Here's a comparison done today with VS 2017, no clang:

VC++ 2017 with VS 2017 linker
>dir *.pdb /os | tail -15
09/08/2017  11:53 AM        67,735,552 ppapi_proxy.dll.pdb
09/08/2017  11:51 AM        69,316,608 ui_views_mus_lib.dll.pdb
09/08/2017  11:54 AM        77,565,952 blink_core.dll.pdb
09/08/2017  11:51 AM        80,687,104 aura.dll.pdb
09/08/2017  11:51 AM        83,496,960 service.dll.pdb
09/08/2017  11:56 AM        92,205,056 headless.dll.pdb
09/08/2017  11:51 AM       100,536,320 ui.service.exe.pdb
09/08/2017  11:53 AM       101,822,464 blink_platform.dll.pdb
09/08/2017  11:51 AM       106,729,472 views.dll.pdb
09/08/2017  11:49 AM       118,583,296 libGLESv2.dll.pdb
09/08/2017  11:50 AM       158,912,512 net.dll.pdb
09/08/2017  11:56 AM     1,258,639,360 content.dll.pdb
09/08/2017  11:59 AM     2,049,044,480 chrome.dll.pdb
             202 File(s)  5,938,790,400 bytes

So, clang + VS 2017 generates PDBs that are 62.5% smaller than a few weeks ago, and 33.6% smaller than VS 2017's compiler+linker.

The main conclusions are:
1) VS 2017 (with or without clang) creates larger fastlink PDBs than VS 2015
2) Compiling with clang now creates smaller fastlink PDBs than compiling with the VC++ compiler
3) fastlink PDB size is no longer a blocking issue for upgrading to VC++ 2017
4) The fastlink PDB size crisis appears to be fully resolved

This is fantastic, thanks everyone, especially Zach for the quick and thorough investigation.

Let's mark it fixed assuming the roll sticks.