I just checked with https://goto.google.com/ngvcj build seems to be slow from build id 17811.

I just took a glance to https://build.chromium.org/p/chromium/builders/Mac/builds/17811, and felt migrating to GN could be most suspicious change in this build.

What do you think?

ipc_fuzzer is a GN-only target, so it would be new after the bot flipped to GN. I've noticed that this file takes a long time to compile locally, too, since it contains literally every IPC message in Chromium.

Perhaps we should remove it from All?

If we can remove it from `all`, it would be good for build time.
Can we do that?

Not really, if we want to keep it in the build at all; "all" means "all". 

However, ipc_fuzzer is not (AFAIK) a GN-only target, but perhaps it's not referenced from all.gyp and not built by default in GYP builds?

@tsepez - any idea what might be going on here or who should look at this?

@mbarbella has done most of the work here lately.

Sorry for the delay here. Looks like we're building these targets in all even when we haven't enabled the IPC fuzzer in gn builds (by setting enable_ipc_fuzzer = true in gn args). Definitely seems incorrect, and doesn't match the behavior from gyp builds.

For GN (but not GYP) enable_ipc_fuzzer is set to true based on some conditions here: https://cs.chromium.org/chromium/src/tools/ipc_fuzzer/ipc_fuzzer.gni?sq=package:chromium&dr=C&l=13. We can probably remove the is_mac condition if the Mac builders are too slow. 

Ah, missed that. That seems to be the issue, then. If it does seem like this build is just too slow, we can remove the condition. It will also mean that we'd lose some potential coverage in the CQ for IPC fuzzer builds, though.

The weird thing is that only Mac is slow. Does anyone know the reason?

Ah, OK, it's slow even on Linux. It's just because ipc_fuzzer is not built by default. I added enable_ipc_fuzzer = true in my gn args, and I confirmed compiling tools/ipc_fuzzer/message_lib/message_names.cc takes long time.

For the record:

local compile: 40s (on my Z620 Linux)
remote compile: 68s (goma)

So, on Linux, it's slow but not too slow. On Mac, it needs more than 10 minutes and it always causes goma timeout.

On Linux release build, debug symbol is omitted (-g0).
However, on Mac release build, debug symbol is enabled (-gdwarf-2).
When I manually set symbol_level=0 on Mac, this isn't that slow (we need 1~2 minutes, though; but not 10 minutes).

Also, goma says it requires 3GB memory to compile it. If we can split the source, it would be better (I'm not sure it is feasible, though).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/fbd5dc3864acb34635f84586470c14cc06cf09d9

commit fbd5dc3864acb34635f84586470c14cc06cf09d9
Author: shinyak <shinyak@chromium.org>
Date: Tue Aug 02 02:23:49 2016

Always local fallback for message_names.cc

On Mac, compile for tools/ipc_fuzzer/message_lib/message_names.cc is
always time out with goma.

If compile takes more than 10 minutes on goma executor, goma returns
error because of timeout. The compile for this file always spends more
than 10 minutes to wait for response from goma server, and it's re-run
locally after the timeout.

Let's make it local fallback always to make build faster.

BUG= 630502 

Review-Url: https://codereview.chromium.org/2197083002

[modify] https://crrev.com/fbd5dc3864acb34635f84586470c14cc06cf09d9/scripts/slave/compile.py

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/bfb8e163354aaced176812c22ec2ca4439e8c5d4

commit bfb8e163354aaced176812c22ec2ca4439e8c5d4
Author: recipe-roller <recipe-roller@chromium.org>
Date: Tue Aug 02 02:28:30 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/c39ead574dbf15a7b9c749b5a89d5dc92367e7e3

commit c39ead574dbf15a7b9c749b5a89d5dc92367e7e3
Author: recipe-roller <recipe-roller@chromium.org>
Date: Tue Aug 02 02:46:29 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/fbd5dc3864acb34635f84586470c14cc06cf09d9 Always local fallback for message_names.cc (shinyak@chromium.org)

R=shinyak@chromium.org
BUG= 630502 

TBR=martiniss@chromium.org,phajdan.jr@chromium.org

Review-Url: https://codereview.chromium.org/2204813002

[modify] https://crrev.com/c39ead574dbf15a7b9c749b5a89d5dc92367e7e3/infra/config/recipes.cfg

Hmm, I changed compile.py so that tools/ipc_fuzzer/message_lib/message_names.cc is compiled locally. however, it still takes more than 10 minutes.

Since goma says local_run_reason is "should fallback", I believe it's successfully fallen to 'local compile' mode. Then it's definitely slow?

Building message_names.o takes 7m40s locally for me. If I remove -gdwarf-2, it "only" takes 40s. So debug info slows down this compilation almost 10x, which seems like a compiler bug. I'll file upstream.

thakis-macpro:gn thakis$ time ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF obj/tools/ipc_fuzzer/message_lib/ipc_message_lib/message_names.o.d -DV8_DEPRECATION_WARNINGS -DENABLE_NOTIFICATIONS -DENABLE_PEPPER_CDMS -DENABLE_PLUGINS=1 -DENABLE_PDF=1 -DENABLE_PRINTING=1 -DENABLE_BASIC_PRINTING=1 -DENABLE_PRINT_PREVIEW=1 -DENABLE_SPELLCHECK=1 -DUSE_BROWSER_SPELLCHECKER=1 -DNO_TCMALLOC -DUSE_EXTERNAL_POPUP_MENU=1 -DENABLE_WEBRTC=1 -DENABLE_EXTENSIONS=1 -DENABLE_TASK_MANAGER=1 -DENABLE_THEMES=1 -DENABLE_CAPTIVE_PORTAL_DETECTION=1 -DENABLE_SESSION_SERVICE=1 -DENABLE_PLUGIN_INSTALLATION=1 -DENABLE_SUPERVISED_USERS=1 -DENABLE_SERVICE_DISCOVERY=1 -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -DENABLE_MEDIA_ROUTER=1 -DFIELDTRIAL_TESTING_ENABLED -DCR_CLANG_REVISION=274142-1 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -D__ASSERT_MACROS_DEFINE_VERSIONS_WITHOUT_UNDERSCORE=0 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DENABLE_IPC_FUZZER -DUSE_CUPS -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DU_NOEXCEPT= -DU_STATIC_IMPLEMENTATION -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DV8_USE_EXTERNAL_STARTUP_DATA -DENABLE_IPC_FUZZER -DSK_IGNORE_DW_GRAY_FIX -DSK_IGNORE_LINEONLY_AA_CONVEX_PATH_OPTS -DSK_SUPPORT_GPU=1 -DSK_BUILD_FOR_MAC -DENABLE_WEBSOCKETS -DMESA_EGL_NO_X11_HEADERS -DUSE_LIBJPEG_TURBO=1 -DENABLE_LAYOUT_UNIT_IN_INLINE_BOXES=0 -DENABLE_OILPAN=1 -DWTF_USE_CONCATENATED_IMPULSE_RESPONSES=1 -DWTF_USE_ICCJPEG=1 -DWTF_USE_QCMSLIB=1 -DFEATURE_ENABLE_SSL -DFEATURE_ENABLE_VOICEMAIL -DEXPAT_RELATIVE_PATH -DGTEST_RELATIVE_PATH -DNO_MAIN_THREAD_WRAPPING -DNO_SOUND_SYSTEM -DOSX -DWEBRTC_MAC -DWEBRTC_POSIX -DXML_STATIC -I../.. -Igen -I../../tools/ipc_fuzzer -I../../third_party/khronos -I../../gpu -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -Igen/chrome -I../../third_party/protobuf/src -Igen/protoc_out -I../../third_party/protobuf/src -I../../third_party/dom_distiller_js/dist/proto_gen -Igen/components/strings -Igen/components/strings -Igen/components/strings -Igen/components/strings -I../../skia/config -I../../skia/ext -I../../third_party/skia/include/c -I../../third_party/skia/include/config -I../../third_party/skia/include/core -I../../third_party/skia/include/effects -I../../third_party/skia/include/images -I../../third_party/skia/include/lazy -I../../third_party/skia/include/pathops -I../../third_party/skia/include/pdf -I../../third_party/skia/include/pipe -I../../third_party/skia/include/ports -I../../third_party/skia/include/utils -I../../third_party/skia/include/gpu -I../../third_party/skia/src/gpu -I../../third_party/WebKit -Igen/third_party/WebKit -I../../v8/include -Igen -I../../v8/include -I../../third_party/boringssl/src/include -I../../third_party/mesa/src/include -Igen/ui/resources -Igen/ui/resources -I../../third_party/libwebm/source -I../../third_party/opus/src/include -I../../third_party/re2/src -Igen -Igen/extensions -Igen/extensions -Igen/extensions -Igen -Igen/extensions/strings -I../../third_party/google_toolbox_for_mac -I../../third_party/google_toolbox_for_mac/src -I../../third_party/google_toolbox_for_mac/src/AppKit -I../../third_party/google_toolbox_for_mac/src/DebugUtils -I../../third_party/google_toolbox_for_mac/src/Foundation -I../../third_party/libwebp -I../../third_party/libjpeg_turbo -I../../third_party/iccjpeg -I../../third_party/libpng -I../../third_party/zlib -I../../third_party/ots/include -I../../third_party/qcms/src -I../../third_party/webrtc_overrides -I../../testing/gtest/include -I../../third_party -I../../third_party/expat/files/lib -fno-strict-aliasing -fstack-protector -fcolor-diagnostics -arch x86_64 -Wall -Werror -Wextra -Wpartial-availability -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-deprecated-register -Wno-unneeded-internal-declaration -Wno-inconsistent-missing-override -Wno-shift-negative-value -Wno-undefined-var-template -Wno-nonportable-include-path -O2 -gdwarf-2 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -mmacosx-version-min=10.7 -fvisibility=hidden -Xclang -load -Xclang ../../third_party/llvm-build/Release+Asserts/lib/libFindBadConstructs.dylib -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang check-templates -Xclang -plugin-arg-find-bad-constructs -Xclang follow-macro-expansion -Xclang -plugin-arg-find-bad-constructs -Xclang check-implicit-copy-ctors -Wheader-hygiene -Wstring-conversion -fno-threadsafe-statics -fvisibility-inlines-hidden -std=c++11 -stdlib=libc++ -fno-rtti -fno-exceptions -c ../../tools/ipc_fuzzer/message_lib/message_names.cc -o obj/tools/ipc_fuzzer/message_lib/ipc_message_lib/message_names.o

Filed https://llvm.org/bugs/show_bug.cgi?id=28820 . Somewhat interestingly, it only repros with -fno-exceptions, so a silly hack could be to build that one file with exceptions enabled :-/

Another find: Moving MessageNames::Add() out-of-line into a different .cc file makes message_names.cc build in < 10 seconds. So another workaround would be to add a message_names_hack.cc with just the definition of MessageNames::Add().

https://codereview.chromium.org/2214713003/ might be a pretty good fix.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7928cc2c14f71116bce5bd74d6c081be292082c8

commit 7928cc2c14f71116bce5bd74d6c081be292082c8
Author: thakis <thakis@chromium.org>
Date: Thu Aug 04 18:51:52 2016

Make message_names.cc build 6-30x faster.

MessageNames uses two std::map<>s where std::unordered_map<>s suffice.  Changing
this also happens to work around a clang bug that causes very slow
compile times -- see the linked bug and also http://llvm.org/PR28820

On my laptop, the file now builds in 10s instead of 67s without debug info and
in 20s instead of 10m (!) with debug info.

BUG= 630502 

Review-Url: https://codereview.chromium.org/2214713003
Cr-Commit-Position: refs/heads/master@{#409852}

[modify] https://crrev.com/7928cc2c14f71116bce5bd74d6c081be292082c8/tools/ipc_fuzzer/message_lib/message_names.h

I think this is fixed. The file builds much faster for me locally now (tens of seconds instead of many minutes – still slow, but not super slow). The goma fallback CL should be reverted.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/3275ec310356a12a988182fe0103906f4a35df34

commit 3275ec310356a12a988182fe0103906f4a35df34
Author: shinyak <shinyak@chromium.org>
Date: Mon Aug 08 01:40:27 2016

Revert of Always local fallback for message_names.cc (patchset #4 id:60001 of https://codereview.chromium.org/2197083002/ )

Reason for revert:
No need to do workaround thanks to https://codereview.chromium.org/2214713003

Original issue's description:
> Always local fallback for message_names.cc
>
> On Mac, compile for tools/ipc_fuzzer/message_lib/message_names.cc is
> always time out with goma.
>
> If compile takes more than 10 minutes on goma executor, goma returns
> error because of timeout. The compile for this file always spends more
> than 10 minutes to wait for response from goma server, and it's re-run
> locally after the timeout.
>
> Let's make it local fallback always to make build faster.
>
> BUG= 630502 
>
> Committed: https://chromium.googlesource.com/chromium/tools/build/+/fbd5dc3864acb34635f84586470c14cc06cf09d9

TBR=yyanagisawa@chromium.org,ukai@chromium.org,tikuta@chromium.org,shinyak@google.com,thakis@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 630502 

Review-Url: https://codereview.chromium.org/2220883002

[modify] https://crrev.com/3275ec310356a12a988182fe0103906f4a35df34/scripts/slave/compile.py

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/df49c8b269ef6f97298ec497268a4b22c3032d15

commit df49c8b269ef6f97298ec497268a4b22c3032d15
Author: recipe-roller <recipe-roller@chromium.org>
Date: Mon Aug 08 01:51:57 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/e665049db431e8cfe9243adb5b3216b1cdaaff52

commit e665049db431e8cfe9243adb5b3216b1cdaaff52
Author: recipe-roller <recipe-roller@chromium.org>
Date: Mon Aug 08 02:10:33 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/3275ec310356a12a988182fe0103906f4a35df34 Revert of Always local fallback for message_names.cc (patchset #4 id:60001 of https://codereview.chromium.org/2197083002/ ) (shinyak@chromium.org)

R=shinyak@chromium.org,ukai@chromium.org,thakis@chromium.org,yyanagisawa@chromium.org,tikuta@chromium.org,shinyak@google.com
BUG= 630502 

TBR=martiniss@chromium.org,phajdan.jr@chromium.org

Review-Url: https://codereview.chromium.org/2227443002

[modify] https://crrev.com/e665049db431e8cfe9243adb5b3216b1cdaaff52/infra/config/recipes.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3f28669979ad8c3a57fa8a5e9c25efb4e819a758

commit 3f28669979ad8c3a57fa8a5e9c25efb4e819a758
Author: recipe-roller <recipe-roller@chromium.org>
Date: Mon Aug 08 02:49:58 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/3275ec310356a12a988182fe0103906f4a35df34 Revert of Always local fallback for message_names.cc (patchset #4 id:60001 of https://codereview.chromium.org/2197083002/ ) (shinyak@chromium.org)

R=shinyak@chromium.org,ukai@chromium.org,thakis@chromium.org,yyanagisawa@chromium.org,tikuta@chromium.org,shinyak@google.com
BUG= 630502 

TBR=martiniss@chromium.org,phajdan.jr@chromium.org

Review-Url: https://codereview.chromium.org/2220893002
Cr-Commit-Position: refs/heads/master@{#410291}

[modify] https://crrev.com/3f28669979ad8c3a57fa8a5e9c25efb4e819a758/infra/config/recipes.cfg