ChromiumOS ASAN Release LKGR builder hangs during compile after switching to GN |
|||||||||||||||||||
Issue descriptionThe "Chromium OS ASAN Release" builder on the chromium.lkgr builder hangs during compiles after switching to GN: https://build.chromium.org/p/chromium.lkgr/builders/ChromiumOS%20ASAN%20Release/builds/3138 https://build.chromium.org/p/chromium.lkgr/builders/ChromiumOS%20ASAN%20Release it's not immediately obvious to me what's going on, apart from apparently hanging during some step in the compile (timing out after 20 min). My initial theory is that something is simply taking too long to compile on a machine w/ only 16GB, but from looking at a trace of the last successful GYP build in https://build.chromium.org/p/chromium.lkgr/builders/ChromiumOS%20ASAN%20Release/builds/3124 I didn't see anything that was particularly slow; Chrome only took about 3 minutes to link.
,
Jul 24 2016
,
Jul 24 2016
Trying to reproduce this locally, it looks like chrome is taking massively longer to link; gold has been running for more than a half hour on my z620 (using ~5GB of ram), so it looks like this is a simple culprit and hopefully easy enough to reproduce. Since this is the very last linux-based builder not working on GN, it seems like it's more useful to leave this on GN and get it to work, rather than have it build in a configuration nothing else is using (i.e., GYP). If any of the Clang / Sanitizer people wanted to look at this and/or offer theories as to why things might be so much slower apart from the general source_set/static_library issue, that would be much appreciated.
,
Jul 24 2016
The Linux desktop asan build appears to only take ~40s to link, so there must be something pathologically worse going on in gold.
,
Jul 25 2016
the ChromeOS ASAN builder is working fine. https://uberchromegw.corp.google.com/i/chromiumos/builders/amd64-generic-asan/builds/16702 The problem may be related to the version of binutils or the version of clang.
,
Jul 25 2016
Also, the 'Linux Chromium ASAN LSan Builder' is fine: https://build.chromium.org/p/chromium.memory/builders/Linux%20Chromium%20OS%20ASan%20LSan%20Builder and that presumably has the same versions of binutils and clang. Which suggests that it's related to the particular build flags. The chromium.memory builder has: dcheck_always_on = true goma_dir = "/b/c/cipd/goma" is_asan = true is_component_build = false is_debug = false is_lsan = true symbol_level = 1 target_os = "chromeos" use_goma = true and the LKGR builder has: enable_ipc_fuzzer = true goma_dir = "/b/c/cipd/goma" is_asan = true is_component_build = false is_debug = false is_lsan = true sanitizer_coverage_flags = "edge" target_os = "chromeos" use_goma = true v8_enable_verify_heap = true Which suggests the problem is enable_ipc_fuzzer, sanitizer_coverage_flags="edge", or v8_enable_verify_heap .
,
Jul 25 2016
It looks like I was mistaken. Chrome actually links fine, but ipc_fuzzer fails to link after taking forever to try. The BUILD.gn file for //tools/ipc_fuzzer/fuzzer has https://cs.chromium.org/chromium/src/tools/ipc_fuzzer/fuzzer/BUILD.gn?rcl=0&l=26 compared to https://cs.chromium.org/chromium/src/tools/ipc_fuzzer/fuzzer/fuzzer.gyp?rcl=0&l=29 and I'm guessing in the GN build we're deleting too many flags; in particular, it looks like we need to re-add the //build/config/sanitizers:default_sanitizer_ldflags config when we remove the :default_sanitizer_flags . Adding _ldflags appears to make things work (ipc_fuzzer links in 12 seconds locally). Hopefully that's all that's missing, but mbarbella@ or inferno@ may need to check and confirm that we don't need anything else.
,
Jul 25 2016
Proposed fix: https://codereview.chromium.org/2178033003. inferno@, I think you're the CF sheriff at the moment? Maybe you can pick it up from here?
,
Jul 25 2016
oh, never mind, I thought this was in a separate repo. I'll own it and hopefully inferno@ or mbarbella@ can just review the fix.
,
Jul 25 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0fbf95f476ce059f7a7a1e5c8ae00e5557c45f9c commit 0fbf95f476ce059f7a7a1e5c8ae00e5557c45f9c Author: dpranke <dpranke@chromium.org> Date: Mon Jul 25 22:03:12 2016 Fix ipc_fuzzer linking in CrOS ASAN builds. R=mbarbella@chromium.org, inferno@chromium.org BUG= 630905 Review-Url: https://codereview.chromium.org/2178033003 Cr-Commit-Position: refs/heads/master@{#407610} [modify] https://crrev.com/0fbf95f476ce059f7a7a1e5c8ae00e5557c45f9c/tools/ipc_fuzzer/fuzzer/BUILD.gn
,
Jul 27 2016
This appears fixed now.
,
Aug 29 2016
,
Oct 7 2016
,
Oct 10 2016
,
Nov 19 2016
,
Jan 21 2017
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Oct 14 2017
|
|||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||
Comment 1 by dpranke@chromium.org
, Jul 24 2016