New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 630905 link

Starred by 2 users

Issue metadata

Status: Archived
Owner:
Closed: Jul 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

ChromiumOS ASAN Release LKGR builder hangs during compile after switching to GN

Project Member Reported by dpranke@chromium.org, Jul 24 2016

Issue description

The "Chromium OS ASAN Release" builder on the chromium.lkgr builder hangs during compiles after switching to GN:

https://build.chromium.org/p/chromium.lkgr/builders/ChromiumOS%20ASAN%20Release/builds/3138
https://build.chromium.org/p/chromium.lkgr/builders/ChromiumOS%20ASAN%20Release

it's not immediately obvious to me what's going on, apart from apparently hanging during some step in the compile (timing out after 20 min).

My initial theory is that something is simply taking too long to compile on a machine w/ only 16GB, but from looking at a trace of the last successful GYP build in 

https://build.chromium.org/p/chromium.lkgr/builders/ChromiumOS%20ASAN%20Release/builds/3124

I didn't see anything that was particularly slow; Chrome only took about 3 minutes to link.

 
Components: Build
Cc: llozano@chromium.org glider@chromium.org kcc@chromium.org
Cc: brucedaw...@chromium.org h...@chromium.org
Trying to reproduce this locally, it looks like chrome is taking massively longer to link; gold has been running for more than a half hour on my z620 (using ~5GB of ram), so it looks like this is a simple culprit and hopefully easy enough to reproduce.

Since this is the very last linux-based builder not working on GN, it seems like it's more useful to leave this on GN and get it to work, rather than have it build in a configuration nothing else is using (i.e., GYP).

If any of the Clang / Sanitizer people wanted to look at this and/or offer theories as to why things might be so much slower apart from the general source_set/static_library issue, that would be much appreciated.
Cc: mcgrathr@chromium.org
The Linux desktop asan build appears to only take ~40s to link, so there must be something pathologically worse going on in gold.
Cc: rahulchaudhry@chromium.org
the ChromeOS ASAN builder is working fine. 

https://uberchromegw.corp.google.com/i/chromiumos/builders/amd64-generic-asan/builds/16702

The problem may be related to the version of binutils or the version of clang.


Also, the 'Linux Chromium ASAN LSan Builder' is fine:

https://build.chromium.org/p/chromium.memory/builders/Linux%20Chromium%20OS%20ASan%20LSan%20Builder

and that presumably has the same versions of binutils and clang.

Which suggests that it's related to the particular build flags.

The chromium.memory builder has:

dcheck_always_on = true
goma_dir = "/b/c/cipd/goma"
is_asan = true
is_component_build = false
is_debug = false
is_lsan = true
symbol_level = 1
target_os = "chromeos"
use_goma = true

and the LKGR builder has:

enable_ipc_fuzzer = true
goma_dir = "/b/c/cipd/goma"
is_asan = true
is_component_build = false
is_debug = false
is_lsan = true
sanitizer_coverage_flags = "edge"
target_os = "chromeos"
use_goma = true
v8_enable_verify_heap = true

Which suggests the problem is enable_ipc_fuzzer, sanitizer_coverage_flags="edge", or v8_enable_verify_heap .

Summary: ChromiumOS ASAN Release LKGR builder hangs during compile after switching to GN (was: ChroomiumOS ASAN Release LKGR builder hangs during compile after switching to GN)
It looks like I was mistaken. Chrome actually links fine, but ipc_fuzzer fails to link after taking forever to try.

The BUILD.gn file for //tools/ipc_fuzzer/fuzzer has

https://cs.chromium.org/chromium/src/tools/ipc_fuzzer/fuzzer/BUILD.gn?rcl=0&l=26

compared to 

https://cs.chromium.org/chromium/src/tools/ipc_fuzzer/fuzzer/fuzzer.gyp?rcl=0&l=29

and I'm guessing in the GN build we're deleting too many flags; in particular, it looks like we need to re-add the //build/config/sanitizers:default_sanitizer_ldflags config when we remove the :default_sanitizer_flags . Adding _ldflags appears to make things work (ipc_fuzzer links in 12 seconds locally). 

Hopefully that's all that's missing, but mbarbella@ or inferno@ may need to check and confirm that we don't need anything else.
Owner: infe...@chromium.org
Proposed fix: https://codereview.chromium.org/2178033003.

inferno@, I think you're the CF sheriff at the moment? Maybe you can pick it up from here?
Owner: dpranke@chromium.org
oh, never mind, I thought this was in a separate repo. I'll own it and hopefully inferno@ or mbarbella@ can just review the fix.
Project Member

Comment 10 by bugdroid1@chromium.org, Jul 25 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0fbf95f476ce059f7a7a1e5c8ae00e5557c45f9c

commit 0fbf95f476ce059f7a7a1e5c8ae00e5557c45f9c
Author: dpranke <dpranke@chromium.org>
Date: Mon Jul 25 22:03:12 2016

Fix ipc_fuzzer linking in CrOS ASAN builds.

R=mbarbella@chromium.org, inferno@chromium.org
BUG= 630905 

Review-Url: https://codereview.chromium.org/2178033003
Cr-Commit-Position: refs/heads/master@{#407610}

[modify] https://crrev.com/0fbf95f476ce059f7a7a1e5c8ae00e5557c45f9c/tools/ipc_fuzzer/fuzzer/BUILD.gn

Status: Fixed (was: Started)
This appears fixed now.
Labels: VerifyIn-54
Labels: VerifyIn-55

Comment 14 by dchan@chromium.org, Oct 10 2016

Labels: -VerifyIn-55

Comment 15 by dchan@google.com, Nov 19 2016

Labels: VerifyIn-56

Comment 16 by dchan@google.com, Jan 21 2017

Labels: VerifyIn-57

Comment 17 by dchan@google.com, Mar 4 2017

Labels: VerifyIn-58

Comment 18 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 19 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61

Comment 21 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)

Sign in to add a comment