Here's the backtrace.

Deleted: gdb.txt 52.3 KB

gdb.txt 52.3 KB View Download

What I actually did to get the backtrace was:

* Try the full package build, watch it fail
* From the build environment, modify the bootstrap.py command to add `-d --no-clean`, so it was: `tools/gn/bootstrap/bootstrap.py -v -d --no-clean --gn-gen-args ' is_debug=false system_libdir="lib64" google_api_key="AIzaSyDUIXvzVrt5OkVsgXhQ6NFfvWlA44by-aw" google_default_client_id="449907151817.apps.googleusercontent.com" google_default_client_secret="miEreAep8nuvTdvLums6qyLK" is_clang=false use_sysroot=false use_gold=false fieldtrial_testing_like_official_build=true ffmpeg_branding="Chromium" proprietary_codecs=false treat_warnings_as_errors=false linux_use_bundled_binutils=false use_custom_libcxx=false  use_gio=true use_pulseaudio=true icu_use_data_file=true enable_nacl=false is_component_ffmpeg=true is_component_build=true remove_webcore_debug_symbols=true enable_hangout_services_extension=true use_aura=true enable_webrtc=true enable_widevine=true use_gtk3=true'`
* Run `gdb out_bootstrap/gn`
* From gdb, start gn with the args of the run that crashes: `run gen /builddir/build/BUILD/chromium-64.0.3282.167/out/Debug --args='is_debug=false system_libdir="lib64" google_api_key="AIzaSyDUIXvzVrt5OkVsgXhQ6NFfvWlA44by-aw" google_default_client_id="449907151817.apps.googleusercontent.com" google_default_client_secret="miEreAep8nuvTdvLums6qyLK" is_clang=false use_sysroot=false use_gold=false fieldtrial_testing_like_official_build=true ffmpeg_branding="Chromium" proprietary_codecs=false treat_warnings_as_errors=false linux_use_bundled_binutils=false use_custom_libcxx=false  use_gio=true use_pulseaudio=true icu_use_data_file=true enable_nacl=false is_component_ffmpeg=true is_component_build=true remove_webcore_debug_symbols=true enable_hangout_services_extension=true use_aura=true enable_webrtc=true enable_widevine=true use_gtk3=true'`

Any difference if you remove use_custom_libcxx=false from your gn args?

Thomas: nope, same. Re-running bootstrap.py with that arg removed, once gn build is complete and it goes to "Building gn using itself to out/Debug...", it still crashes.

I've set up a Rawhide VM (Fedora-Rawhide-20180204.n.0, Workstation Live x86-64) and tried bootstrapping GN from a 66.0.3346.8 tarball. Bootstrapping with debug symbols works fine, but `gn gen' fails like this:

[0216/125939.635526:FATAL:input_file_manager.cc(287)] Check failed: input_files_.find(name) != input_files_.end().
#0  base::debug::(anonymous namespace)::DebugBreak () at /home/rakuco/chromium/chromium-66.0.3346.8/base/debug/debugger_posix.cc:239
#1  0x000000000053a925 in base::debug::BreakDebugger () at /home/rakuco/chromium/chromium-66.0.3346.8/base/debug/debugger_posix.cc:258
#2  0x000000000045ca8b in logging::LogMessage::~LogMessage (this=0x7ffff4e36540, __in_chrg=<optimized out>) at /home/rakuco/chromium/chromium-66.0.3346.8/base/logging.cc:842
#3  0x00000000008c4654 in InputFileManager::LoadFile (this=0xe1c300, origin=..., build_settings=0xe1ba50, name=..., file=0xe1ba50, err=0x7ffff4e369d0) at /home/rakuco/chromium/chromium-66.0.3346.8/tools/gn/input_file_manager.cc:287
#4  0x00000000008c4295 in InputFileManager::BackgroundLoadFile (this=0xe1c300, origin=..., build_settings=0xe1ba50, name=..., file=0xe1ba50) at /home/rakuco/chromium/chromium-66.0.3346.8/tools/gn/input_file_manager.cc:264
#5  0x00000000008cb262 in base::internal::FunctorTraits<void (InputFileManager::*)(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*), void>::Invoke<scoped_refptr<InputFileManager> const&, LocationRange const&, BuildSettings const* const&, SourceFile const&, InputFile* const&> (method=(void (InputFileManager::*)(InputFileManager * const, const LocationRange &, const BuildSettings *, const SourceFile &, InputFile *)) 0x8c4226 <InputFileManager::BackgroundLoadFile(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*)>, receiver_ptr=..., args#0=..., args#1=@0xe3a0a8: 0xe1ba50, args#2=..., args#3=@0xe3a0c8: 0xe1ba50) at /home/rakuco/chromium/chromium-66.0.3346.8/base/bind_internal.h:211
#6  0x00000000008cafe4 in base::internal::InvokeHelper<false, void>::MakeItSo<void (InputFileManager::* const&)(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*), scoped_refptr<InputFileManager> const&, LocationRange const&, BuildSettings const* const&, SourceFile const&, InputFile* const&> (functor=@0xe3a060: (void (InputFileManager::*)(InputFileManager * const, const LocationRange &, const BuildSettings *, const SourceFile &, InputFile *)) 0x8c4226 <InputFileManager::BackgroundLoadFile(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*)>, args#0=..., args#1=..., args#2=@0xe3a0a8: 0xe1ba50, args#3=..., args#4=@0xe3a0c8: 0xe1ba50) at /home/rakuco/chromium/chromium-66.0.3346.8/base/bind_internal.h:294
#7  0x00000000008c9ef4 in base::internal::Invoker<base::internal::BindState<void (InputFileManager::*)(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*), scoped_refptr<InputFileManager>, LocationRange, BuildSettings const*, SourceFile, InputFile*>, void ()>::RunImpl<void (InputFileManager::* const&)(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*), std::__1::tuple<scoped_refptr<InputFileManager>, LocationRange, BuildSettings const*, SourceFile, InputFile*> const&, 0ul, 1ul, 2ul, 3ul, 4ul>(void (InputFileManager::* const&)(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*), std::__1::tuple<scoped_refptr<InputFileManager>, LocationRange, BuildSettings const*, SourceFile, InputFile*> const&, std::__1::integer_sequence<unsigned long, 0ul, 1ul, 2ul, 3ul, 4ul>) (functor=@0xe3a060: (void (InputFileManager::*)(InputFileManager * const, const LocationRange &, const BuildSettings *, const SourceFile &, InputFile *)) 0x8c4226 <InputFileManager::BackgroundLoadFile(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*)>, bound=...) at /home/rakuco/chromium/chromium-66.0.3346.8/base/bind_internal.h:368
#8  0x00000000008c82f2 in base::internal::Invoker<base::internal::BindState<void (InputFileManager::*)(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*), scoped_refptr<InputFileManager>, LocationRange, BuildSettings const*, SourceFile, InputFile*>, void ()>::Run(base::internal::BindStateBase*) (base=0xe3a040) at /home/rakuco/chromium/chromium-66.0.3346.8/base/bind_internal.h:350
#9  0x000000000047372f in base::OnceCallback<void ()>::Run() && (this=0x7ffff4e36cc0) at /home/rakuco/chromium/chromium-66.0.3346.8/base/callback.h:65
#10 0x0000000000930162 in Scheduler::DoWork(base::OnceCallback<void ()>) (this=0xe1bcb0, closure=...) at /home/rakuco/chromium/chromium-66.0.3346.8/tools/gn/scheduler.cc:181
#11 0x0000000000933671 in base::internal::FunctorTraits<void (Scheduler::*)(base::OnceCallback<void ()>), void>::Invoke<Scheduler*, base::OnceCallback<void ()> >(void (Scheduler::*)(base::OnceCallback<void ()>), Scheduler*&&, base::OnceCallback<void ()>&&) (method=(void (Scheduler::*)(Scheduler * const, base::OnceCallback<void()>)) 0x93013e <Scheduler::DoWork(base::OnceCallback<void ()>)>, receiver_ptr=@0x7ffff4e36d50: 0xe1bcb0, args#0=...) at /home/rakuco/chromium/chromium-66.0.3346.8/base/bind_internal.h:211
#12 0x000000000093283a in base::internal::InvokeHelper<false, void>::MakeItSo<void (Scheduler::*)(base::OnceCallback<void ()>), Scheduler*, base::OnceCallback<void ()> >(void (Scheduler::*&&)(base::OnceCallback<void ()>), Scheduler*&&, base::OnceCallback<void ()>&&) (functor=@0xe38950: (void (Scheduler::*)(Scheduler * const, base::OnceCallback<void()>)) 0x93013e <Scheduler::DoWork(base::OnceCallback<void ()>)>, args#0=@0x7ffff4e36d50: 0xe1bcb0, args#1=...) at /home/rakuco/chromium/chromium-66.0.3346.8/base/bind_internal.h:294
#13 0x0000000000931676 in base::internal::Invoker<base::internal::BindState<void (Scheduler::*)(base::OnceCallback<void ()>), base::internal::UnretainedWrapper<Scheduler>, base::OnceCallback<void ()> >, void ()>::RunImpl<void (Scheduler::*)(base::OnceCallback<void ()>), std::__1::tuple<base::internal::UnretainedWrapper<Scheduler>, base::OnceCallback<void ()> >, 0ul, 1ul>(void (Scheduler::*&&)(base::OnceCallback<void ()>), std::__1::tuple<base::internal::UnretainedWrapper<Scheduler>, base::OnceCallback<void ()> >&&, std::__1::integer_sequence<unsigned long, 0ul, 1ul>) (functor=@0xe38950: (void (Scheduler::*)(Scheduler * const, base::OnceCallback<void()>)) 0x93013e <Scheduler::DoWork(base::OnceCallback<void ()>)>, bound=...) at /home/rakuco/chromium/chromium-66.0.3346.8/base/bind_internal.h:368
#14 0x0000000000930a80 in base::internal::Invoker<base::internal::BindState<void (Scheduler::*)(base::OnceCallback<void ()>), base::internal::UnretainedWrapper<Scheduler>, base::OnceCallback<void ()> >, void ()>::RunOnce(base::internal::BindStateBase*) (base=0xe38930) at /home/rakuco/chromium/chromium-66.0.3346.8/base/bind_internal.h:336
#15 0x000000000047372f in base::OnceCallback<void ()>::Run() && (this=0x7ffff4e376a0) at /home/rakuco/chromium/chromium-66.0.3346.8/base/callback.h:65
#16 0x0000000000581482 in base::debug::TaskAnnotator::RunTask (this=0x7ffff4e372e7, queue_function=0xae4630 <base::internal::(anonymous namespace)::kQueueFunctionName> "base::PostTask", pending_task=0x7ffff4e376a0) at /home/rakuco/chromium/chromium-66.0.3346.8/base/debug/task_annotator.cc:55
#17 0x00000000004a49fe in base::internal::TaskTracker::RunOrSkipTask (this=0xe16ed0, task=..., sequence=0xe3a0e0, can_run_task=true) at /home/rakuco/chromium/chromium-66.0.3346.8/base/task_scheduler/task_tracker.cc:439
#18 0x00000000005595c8 in base::internal::TaskTrackerPosix::RunOrSkipTask (this=0xe16ed0, task=..., sequence=0xe3a0e0, can_run_task=true) at /home/rakuco/chromium/chromium-66.0.3346.8/base/task_scheduler/task_tracker_posix.cc:25
#19 0x00000000004a3dfe in base::internal::TaskTracker::RunNextTask (this=0xe16ed0, sequence=..., observer=0xe1b4a0) at /home/rakuco/chromium/chromium-66.0.3346.8/base/task_scheduler/task_tracker.cc:341
#20 0x00000000006170ed in base::internal::SchedulerWorker::Thread::ThreadMain (this=0xe1b860) at /home/rakuco/chromium/chromium-66.0.3346.8/base/task_scheduler/scheduler_worker.cc:81
#21 0x0000000000559855 in base::(anonymous namespace)::ThreadFunc (params=0xe1b8f0) at /home/rakuco/chromium/chromium-66.0.3346.8/base/threading/platform_thread_posix.cc:75
#22 0x00007ffff7254574 in start_thread (arg=<optimized out>) at pthread_create.c:463
#23 0x00007ffff7b1245f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

If I comment out the libc++ bits in bootstrap.py and use the system's libstdc++, I get a backtrace similar to Adam's, with no DCHECK being hit.

Using Rawhide's clang instead of GCC 8 also succeeds, regardless of the standard C++ library being used.

This was fun :-) I'm tempted to say it's a bug in GCC 8 that fails to generate the right code here:

//tools/gn/input_file_manager.cc
bool InputFileManager::AsyncLoadFile(const LocationRange& origin,
                                     const BuildSettings* build_settings,
                                     const SourceFile& file_name,
                                     const FileLoadCallback& callback,
                                     Err* err) {
  ...
        schedule_this = base::Bind(&InputFileManager::BackgroundLoadFile,
                                 this,
                                 origin,
                                 build_settings,
                                 file_name,
                                 &data->file);

GCC seems to be having trouble passing the right addresses through the base::Bind() calls, specifically between the public BindState() constructor and one of its private overloads:

  template <typename ForwardFunctor, typename... ForwardBoundArgs>
  explicit BindState(BindStateBase::InvokeFuncStorage invoke_func,
                     ForwardFunctor&& functor,
                     ForwardBoundArgs&&... bound_args)
      // IsCancellable is std::false_type if
      // CallbackCancellationTraits<>::IsCancelled returns always false.
      // Otherwise, it's std::true_type.
      : BindState(IsCancellable{},
                  invoke_func,
                  std::forward<ForwardFunctor>(functor),
                  std::forward<ForwardBoundArgs>(bound_args)...) {}

which calls

  template <typename ForwardFunctor, typename... ForwardBoundArgs>
  explicit BindState(std::false_type,
                     BindStateBase::InvokeFuncStorage invoke_func,
                     ForwardFunctor&& functor,
                     ForwardBoundArgs&&... bound_args)
      : BindStateBase(invoke_func, &Destroy),
        functor_(std::forward<ForwardFunctor>(functor)),
        bound_args_(std::forward<ForwardBoundArgs>(bound_args)...) {
    DCHECK(!IsNull(functor_));
  }

the relevant disassembly (with Fedora 27's GCC 7.3.1-2-fc27 that works as expected) for the public constructor is

   0x000000000070d547 <+53>:    mov    %rax,%r12
   0x000000000070d54a <+56>:    mov    0x10(%rbp),%rdi
   0x000000000070d54e <+60>:    callq  0x5f100e <std::forward<SourceFile const&>(std::remove_reference<SourceFile const&>::type&)>
   0x000000000070d553 <+65>:    mov    %rax,%rbx
   ...
   0x000000000070d5a0 <+142>:   push   %rbx
   0x000000000070d5a1 <+143>:   movzbl -0x71(%rbp),%ecx
   0x000000000070d5a5 <+147>:   push   %rcx
   0x000000000070d5a6 <+148>:   mov    %r15,%r9
   0x000000000070d5a9 <+151>:   mov    %r14,%r8
   0x000000000070d5ac <+154>:   mov    %r13,%rcx
   0x000000000070d5af <+157>:   mov    %rax,%rdi
   0x000000000070d5b2 <+160>:   callq  0x70e300 <base::internal::BindState<void (InputFileManager::*)(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*), scoped_refptr<InputFileManager>, LocationRange, BuildSettings const*, SourceFile, InputFile*>::BindState<void (InputFileManager::*)(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*), InputFileManager*, LocationRange const&, BuildSettings const*&, SourceFile const&, InputFile*>(std::integral_constant<bool, false>, void (*)(), void (InputFileManager::*&&)(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*), InputFileManager*&&, LocationRange const&, BuildSettings const*&, SourceFile const&, InputFile*&&)>

and the relevant disassembly for the private constructor is

  477       explicit BindState(std::false_type,
   0x000000000070e300 <+0>:     push   %rbp
   0x000000000070e301 <+1>:     mov    %rsp,%rbp
   ...
  483             bound_args_(std::forward<ForwardBoundArgs>(bound_args)...) {
   ...
   0x000000000070e39b <+155>:   mov    0x18(%rbp),%rax
   0x000000000070e39f <+159>:   mov    %rax,%rdi
   0x000000000070e3a2 <+162>:   callq  0x5f100e <std::forward<SourceFile const&>(std::remove_reference<SourceFile const&>::type&)>

If we consider that, in reverse chronological order, %RBP, %RIP, %RCX and %RBX were pushed to the stack and %RBX contains |file_name|, mov 0x18(%rbp),%rax correctly loads the right address to pass to std::forward() there.

With Rawhide's GCC 8, the public constructor looks like this:

   0x00000000006bfccd <+53>:    mov    %rax,%r12
   0x00000000006bfcd0 <+56>:    mov    0x10(%rbp),%rdi
   0x00000000006bfcd4 <+60>:    callq  0x5e3d88 <std::forward<SourceFile const&>(std::remove_reference<SourceFile const&>::type&)>
   0x00000000006bfcd9 <+65>:    mov    %rax,%rbx
   ...
   0x00000000006bfd22 <+138>:   push   %rbx
   0x00000000006bfd23 <+139>:   mov    %r15,%r9
   0x00000000006bfd26 <+142>:   mov    %r14,%r8
   0x00000000006bfd29 <+145>:   mov    %r13,%rcx
   0x00000000006bfd2c <+148>:   mov    %rax,%rdi
   0x00000000006bfd2f <+151>:   callq  0x6c0af6 <base::internal::BindState<void (InputFileManager::*)(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*), scoped_refptr<InputFileManager>, LocationRange, BuildSettings const*, SourceFile, InputFile*>::BindState<void (InputFileManager::*)(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*), InputFileManager*, LocationRange const&, BuildSettings const*&, SourceFile const&, InputFile*>(std::integral_constant<bool, false>, void (*)(), void (InputFileManager::*&&)(LocationRange const&, BuildSettings const*, SourceFile const&, InputFile*), InputFileManager*&&, LocationRange const&, BuildSettings const*&, SourceFile const&, InputFile*&&)>

(Note how the MOVZBL and PUSH %RCX calls are missing)

The private constructor remains the same:

  477       explicit BindState(std::false_type,
   0x00000000006c0af6 <+0>:     push   %rbp
   0x00000000006c0af7 <+1>:     mov    %rsp,%rbp
   ...
  483             bound_args_(std::forward<ForwardBoundArgs>(bound_args)...) {
   ...
   0x00000000006c0b91 <+155>:   mov    0x18(%rbp),%rax
   0x00000000006c0b95 <+159>:   mov    %rax,%rdi
   0x00000000006c0b98 <+162>:   callq  0x5e3d88 <std::forward<SourceFile const&>(std::remove_reference<SourceFile const&>::type&)>

We're still trying to read 4 bytes above the stack, but since we didn't push %RCX we should be reading 3 bytes instead (i.e. MOV 0x10(%RBP), %RAX). In fact, 'print $rbp + 0x10' in GDB shows the right address for the SourceFile variable, and 'print $rbp + 0x18' shows the address of the InputFile* we pass as the last argument to the base::Base() call.

Since we pass the wrong address, everything goes downhill until things crash.

Turns out GCC is tracking this upstream:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84286

they suggest building with -fabi-version=11 as a workaround for now (that suggestion is in a Fedora downstream bug, https://bugzilla.redhat.com/show_bug.cgi?id=1546255#c1 ).

Thank you very much Raphael for really digging into this!

awilliam@: FYI it is recommended to build Chrome using the version of clang that ships with the tarballs (the clang sources are there, but not the binaries).

You can build clang with
$ tools/clang/scripts/update.py --force-local-build --without-android --use-system-cmake --if-needed --gcc-toolchain=/usr --skip-checkout

And before you bootstrap gn, set your environment variables like the following:
CC=third_party/llvm-build/Release+Asserts/bin/clang
CXX=third_party/llvm-build/Release+Asserts/bin/clang++
LD=third_party/llvm-build/Release+Asserts/bin/lld
AR=third_party/llvm-build/Release+Asserts/bin/llvm-ar

Thomas: yes, we're aware of that, thanks, but it goes against Fedora's policies. So, we get to have fun like this. :)

According to https://bugzilla.redhat.com/show_bug.cgi?id=1546255 and https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84286, the issue has been fixed in GCC, so I guess we can close this bug.