New issue
Advanced search Search tips

Issue 634552 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Aug 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 2
Type: Bug



Sign in to add a comment

StackOverflow in the Symbolizer Due to Singleton on Threads Where Singletons Aren't Allowed

Project Member Reported by gab@chromium.org, Aug 5 2016

Issue description

Caught when writing https://codereview.chromium.org/2213933003/.

Looks like on OS_LINUX (and I repro'ed locally in both Debug/Release), the AssertSingletonAllowed call in TaskSchedulerTaskTrackerTest.SingletonAllowed for CONTINUE_ON_SHUTDOWN [1] crashes instead of hitting the DCHECK.

More weird things:
 1) Replacing the checks and enforcement to use IOAllowed instead (https://codereview.chromium.org/2215193003) passes... but IOAllowed and SingletonAllowed use the exact same logic...?!
 2) The crash is very mysterious... to debug I removed EXPECT_DCHECK_DEATH (let the statement run free of gtest wrappers to catch full output), it indeed crashes without logging NOTREACHED(), and if adding a LOG(ERROR) right before the NOTREACHED() in AssertSingletonAllowed(), the log message spams the console for *many* repeated lines before the crash (but without the NOTREACHED() the log only appears once as expected...)?!

And these are all static calls using TLS so even if there were somehow other callers not owned by TaskTracker in the scope of this unittest, it shouldn't matter..?!

Disabling that death test on Linux for now as I'm utterly clueless...

[1] ContinueOnShutdown/TaskSchedulerTaskTrackerTest.SingletonAllowed/0
 
Project Member

Comment 1 by bugdroid1@chromium.org, Aug 5 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5e69cff93950ddd3959f995dcffff2069ff097c0

commit 5e69cff93950ddd3959f995dcffff2069ff097c0
Author: gab <gab@chromium.org>
Date: Fri Aug 05 03:25:40 2016

Change EXPECT/ASSERT_DCHECK_DEATH macro to not expose the |regex| parameter.

Most users of the macro were ignoring the regex, now it makes sure a DCHECK
caused the death (message contains "Check failed") without adding burden on
callers to encode the exact error message.

This catched an error in SchedulerLock which was generating an exception in
std::unordered_map::at() instead of hitting the DCHECK it was intending to
hit. Fixed in precursor CL @ https://codereview.chromium.org/2218443002/.

Also caught  http://crbug.com/634552 , disabled TaskSchedulerTaskTrackerTest.SingletonAllowed's
death test on OS_LINUX for now because of it.

Also updated thread_restrictions.cc to use NOTREACHED() instead of LOG(FATAL)
in order for it to generate a DCHECK failure (it's only defined when DCHECK_IS_ON()
so it makes sense to align it with that).

BUG= 553459 ,  634552 

Review-Url: https://codereview.chromium.org/2213933003
Cr-Commit-Position: refs/heads/master@{#409980}

[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/bind_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/memory/weak_ptr_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/message_loop/message_pump_io_ios_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/message_loop/message_pump_libevent_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/metrics/field_trial_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/metrics/histogram_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/metrics/persistent_sample_map_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/metrics/sample_map_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/metrics/sample_vector_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/numerics/safe_numerics_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/synchronization/atomic_flag_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/task_scheduler/priority_queue_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/task_scheduler/scheduler_lock_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/task_scheduler/scheduler_worker_pool_impl_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/task_scheduler/scheduler_worker_stack_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/task_scheduler/task_tracker_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/test/gtest_util.h
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/threading/non_thread_safe_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/threading/simple_thread_unittest.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/threading/thread_restrictions.cc
[modify] https://crrev.com/5e69cff93950ddd3959f995dcffff2069ff097c0/base/threading/thread_unittest.cc

Comment 2 by gab@chromium.org, Aug 5 2016

Note: if we can't figure it out we could at least re-enable as an EXPECT_DEATH instead of EXPECT_DCHECK_DEATH on OS_LINUX... but I'd like use to dig to the bottom of that one.
Definitely a stack overflow. Ironically, we die trying to output the stack.

The symbolizer references a singleton, which isn't allowed on this thread. That assert fails and triggers another output request to the symbolizer. Rinse and repeat.

#17 in base::ThreadRestrictions::AssertSingletonAllowed () at ../../base/threading/thread_restrictions.cc:57
#18 in base::Singleton<base::debug::SandboxSymbolizeHelper, base::DefaultSingletonTraits<base::debug::SandboxSymbolizeHelper>, base::debug::SandboxSymbolizeHelper>::get () at ../../base/memory/singleton.h:234
#19 in base::debug::SandboxSymbolizeHelper::GetInstance () at ../../base/debug/stack_trace_posix.cc:464
#20 in base::debug::SandboxSymbolizeHelper::OpenObjectFileContainingPc () at ../../base/debug/stack_trace_posix.cc:536
#21 in google::SymbolizeAndDemangle () at ../../base/third_party/symbolize/symbolize.cc:762
#22 in google::Symbolize () at ../../base/third_party/symbolize/symbolize.cc:849
#23 in base::debug::() at ../../base/debug/stack_trace_posix.cc:169
#24 in base::debug::StackTrace::OutputToStream () at ../../base/debug/stack_trace_posix.cc:742
#25 in logging::LogMessage::~LogMessage () at ../../base/logging.cc:534
#26 in base::ThreadRestrictions::AssertSingletonAllowed () at ../../base/threading/thread_restrictions.cc:57
#27 in base::Singleton<base::debug::SandboxSymbolizeHelper, base::DefaultSingletonTraits<base::debug::SandboxSymbolizeHelper>, base::debug::SandboxSymbolizeHelper>::get () at ../../base/memory/singleton.h:234
#28 in base::debug::SandboxSymbolizeHelper::GetInstance () at ../../base/debug/stack_trace_posix.cc:464
#29 in base::debug::SandboxSymbolizeHelper::OpenObjectFileContainingPc () at ../../base/debug/stack_trace_posix.cc:536
#30 in google::SymbolizeAndDemangle () at ../../base/third_party/symbolize/symbolize.cc:762
#31 in google::Symbolize () at ../../base/third_party/symbolize/symbolize.cc:849
#32 in base::debug::() at ../../base/debug/stack_trace_posix.cc:169
#33 in base::debug::StackTrace::OutputToStream () at ../../base/debug/stack_trace_posix.cc:742
#34 in logging::LogMessage::~LogMessage () at ../../base/logging.cc:534
#35 in base::ThreadRestrictions::AssertSingletonAllowed () at ../../base/threading/thread_restrictions.cc:57
#36 in base::Singleton<base::debug::SandboxSymbolizeHelper, base::DefaultSingletonTraits<base::debug::SandboxSymbolizeHelper>, base::debug::SandboxSymbolizeHelper>::get () at ../../base/memory/singleton.h:234
#37 in base::debug::SandboxSymbolizeHelper::GetInstance () at ../../base/debug/stack_trace_posix.cc:464
#38 in base::debug::SandboxSymbolizeHelper::OpenObjectFileContainingPc () at ../../base/debug/stack_trace_posix.cc:536
#39 in google::SymbolizeAndDemangle () at ../../base/third_party/symbolize/symbolize.cc:762
#40 in google::Symbolize () at ../../base/third_party/symbolize/symbolize.cc:849
#41 in base::debug::() at ../../base/debug/stack_trace_posix.cc:169
#42 in base::debug::StackTrace::OutputToStream () at ../../base/debug/stack_trace_posix.cc:742
#43 in logging::LogMessage::~LogMessage () at ../../base/logging.cc:534
#44 in base::ThreadRestrictions::AssertSingletonAllowed () at ../../base/threading/thread_restrictions.cc:57

[...]

#19313 in base::ThreadRestrictions::AssertSingletonAllowed () at ../../base/threading/thread_restrictions.cc:57
#19314 in base::Singleton<base::debug::SandboxSymbolizeHelper, base::DefaultSingletonTraits<base::debug::SandboxSymbolizeHelper>, base::debug::SandboxSymbolizeHelper>::get () at ../../base/memory/singleton.h:234
#19315 in base::debug::SandboxSymbolizeHelper::GetInstance () at ../../base/debug/stack_trace_posix.cc:464
#19316 in base::debug::SandboxSymbolizeHelper::OpenObjectFileContainingPc () at ../../base/debug/stack_trace_posix.cc:536
#19317 in google::SymbolizeAndDemangle () at ../../base/third_party/symbolize/symbolize.cc:762
#19318 in google::Symbolize () at ../../base/third_party/symbolize/symbolize.cc:849
#19319 in base::debug::() at ../../base/debug/stack_trace_posix.cc:169
#19320 in base::debug::StackTrace::OutputToStream () at ../../base/debug/stack_trace_posix.cc:742
#19321 in logging::LogMessage::~LogMessage () at ../../base/logging.cc:534
#19322 in base::ThreadRestrictions::AssertSingletonAllowed () at ../../base/threading/thread_restrictions.cc:57
#19323 in base::internal::FunctorTraits<void () at ../../base/bind_internal.h:164
#19324 in base::internal::InvokeHelper<false, void>::MakeItSo<void () at ../../base/bind_internal.h:283
#19325 in base::internal::Invoker<base::internal::BindState<void () at ../../base/bind_internal.h:346
#19326 in base::internal::Invoker<base::internal::BindState<void () at ../../base/bind_internal.h:324
#19327 in base::Callback<void () at ../../base/callback.h:389
#19328 in base::debug::TaskAnnotator::RunTask () at ../../base/debug/task_annotator.cc:51
#19329 in base::internal::TaskTracker::RunNextTaskInSequence () at ../../base/task_scheduler/task_tracker.cc:220
#19330 in base::internal::TaskSchedulerTaskTrackerTest_SingletonAllowed_Test::TestBody () at ../../base/task_scheduler/task_tracker_unittest.cc:385
#19331 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> () at ../../testing/gtest/src/gtest.cc:2402
#19332 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> () at ../../testing/gtest/src/gtest.cc:2455

Cc: fdoray@chromium.org
Status: Started (was: Assigned)
Project Member

Comment 5 by bugdroid1@chromium.org, Aug 9 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/63f454db7093f1f578fc04632c5f5423eba14cd3

commit 63f454db7093f1f578fc04632c5f5423eba14cd3
Author: robliao <robliao@chromium.org>
Date: Tue Aug 09 20:17:47 2016

Make the SandboxSymbolizeHelper Singleton Leaky and Enable Impacted Test

If the SandboxSymbolizeHelper Singleton is obtained on a non-joinable
thread, that singleton itself will assert and attempt to symbolize the
current stack, resulting in another request to get the
SandboxSymbolizeHelper Singleton, ultimately resulting in a stack
overflow.

Since the symbolizer is expected to last the entire process, it can be
leaky.

TaskSchedulerTaskTrackerTest SingletonAllowed Tests on POSIX can be run
after this fix, so it's enabled here as well.

BUG= 634552 

Review-Url: https://codereview.chromium.org/2221063004
Cr-Commit-Position: refs/heads/master@{#410792}

[modify] https://crrev.com/63f454db7093f1f578fc04632c5f5423eba14cd3/base/debug/stack_trace_posix.cc
[modify] https://crrev.com/63f454db7093f1f578fc04632c5f5423eba14cd3/base/task_scheduler/task_tracker_unittest.cc

Status: Fixed (was: Started)

Sign in to add a comment