New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 663557 link

Starred by 2 users
Status: Fixed
Owner:
█ roc...@chromium.org
please use my google.com address
Closed: Nov 2016
Cc:
joedow@chromium.org
█ roc...@chromium.org
dcheng@chromium.org
█ tibell@chromium.org
sa...@chromium.org
benwells@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Lock order inversion reported in mojo

Project Member Reported by benwells@chromium.org, Nov 9 2016 
See TSan bot for details:

https://build.chromium.org/p/chromium.memory.full/builders/Linux%20TSan%20Tests/builds/3361

Error output:

BackgroundSyncBrowserTest.WaitUntil (run #1):
[ RUN      ] BackgroundSyncBrowserTest.WaitUntil
[7919:7919:1107/164548:8543550337:WARNING:audio_manager.cc(317)] Multiple instances of AudioManager detected
[7919:7919:1107/164548:8543550577:WARNING:audio_manager.cc(278)] Multiple instances of AudioManager detected
Xlib:  extension "RANDR" missing on display ":9".
[7919:7919:1107/164550:8544612436:INFO:CONSOLE(13)] "sendResultToTest: ok - service worker registered", source: https://127.0.0.1:49511/background_sync/background_sync_test_helpers.js (13)
[7919:7919:1107/164550:8544838762:INFO:CONSOLE(13)] "sendResultToTest: ok - delay registered", source: https://127.0.0.1:49511/background_sync/background_sync_test_helpers.js (13)
[7919:7919:1107/164550:8544851181:INFO:CONSOLE(13)] "sendResultToTest: ok - delay found", source: https://127.0.0.1:49511/background_sync/background_sync_test_helpers.js (13)
[7919:7919:1107/164550:8544869255:INFO:CONSOLE(13)] "sendResultToTest: ok - delay completing", source: https://127.0.0.1:49511/background_sync/background_sync_test_helpers.js (13)
[7919:7919:1107/164550:8544920843:INFO:CONSOLE(13)] "sendResultToTest: ok - delay completed", source: https://127.0.0.1:49511/background_sync/background_sync_test_helpers.js (13)
[7919:7919:1107/164550:8544927307:INFO:CONSOLE(13)] "sendResultToTest: error - delay not found", source: https://127.0.0.1:49511/background_sync/background_sync_test_helpers.js (13)
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=7919)
  Cycle in lock order graph: M2764 (0x7b140002fb88) => M2768 (0x7b0c00031140) => M2764

  Mutex M2768 acquired here while holding mutex M2764 in thread T12:
    #0 pthread_mutex_lock <null> (content_browsertests+0x0000004cc4de)
    #1 base::internal::LockImpl::Lock() base/synchronization/lock_impl_posix.cc:64:12 (content_browsertests+0x0000028f07a5)
    #2 Acquire base/synchronization/lock.h:26:26 (content_browsertests+0x00000255f740)
    #3 MayAutoLock mojo/public/cpp/bindings/lib/may_auto_lock.h:17 (content_browsertests+0x00000255f740)
    #4 mojo::internal::MultiplexRouter::Accept(mojo::Message*) mojo/public/cpp/bindings/lib/multiplex_router.cc:526 (content_browsertests+0x00000255f740)
    #5 mojo::FilterChain::Accept(mojo::Message*) mojo/public/cpp/bindings/lib/filter_chain.cc:40:17 (content_browsertests+0x00000255871d)
    #6 mojo::Connector::ReadSingleMessage(unsigned int*) mojo/public/cpp/bindings/lib/connector.cc:247:51 (content_browsertests+0x000002556915)
    #7 ReadAllAvailableMessages mojo/public/cpp/bindings/lib/connector.cc:272:10 (content_browsertests+0x000002556d27)
    #8 OnHandleReadyInternal mojo/public/cpp/bindings/lib/connector.cc:205 (content_browsertests+0x000002556d27)
    #9 mojo::Connector::OnWatcherHandleReady(unsigned int) mojo/public/cpp/bindings/lib/connector.cc:183 (content_browsertests+0x000002556d27)
    #10 Invoke<mojo::Connector *, unsigned int> base/bind_internal.h:214:12 (content_browsertests+0x000002556ed4)
    #11 MakeItSo<void (mojo::Connector::*const &)(unsigned int), mojo::Connector *, unsigned int> base/bind_internal.h:285 (content_browsertests+0x000002556ed4)
    #12 RunImpl<void (mojo::Connector::*const &)(unsigned int), const std::__1::tuple<base::internal::UnretainedWrapper<mojo::Connector> > &, 0> base/bind_internal.h:361 (content_browsertests+0x000002556ed4)
    #13 base::internal::Invoker<base::internal::BindState<void (mojo::Connector::*)(unsigned int), base::internal::UnretainedWrapper<mojo::Connector> >, void (unsigned int)>::Run(base::internal::BindStateBase*, unsigned int&&) base/bind_internal.h:339 (content_browsertests+0x000002556ed4)
    #14 Run base/callback.h:64:12 (content_browsertests+0x000002b749b9)
    #15 OnHandleReady mojo/public/cpp/system/watcher.cc:122 (content_browsertests+0x000002b749b9)
    #16 mojo::Watcher::CallOnHandleReady(unsigned long, unsigned int, MojoHandleSignalsState, unsigned int) mojo/public/cpp/system/watcher.cc:142 (content_browsertests+0x000002b749b9)
    #17 mojo::edk::(anonymous namespace)::CallWatchCallback(void (*)(unsigned long, unsigned int, MojoHandleSignalsState, unsigned int), unsigned long, unsigned int, mojo::edk::HandleSignalsState const&, unsigned int) mojo/edk/system/core.cc:57:3 (content_browsertests+0x00000251c05c)
    #18 Invoke<void (*const &)(unsigned long, unsigned int, MojoHandleSignalsState, unsigned int), const unsigned long &, unsigned int, const mojo::edk::HandleSignalsState &, unsigned int> base/bind_internal.h:164:12 (content_browsertests+0x0000025207ad)
    #19 MakeItSo<void (*const &)(void (*)(unsigned long, unsigned int, MojoHandleSignalsState, unsigned int), unsigned long, unsigned int, const mojo::edk::HandleSignalsState &, unsigned int), void (*const &)(unsigned long, unsigned int, MojoHandleSignalsState, unsigned int), const unsigned long &, unsigned int, const mojo::edk::HandleSignalsState &, unsigned int> base/bind_internal.h:285 (content_browsertests+0x0000025207ad)
    #20 RunImpl<void (*const &)(void (*)(unsigned long, unsigned int, MojoHandleSignalsState, unsigned int), unsigned long, unsigned int, const mojo::edk::HandleSignalsState &, unsigned int), const std::__1::tuple<void (*)(unsigned long, unsigned int, MojoHandleSignalsState, unsigned int), unsigned long> &, 0, 1> base/bind_internal.h:361 (content_browsertests+0x0000025207ad)
    #21 base::internal::Invoker<base::internal::BindState<void (*)(void (*)(unsigned long, unsigned int, MojoHandleSignalsState, unsigned int), unsigned long, unsigned int, mojo::edk::HandleSignalsState const&, unsigned int), void (*)(unsigned long, unsigned int, MojoHandleSignalsState, unsigned int), unsigned long>, void (unsigned int, mojo::edk::HandleSignalsState const&, unsigned int)>::Run(base::internal::BindStateBase*, unsigned int&&, mojo::edk::HandleSignalsState const&, unsigned int&&) base/bind_internal.h:339 (content_browsertests+0x0000025207ad)
    #22 Run base/callback.h:64:12 (content_browsertests+0x000002547d49)
    #23 mojo::edk::Watcher::MaybeInvokeCallback(unsigned int, mojo::edk::HandleSignalsState const&, unsigned int) mojo/edk/system/watcher.cc:24 (content_browsertests+0x000002547d49)
    #24 mojo::edk::RequestContext::~RequestContext() mojo/edk/system/request_context.cc:59:22 (content_browsertests+0x00000253f128)
    #25 mojo::edk::RequestContext::~RequestContext() mojo/edk/system/request_context.cc:60:5 (content_browsertests+0x00000253f133)
    #26 mojo::edk::NodeChannel::OnChannelMessage(void const*, unsigned long, std::__1::unique_ptr<std::__1::vector<mojo::edk::PlatformHandle, std::__1::allocator<mojo::edk::PlatformHandle> >, mojo::edk::PlatformHandleVectorDeleter>) mojo/edk/system/node_channel.cc:766:1 (content_browsertests+0x000002552e30)
    #27 mojo::edk::Channel::OnReadComplete(unsigned long, unsigned long*) mojo/edk/system/channel.cc:563:18 (content_browsertests+0x00000254b9eb)
    #28 mojo::edk::(anonymous namespace)::ChannelPosix::OnFileCanReadWithoutBlocking(int) mojo/edk/system/channel_posix.cc:314:14 (content_browsertests+0x00000254de8e)
    #29 non-virtual thunk to mojo::edk::(anonymous namespace)::ChannelPosix::OnFileCanReadWithoutBlocking(int) mojo/edk/system/channel_posix.cc:274:8 (content_browsertests+0x00000254e0f4)
    #30 OnFileCanReadWithoutBlocking base/message_loop/message_pump_libevent.cc:96:13 (content_browsertests+0x0000028b5844)
    #31 base::MessagePumpLibevent::OnLibeventNotification(int, short, void*) base/message_loop/message_pump_libevent.cc:337 (content_browsertests+0x0000028b5844)
    #32 event_process_active base/third_party/libevent/event.c:381:4 (content_browsertests+0x00000295c025)
    #33 event_base_loop base/third_party/libevent/event.c:521 (content_browsertests+0x00000295c025)
    #34 base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) base/message_loop/message_pump_libevent.cc:222:5 (content_browsertests+0x0000028b5a48)
    #35 base::MessageLoop::RunHandler() base/message_loop/message_loop.cc:378:10 (content_browsertests+0x0000028af88b)
    #36 base::RunLoop::Run() base/run_loop.cc:35:10 (content_browsertests+0x0000028da989)
    #37 base::Thread::Run(base::RunLoop*) base/threading/thread.cc:245:13 (content_browsertests+0x0000029079d9)
    #38 content::BrowserThreadImpl::IOThreadRun(base::RunLoop*) content/browser/browser_thread_impl.cc:253:11 (content_browsertests+0x000001de4d9f)
    #39 content::BrowserThreadImpl::Run(base::RunLoop*) content/browser/browser_thread_impl.cc:288:14 (content_browsertests+0x000001de4fc2)
    #40 base::Thread::ThreadMain() base/threading/thread.cc:333:3 (content_browsertests+0x000002907c34)
    #41 base::(anonymous namespace)::ThreadFunc(void*) base/threading/platform_thread_posix.cc:71:13 (content_browsertests+0x00000290101d)

  Mutex M2764 previously acquired by the same thread here:
    #0 pthread_mutex_lock <null> (content_browsertests+0x0000004cc4de)
    #1 base::internal::LockImpl::Lock() base/synchronization/lock_impl_posix.cc:64:12 (content_browsertests+0x0000028f07a5)
    #2 Acquire base/synchronization/lock.h:26:26 (content_browsertests+0x000002547cf1)
    #3 AutoLock base/synchronization/lock.h:115 (content_browsertests+0x000002547cf1)
    #4 mojo::edk::Watcher::MaybeInvokeCallback(unsigned int, mojo::edk::HandleSignalsState const&, unsigned int) mojo/edk/system/watcher.cc:20 (content_browsertests+0x000002547cf1)
    #5 mojo::edk::RequestContext::~RequestContext() mojo/edk/system/request_context.cc:59:22 (content_browsertests+0x00000253f128)
    #6 mojo::edk::RequestContext::~RequestContext() mojo/edk/system/request_context.cc:60:5 (content_browsertests+0x00000253f133)
    #7 mojo::edk::NodeChannel::OnChannelMessage(void const*, unsigned long, std::__1::unique_ptr<std::__1::vector<mojo::edk::PlatformHandle, std::__1::allocator<mojo::edk::PlatformHandle> >, mojo::edk::PlatformHandleVectorDeleter>) mojo/edk/system/node_channel.cc:766:1 (content_browsertests+0x000002552e30)
    #8 mojo::edk::Channel::OnReadComplete(unsigned long, unsigned long*) mojo/edk/system/channel.cc:563:18 (content_browsertests+0x00000254b9eb)
    #9 mojo::edk::(anonymous namespace)::ChannelPosix::OnFileCanReadWithoutBlocking(int) mojo/edk/system/channel_posix.cc:314:14 (content_browsertests+0x00000254de8e)
    #10 non-virtual thunk to mojo::edk::(anonymous namespace)::ChannelPosix::OnFileCanReadWithoutBlocking(int) mojo/edk/system/channel_posix.cc:274:8 (content_browsertests+0x00000254e0f4)
    #11 OnFileCanReadWithoutBlocking base/message_loop/message_pump_libevent.cc:96:13 (content_browsertests+0x0000028b5844)
    #12 base::MessagePumpLibevent::OnLibeventNotification(int, short, void*) base/message_loop/message_pump_libevent.cc:337 (content_browsertests+0x0000028b5844)
    #13 event_process_active base/third_party/libevent/event.c:381:4 (content_browsertests+0x00000295c025)
    #14 event_base_loop base/third_party/libevent/event.c:521 (content_browsertests+0x00000295c025)
    #15 base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) base/message_loop/message_pump_libevent.cc:222:5 (content_browsertests+0x0000028b5a48)
    #16 base::MessageLoop::RunHandler() base/message_loop/message_loop.cc:378:10 (content_browsertests+0x0000028af88b)
    #17 base::RunLoop::Run() base/run_loop.cc:35:10 (content_browsertests+0x0000028da989)
    #18 base::Thread::Run(base::RunLoop*) base/threading/thread.cc:245:13 (content_browsertests+0x0000029079d9)
    #19 content::BrowserThreadImpl::IOThreadRun(base::RunLoop*) content/browser/browser_thread_impl.cc:253:11 (content_browsertests+0x000001de4d9f)
    #20 content::BrowserThreadImpl::Run(base::RunLoop*) content/browser/browser_thread_impl.cc:288:14 (content_browsertests+0x000001de4fc2)
    #21 base::Thread::ThreadMain() base/threading/thread.cc:333:3 (content_browsertests+0x000002907c34)
    #22 base::(anonymous namespace)::ThreadFunc(void*) base/threading/platform_thread_posix.cc:71:13 (content_browsertests+0x00000290101d)

  Mutex M2764 acquired here while holding mutex M2768 in thread T12:
    #0 pthread_mutex_lock <null> (content_browsertests+0x0000004cc4de)
    #1 base::internal::LockImpl::Lock() base/synchronization/lock_impl_posix.cc:64:12 (content_browsertests+0x0000028f07a5)
    #2 Acquire base/synchronization/lock.h:26:26 (content_browsertests+0x000002547cf1)
    #3 AutoLock base/synchronization/lock.h:115 (content_browsertests+0x000002547cf1)
    #4 mojo::edk::Watcher::MaybeInvokeCallback(unsigned int, mojo::edk::HandleSignalsState const&, unsigned int) mojo/edk/system/watcher.cc:20 (content_browsertests+0x000002547cf1)
    #5 mojo::edk::RequestContext::~RequestContext() mojo/edk/system/request_context.cc:59:22 (content_browsertests+0x00000253f128)
    #6 mojo::edk::Core::WriteMessageNew(unsigned int, unsigned long, unsigned int) mojo/edk/system/core.cc:702:1 (content_browsertests+0x00000251dde6)
    #7 MojoWriteMessageNewImpl mojo/edk/embedder/entrypoints.cc:120:18 (content_browsertests+0x000000df173c)
    #8 MojoWriteMessageNew mojo/public/c/system/thunks.cc:210:10 (content_browsertests+0x000002b75674)
    #9 WriteMessageNew mojo/public/cpp/system/message_pipe.h:97:10 (content_browsertests+0x000002556ab7)
    #10 mojo::Connector::Accept(mojo::Message*) mojo/public/cpp/bindings/lib/connector.cc:127 (content_browsertests+0x000002556ab7)
    #11 mojo::(anonymous namespace)::SendRunOrClosePipeMessage(mojo::MessageReceiver*, mojo::StructPtr<mojo::pipe_control::RunOrClosePipeInput>, mojo::internal::SerializationContext*) mojo/public/cpp/bindings/lib/pipe_control_message_proxy.cc:36:23 (content_browsertests+0x00000256672a)
    #12 mojo::PipeControlMessageProxy::NotifyPeerEndpointClosed(unsigned int) mojo/public/cpp/bindings/lib/pipe_control_message_proxy.cc:57:3 (content_browsertests+0x000002566515)
    #13 mojo::internal::MultiplexRouter::CloseEndpointHandle(unsigned int, bool) mojo/public/cpp/bindings/lib/multiplex_router.cc:417:28 (content_browsertests+0x00000255e822)
    #14 non-virtual thunk to mojo::internal::MultiplexRouter::CloseEndpointHandle(unsigned int, bool) mojo/public/cpp/bindings/lib/multiplex_router.cc:394:23 (content_browsertests+0x00000255ef3e)
    #15 reset mojo/public/cpp/bindings/lib/scoped_interface_endpoint_handle.cc:38:22 (content_browsertests+0x000002566cab)
    #16 mojo::ScopedInterfaceEndpointHandle::~ScopedInterfaceEndpointHandle() mojo/public/cpp/bindings/lib/scoped_interface_endpoint_handle.cc:23 (content_browsertests+0x000002566cab)
    #17 mojo::InterfaceEndpointClient::~InterfaceEndpointClient() mojo/public/cpp/bindings/lib/interface_endpoint_client.cc:173:1 (content_browsertests+0x0000025595fd)
    #18 mojo::InterfaceEndpointClient::~InterfaceEndpointClient() mojo/public/cpp/bindings/lib/interface_endpoint_client.cc:166:53 (content_browsertests+0x000002559669)
    #19 operator() buildtools/third_party/libc++/trunk/include/memory:2529:13 (content_browsertests+0x0000039e67f4)
    #20 reset buildtools/third_party/libc++/trunk/include/memory:2735 (content_browsertests+0x0000039e67f4)
    #21 ~AssociatedInterfacePtrState mojo/public/cpp/bindings/lib/associated_interface_ptr_state.h:41 (content_browsertests+0x0000039e67f4)
    #22 ~AssociatedInterfacePtr mojo/public/cpp/bindings/associated_interface_ptr.h:55 (content_browsertests+0x0000039e67f4)
    #23 service_manager::ServiceContext::~ServiceContext() services/service_manager/public/cpp/lib/service_context.cc:45 (content_browsertests+0x0000039e67f4)
    #24 service_manager::ServiceContext::~ServiceContext() services/service_manager/public/cpp/lib/service_context.cc:45:35 (content_browsertests+0x0000039e6979)
    #25 operator() buildtools/third_party/libc++/trunk/include/memory:2529:13 (content_browsertests+0x000000afcb96)
    #26 reset buildtools/third_party/libc++/trunk/include/memory:2735 (content_browsertests+0x000000afcb96)
    #27 content::ServiceManagerConnectionImpl::IOThreadContext::ShutDownOnIOThread() content/common/service_manager/service_manager_connection_impl.cc:229 (content_browsertests+0x000000afcb96)
    #28 Invoke<const scoped_refptr<content::ServiceManagerConnectionImpl::IOThreadContext> &> base/bind_internal.h:214:12 (content_browsertests+0x000000afbc26)
    #29 MakeItSo<void (content::ServiceManagerConnectionImpl::IOThreadContext::*const &)(), const scoped_refptr<content::ServiceManagerConnectionImpl::IOThreadContext> &> base/bind_internal.h:285 (content_browsertests+0x000000afbc26)
    #30 RunImpl<void (content::ServiceManagerConnectionImpl::IOThreadContext::*const &)(), const std::__1::tuple<scoped_refptr<content::ServiceManagerConnectionImpl::IOThreadContext> > &, 0> base/bind_internal.h:361 (content_browsertests+0x000000afbc26)
    #31 base::internal::Invoker<base::internal::BindState<void (content::ServiceManagerConnectionImpl::IOThreadContext::*)(), scoped_refptr<content::ServiceManagerConnectionImpl::IOThreadContext> >, void ()>::Run(base::internal::BindStateBase*) base/bind_internal.h:339 (content_browsertests+0x000000afbc26)
    #32 Run base/callback.h:47:12 (content_browsertests+0x000002949eaa)
    #33 base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) base/debug/task_annotator.cc:52 (content_browsertests+0x000002949eaa)
    #34 base::MessageLoop::RunTask(base::PendingTask*) base/message_loop/message_loop.cc:413:19 (content_browsertests+0x0000028afca9)
    #35 base::MessageLoop::DeferOrRunPendingTask(base::PendingTask) base/message_loop/message_loop.cc:422:5 (content_browsertests+0x0000028b01ad)
    #36 base::MessageLoop::DoWork() base/message_loop/message_loop.cc:515:13 (content_browsertests+0x0000028b08c4)
    #37 base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) base/message_loop/message_pump_libevent.cc:218:31 (content_browsertests+0x0000028b5c10)
    #38 base::MessageLoop::RunHandler() base/message_loop/message_loop.cc:378:10 (content_browsertests+0x0000028af88b)
    #39 base::RunLoop::Run() base/run_loop.cc:35:10 (content_browsertests+0x0000028da989)
    #40 base::Thread::Run(base::RunLoop*) base/threading/thread.cc:245:13 (content_browsertests+0x0000029079d9)
    #41 content::BrowserThreadImpl::IOThreadRun(base::RunLoop*) content/browser/browser_thread_impl.cc:253:11 (content_browsertests+0x000001de4d9f)
    #42 content::BrowserThreadImpl::Run(base::RunLoop*) content/browser/browser_thread_impl.cc:288:14 (content_browsertests+0x000001de4fc2)
    #43 base::Thread::ThreadMain() base/threading/thread.cc:333:3 (content_browsertests+0x000002907c34)
    #44 base::(anonymous namespace)::ThreadFunc(void*) base/threading/platform_thread_posix.cc:71:13 (content_browsertests+0x00000290101d)

  Mutex M2768 previously acquired by the same thread here:
    #0 pthread_mutex_lock <null> (content_browsertests+0x0000004cc4de)
    #1 base::internal::LockImpl::Lock() base/synchronization/lock_impl_posix.cc:64:12 (content_browsertests+0x0000028f07a5)
    #2 Acquire base/synchronization/lock.h:26:26 (content_browsertests+0x00000255e706)
    #3 MayAutoLock mojo/public/cpp/bindings/lib/may_auto_lock.h:17 (content_browsertests+0x00000255e706)
    #4 mojo::internal::MultiplexRouter::CloseEndpointHandle(unsigned int, bool) mojo/public/cpp/bindings/lib/multiplex_router.cc:398 (content_browsertests+0x00000255e706)
    #5 non-virtual thunk to mojo::internal::MultiplexRouter::CloseEndpointHandle(unsigned int, bool) mojo/public/cpp/bindings/lib/multiplex_router.cc:394:23 (content_browsertests+0x00000255ef3e)
    #6 reset mojo/public/cpp/bindings/lib/scoped_interface_endpoint_handle.cc:38:22 (content_browsertests+0x000002566cab)
    #7 mojo::ScopedInterfaceEndpointHandle::~ScopedInterfaceEndpointHandle() mojo/public/cpp/bindings/lib/scoped_interface_endpoint_handle.cc:23 (content_browsertests+0x000002566cab)
    #8 mojo::InterfaceEndpointClient::~InterfaceEndpointClient() mojo/public/cpp/bindings/lib/interface_endpoint_client.cc:173:1 (content_browsertests+0x0000025595fd)
    #9 mojo::InterfaceEndpointClient::~InterfaceEndpointClient() mojo/public/cpp/bindings/lib/interface_endpoint_client.cc:166:53 (content_browsertests+0x000002559669)
    #10 operator() buildtools/third_party/libc++/trunk/include/memory:2529:13 (content_browsertests+0x0000039e67f4)
    #11 reset buildtools/third_party/libc++/trunk/include/memory:2735 (content_browsertests+0x0000039e67f4)
    #12 ~AssociatedInterfacePtrState mojo/public/cpp/bindings/lib/associated_interface_ptr_state.h:41 (content_browsertests+0x0000039e67f4)
    #13 ~AssociatedInterfacePtr mojo/public/cpp/bindings/associated_interface_ptr.h:55 (content_browsertests+0x0000039e67f4)
    #14 service_manager::ServiceContext::~ServiceContext() services/service_manager/public/cpp/lib/service_context.cc:45 (content_browsertests+0x0000039e67f4)
    #15 service_manager::ServiceContext::~ServiceContext() services/service_manager/public/cpp/lib/service_context.cc:45:35 (content_browsertests+0x0000039e6979)
    #16 operator() buildtools/third_party/libc++/trunk/include/memory:2529:13 (content_browsertests+0x000000afcb96)
    #17 reset buildtools/third_party/libc++/trunk/include/memory:2735 (content_browsertests+0x000000afcb96)
    #18 content::ServiceManagerConnectionImpl::IOThreadContext::ShutDownOnIOThread() content/common/service_manager/service_manager_connection_impl.cc:229 (content_browsertests+0x000000afcb96)
    #19 Invoke<const scoped_refptr<content::ServiceManagerConnectionImpl::IOThreadContext> &> base/bind_internal.h:214:12 (content_browsertests+0x000000afbc26)
    #20 MakeItSo<void (content::ServiceManagerConnectionImpl::IOThreadContext::*const &)(), const scoped_refptr<content::ServiceManagerConnectionImpl::IOThreadContext> &> base/bind_internal.h:285 (content_browsertests+0x000000afbc26)
    #21 RunImpl<void (content::ServiceManagerConnectionImpl::IOThreadContext::*const &)(), const std::__1::tuple<scoped_refptr<content::ServiceManagerConnectionImpl::IOThreadContext> > &, 0> base/bind_internal.h:361 (content_browsertests+0x000000afbc26)
    #22 base::internal::Invoker<base::internal::BindState<void (content::ServiceManagerConnectionImpl::IOThreadContext::*)(), scoped_refptr<content::ServiceManagerConnectionImpl::IOThreadContext> >, void ()>::Run(base::internal::BindStateBase*) base/bind_internal.h:339 (content_browsertests+0x000000afbc26)
    #23 Run base/callback.h:47:12 (content_browsertests+0x000002949eaa)
    #24 base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) base/debug/task_annotator.cc:52 (content_browsertests+0x000002949eaa)
    #25 base::MessageLoop::RunTask(base::PendingTask*) base/message_loop/message_loop.cc:413:19 (content_browsertests+0x0000028afca9)
    #26 base::MessageLoop::DeferOrRunPendingTask(base::PendingTask) base/message_loop/message_loop.cc:422:5 (content_browsertests+0x0000028b01ad)
    #27 base::MessageLoop::DoWork() base/message_loop/message_loop.cc:515:13 (content_browsertests+0x0000028b08c4)
    #28 base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) base/message_loop/message_pump_libevent.cc:218:31 (content_browsertests+0x0000028b5c10)
    #29 base::MessageLoop::RunHandler() base/message_loop/message_loop.cc:378:10 (content_browsertests+0x0000028af88b)
    #30 base::RunLoop::Run() base/run_loop.cc:35:10 (content_browsertests+0x0000028da989)
    #31 base::Thread::Run(base::RunLoop*) base/threading/thread.cc:245:13 (content_browsertests+0x0000029079d9)
    #32 content::BrowserThreadImpl::IOThreadRun(base::RunLoop*) content/browser/browser_thread_impl.cc:253:11 (content_browsertests+0x000001de4d9f)
    #33 content::BrowserThreadImpl::Run(base::RunLoop*) content/browser/browser_thread_impl.cc:288:14 (content_browsertests+0x000001de4fc2)
    #34 base::Thread::ThreadMain() base/threading/thread.cc:333:3 (content_browsertests+0x000002907c34)
    #35 base::(anonymous namespace)::ThreadFunc(void*) base/threading/platform_thread_posix.cc:71:13 (content_browsertests+0x00000290101d)

  Thread T12 'Chrome_IOThread' (tid=7965, running) created by main thread at:
    #0 pthread_create <null> (content_browsertests+0x0000004b3495)
    #1 base::(anonymous namespace)::CreateThread(unsigned long, bool, base::PlatformThread::Delegate*, base::PlatformThreadHandle*, base::ThreadPriority) base/threading/platform_thread_posix.cc:110:13 (content_browsertests+0x000002900ce7)
    #2 base::PlatformThread::CreateWithPriority(unsigned long, base::PlatformThread::Delegate*, base::PlatformThreadHandle*, base::ThreadPriority) base/threading/platform_thread_posix.cc:191:10 (content_browsertests+0x000002900be5)
    #3 base::Thread::StartWithOptions(base::Thread::Options const&) base/threading/thread.cc:112:15 (content_browsertests+0x000002907537)
    #4 content::BrowserThreadImpl::StartWithOptions(base::Thread::Options const&) content/browser/browser_thread_impl.cc:359:25 (content_browsertests+0x000001de54e4)
    #5 content::BrowserMainLoop::CreateThreads() content/browser/browser_main_loop.cc:942:32 (content_browsertests+0x000001dd287a)
    #6 Invoke<content::BrowserMainLoop *> base/bind_internal.h:214:12 (content_browsertests+0x000001dd7275)
    #7 MakeItSo<int (content::BrowserMainLoop::*const &)(), content::BrowserMainLoop *> base/bind_internal.h:285 (content_browsertests+0x000001dd7275)
    #8 RunImpl<int (content::BrowserMainLoop::*const &)(), const std::__1::tuple<base::internal::UnretainedWrapper<content::BrowserMainLoop> > &, 0> base/bind_internal.h:361 (content_browsertests+0x000001dd7275)
    #9 base::internal::Invoker<base::internal::BindState<int (content::BrowserMainLoop::*)(), base::internal::UnretainedWrapper<content::BrowserMainLoop> >, int ()>::Run(base::internal::BindStateBase*) base/bind_internal.h:339 (content_browsertests+0x000001dd7275)
    #10 Run base/callback.h:64:12 (content_browsertests+0x00000225b49a)
    #11 content::StartupTaskRunner::RunAllTasksNow() content/browser/startup_task_runner.cc:45 (content_browsertests+0x00000225b49a)
    #12 content::BrowserMainLoop::CreateStartupTasks() content/browser/browser_main_loop.cc:850:25 (content_browsertests+0x000001dd1d88)
    #13 content::BrowserMainRunnerImpl::Initialize(content::MainFunctionParams const&) content/browser/browser_main_runner.cc:140:17 (content_browsertests+0x000001dd7b42)
    #14 ShellBrowserMain(content::MainFunctionParams const&, std::__1::unique_ptr<content::BrowserMainRunner, std::__1::default_delete<content::BrowserMainRunner> > const&) content/shell/browser/shell_browser_main.cc:23:32 (content_browsertests+0x000002492337)
    #15 content::ShellMainDelegate::RunProcess(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, content::MainFunctionParams const&) content/shell/app/shell_main_delegate.cc:293:16 (content_browsertests+0x000002485633)
    #16 content::RunNamedProcessTypeMain(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, content::MainFunctionParams const&, content::ContentMainDelegate*) content/app/content_main_runner.cc:395:35 (content_browsertests+0x000001cf4d3b)
    #17 content::ContentMainRunnerImpl::Run() content/app/content_main_runner.cc:776:12 (content_browsertests+0x000001cf5a27)
    #18 content::ContentMain(content::ContentMainParams const&) content/app/content_main.cc:20:28 (content_browsertests+0x000001cecb7e)
    #19 content::BrowserTestBase::SetUp() content/public/test/browser_test_base.cc:308:3 (content_browsertests+0x000002434dfb)
    #20 content::ContentBrowserTest::SetUp() content/public/test/content_browser_test.cc:94:20 (content_browsertests+0x00000242ea36)
    #21 content::BackgroundSyncBrowserTest::SetUp() content/browser/background_sync/background_sync_browsertest.cc:131:25 (content_browsertests+0x000000550ef3)
    #22 HandleExceptionsInMethodIfSupported<testing::Test, void> testing/gtest/src/gtest.cc:2458:12 (content_browsertests+0x00000263e70d)
    #23 testing::Test::Run() testing/gtest/src/gtest.cc:2470 (content_browsertests+0x00000263e70d)
    #24 testing::TestInfo::Run() testing/gtest/src/gtest.cc:2656:11 (content_browsertests+0x00000263f89d)
    #25 testing::TestCase::Run() testing/gtest/src/gtest.cc:2774:28 (content_browsertests+0x000002640176)
    #26 testing::internal::UnitTestImpl::RunAllTests() testing/gtest/src/gtest.cc:4647:43 (content_browsertests+0x000002649656)
    #27 HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> testing/gtest/src/gtest.cc:2458:12 (content_browsertests+0x000002648ff9)
    #28 testing::UnitTest::Run() testing/gtest/src/gtest.cc:4255 (content_browsertests+0x000002648ff9)
    #29 RUN_ALL_TESTS testing/gtest/include/gtest/gtest.h:2237:46 (content_browsertests+0x0000024619c9)
    #30 base::TestSuite::Run() base/test/test_suite.cc:246 (content_browsertests+0x0000024619c9)
    #31 content::ContentTestLauncherDelegate::RunTestSuite(int, char**) content/test/content_test_launcher.cc:105:48 (content_browsertests+0x0000024340bb)
    #32 content::LaunchTests(content::TestLauncherDelegate*, int, int, char**) content/public/test/test_launcher.cc:520:31 (content_browsertests+0x00000244e9d4)
    #33 main content/test/content_test_launcher.cc:131:10 (content_browsertests+0x000002434042)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) (/b/swarm_slave/w/ir_q6tss/out/Release/content_browsertests+0x4cc4de) in __interceptor_pthread_mutex_lock
Project Member

Comment 1 by bugdroid1@chromium.org, Nov 9 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1a665b53fe46aba0c80416b9b354760fc1ccf749

commit 1a665b53fe46aba0c80416b9b354760fc1ccf749
Author: benwells <benwells@chromium.org>
Date: Wed Nov 09 05:04:42 2016

Revert of Service Manager: Implement graceful service termination (patchset #1 id:1 of https://codereview.chromium.org/2480603004/ )

Reason for revert:
Seems to be the cause of lock order inversions on TSan bots.

See  https://crbug.com/663557  for details.

BUG= 663557 

Original issue's description:
> Service Manager: Implement graceful service termination
>
> Adds a new ServiceControl client interface associated with
> a service's Service interface. This allows a service to notify
> the Service Manager that it's ready to be terminated by calling
> RequestQuit().
>
> Adds an ack reply to Service.OnConnect. The Service Manager will
> ignore any RequestQuit() issued while an OnConnect is still
> pending acknowledgement.
>
> Fixes a race in existing service_manager_unittests which was
> causing
> some tests to hang fairly regularly, using RequestQuit()
> to avoid races between service shutdown and incoming connections.
>
> Also fixes incorrect base::SimpleThread usage in some tests.
>
> BUG=654986, 662177 
>
> Committed: https://crrev.com/5b9cb3df6c06015d79aeb688c2b8905016086b9e
> Cr-Commit-Position: refs/heads/master@{#430414}

TBR=ben@chromium.org,tsepez@chromium.org,blundell@chromium.org,rockot@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG=654986, 662177 

Review-Url: https://codereview.chromium.org/2486073003
Cr-Commit-Position: refs/heads/master@{#430868}

[modify] https://crrev.com/1a665b53fe46aba0c80416b9b354760fc1ccf749/chrome/test/base/mojo_test_connector.cc
[modify] https://crrev.com/1a665b53fe46aba0c80416b9b354760fc1ccf749/services/service_manager/public/cpp/lib/service_context.cc
[modify] https://crrev.com/1a665b53fe46aba0c80416b9b354760fc1ccf749/services/service_manager/public/cpp/service_context.h
[modify] https://crrev.com/1a665b53fe46aba0c80416b9b354760fc1ccf749/services/service_manager/public/interfaces/BUILD.gn
[modify] https://crrev.com/1a665b53fe46aba0c80416b9b354760fc1ccf749/services/service_manager/public/interfaces/service.mojom
[delete] https://crrev.com/3f4b9da10b124cf841a5691abd4ede58f768be42/services/service_manager/public/interfaces/service_control.mojom
[modify] https://crrev.com/1a665b53fe46aba0c80416b9b354760fc1ccf749/services/service_manager/service_manager.cc
[modify] https://crrev.com/1a665b53fe46aba0c80416b9b354760fc1ccf749/services/service_manager/tests/connect/connect_test_package.cc
[modify] https://crrev.com/1a665b53fe46aba0c80416b9b354760fc1ccf749/services/service_manager/tests/lifecycle/app_client.cc
[modify] https://crrev.com/1a665b53fe46aba0c80416b9b354760fc1ccf749/services/service_manager/tests/lifecycle/app_client.h

Comment 2 by roc...@chromium.org, Nov 9 2016

Cc: roc...@chromium.org joedow@chromium.org dcheng@chromium.org benwells@chromium.org
 Issue 663854  has been merged into this issue.

Comment 3 by roc...@chromium.org, Nov 9 2016 

This is caused by multiplexing routers (both mojo::MultiplexRouter and IPC::ChannelAssociatedGroupController) grabbing a lock during message dispatch which they also hold while writing the endpoint-closed control message.

If both endpoints of an associated interface live in the same process, there's a LOI there.
Project Member

Comment 4 by bugdroid1@chromium.org, Nov 10 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/74b37c31ba3657034f634e46797741a80a2c3f32

commit 74b37c31ba3657034f634e46797741a80a2c3f32
Author: benwells <benwells@chromium.org>
Date: Thu Nov 10 06:46:23 2016

Add suppression for lock order inversion in Mojo.

This inversion should be removed if the potential deadlock is fised.

TBR=glider@chromium.org
BUG= 663557 

Review-Url: https://codereview.chromium.org/2496473002
Cr-Commit-Position: refs/heads/master@{#431197}

[modify] https://crrev.com/74b37c31ba3657034f634e46797741a80a2c3f32/build/sanitizers/tsan_suppressions.cc
Project Member

Comment 5 by bugdroid1@chromium.org, Nov 10 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/58909542197b704efe10526c169e4b502f799b57

commit 58909542197b704efe10526c169e4b502f799b57
Author: rockot <rockot@chromium.org>
Date: Thu Nov 10 20:05:45 2016

Mojo Bindings: Fix lock-order inversion in associated controllers

The two AssociatedGroupController implementations we have both own a lock
which is acquired during message dispatch, among other operations.
In the EDK layer a Watcher lock is also acquired further up the stack.

Because sending a message may indirectly require notifying the same
Watcher lock, it must never be true that the AssociatedGroupController's
lock is held while its pipe is written to.

This fixes the lock-order inversion resulting from the
fact that pipe control messages were being sent under lock, and removes
the associated TSAN suppression.

BUG= 663557 
TBR=glider@chromium.org

R=yzshen@chromium.org

Review-Url: https://codereview.chromium.org/2494483003
Cr-Commit-Position: refs/heads/master@{#431331}

[modify] https://crrev.com/58909542197b704efe10526c169e4b502f799b57/build/sanitizers/tsan_suppressions.cc
[modify] https://crrev.com/58909542197b704efe10526c169e4b502f799b57/ipc/ipc_mojo_bootstrap.cc
[modify] https://crrev.com/58909542197b704efe10526c169e4b502f799b57/mojo/public/cpp/bindings/lib/multiplex_router.cc
[modify] https://crrev.com/58909542197b704efe10526c169e4b502f799b57/mojo/public/cpp/bindings/lib/multiplex_router.h
[modify] https://crrev.com/58909542197b704efe10526c169e4b502f799b57/mojo/public/cpp/bindings/lib/pipe_control_message_proxy.cc
[modify] https://crrev.com/58909542197b704efe10526c169e4b502f799b57/mojo/public/cpp/bindings/pipe_control_message_proxy.h

Comment 6 by roc...@chromium.org, Nov 10 2016

Status: Fixed (was: Assigned)
►

Sign in to add a comment