Whoa, there's even a data race on a VPTR:

==================
WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=22605)
  Read of size 8 at 0x7d1c0000b210 by thread T12:
    #0 content::MojoShellConnectionImpl::IOThreadContext::OnConnect(shell::Identity const&, shell::InterfaceRegistry*) content/common/mojo/mojo_shell_connection_impl.cc:193:31 (content_browsertests+0x000000a8aada)
    #1 shell::ServiceContext::OnConnect(shell::Identity const&, mojo::InterfaceRequest<shell::mojom::InterfaceProvider>, shell::CapabilityRequest const&) services/shell/public/cpp/lib/service_context.cc:73:18 (content_browsertests+0x0000040d4a0c)
    #2 shell::mojom::ServiceStub::Accept(mojo::Message*) out/Release/gen/services/shell/public/interfaces/service.mojom.cc:637:14 (content_browsertests+0x000000b3b73a)
    #3 mojo::internal::Router::HandleMessageInternal(mojo::Message*) mojo/public/cpp/bindings/lib/router.cc:291:32 (content_browsertests+0x0000028b5e0e)
    #4 mojo::internal::Router::HandleIncomingMessage(mojo::Message*) mojo/public/cpp/bindings/lib/router.cc:221:10 (content_browsertests+0x0000028b499f)
    #5 mojo::internal::Router::HandleIncomingMessageThunk::Accept(mojo::Message*) mojo/public/cpp/bindings/lib/router.cc:113:19 (content_browsertests+0x0000028b45ea)
    #6 shell::mojom::ServiceRequestValidator::Accept(mojo::Message*) out/Release/gen/services/shell/public/interfaces/service.mojom.cc:707:19 (content_browsertests+0x000000b3bf16)
    #7 mojo::MessageHeaderValidator::Accept(mojo::Message*) mojo/public/cpp/bindings/lib/message_header_validator.cc:93:17 (content_browsertests+0x0000028a774d)
    #8 mojo::Connector::ReadSingleMessage(unsigned int*) mojo/public/cpp/bindings/lib/connector.cc:276:51 (content_browsertests+0x00000289ff5b)
    #9 ReadAllAvailableMessages mojo/public/cpp/bindings/lib/connector.cc:302:10 (content_browsertests+0x0000028a0627)
    #10 mojo::Connector::OnHandleReadyInternal(unsigned int) mojo/public/cpp/bindings/lib/connector.cc:234 (content_browsertests+0x0000028a0627)
    #11 mojo::Connector::OnWatcherHandleReady(unsigned int) mojo/public/cpp/bindings/lib/connector.cc:214:3 (content_browsertests+0x0000028a0530)
    #12 Invoke<mojo::Connector *, unsigned int> base/bind_internal.h:214:12 (content_browsertests+0x0000028a0894)
    #13 MakeItSo<void (mojo::Connector::*const &)(unsigned int), mojo::Connector *, unsigned int> base/bind_internal.h:283 (content_browsertests+0x0000028a0894)
    #14 RunImpl<void (mojo::Connector::*const &)(unsigned int), const std::__1::tuple<base::internal::UnretainedWrapper<mojo::Connector> > &, 0> base/bind_internal.h:346 (content_browsertests+0x0000028a0894)
    #15 base::internal::Invoker<base::internal::BindState<void (mojo::Connector::*)(unsigned int), base::internal::UnretainedWrapper<mojo::Connector> >, void (unsigned int)>::Run(base::internal::BindStateBase*, unsigned int&&) base/bind_internal.h:324 (content_browsertests+0x0000028a0894)
    #16 Run base/callback.h:388:12 (content_browsertests+0x000000b42784)
    #17 mojo::Watcher::OnHandleReady(unsigned int) mojo/public/cpp/system/watcher.cc:122 (content_browsertests+0x000000b42784)
    #18 Invoke<const base::WeakPtr<mojo::Watcher> &, const unsigned int &> base/bind_internal.h:214:12 (content_browsertests+0x000000b42a36)
    #19 MakeItSo<void (mojo::Watcher::*const &)(unsigned int), const base::WeakPtr<mojo::Watcher> &, const unsigned int &> base/bind_internal.h:303 (content_browsertests+0x000000b42a36)
    #20 RunImpl<void (mojo::Watcher::*const &)(unsigned int), const std::__1::tuple<base::WeakPtr<mojo::Watcher>, unsigned int> &, 0, 1> base/bind_internal.h:346 (content_browsertests+0x000000b42a36)
    #21 base::internal::Invoker<base::internal::BindState<void (mojo::Watcher::*)(unsigned int), base::WeakPtr<mojo::Watcher>, unsigned int>, void ()>::Run(base::internal::BindStateBase*) base/bind_internal.h:324 (content_browsertests+0x000000b42a36)
    #22 Run base/callback.h:388:12 (content_browsertests+0x000002d11be9)
    #23 base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask const&) base/debug/task_annotator.cc:54 (content_browsertests+0x000002d11be9)
    #24 base::MessageLoop::RunTask(base::PendingTask const&) base/message_loop/message_loop.cc:488:19 (content_browsertests+0x000002c6227c)
    #25 base::MessageLoop::DeferOrRunPendingTask(base::PendingTask) base/message_loop/message_loop.cc:497:5 (content_browsertests+0x000002c6289d)
    #26 base::MessageLoop::DoWork() base/message_loop/message_loop.cc:621:13 (content_browsertests+0x000002c63054)
    #27 base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) base/message_loop/message_pump_libevent.cc:217:31 (content_browsertests+0x000002c68fa0)
    #28 base::MessageLoop::RunHandler() base/message_loop/message_loop.cc:451:10 (content_browsertests+0x000002c61ae0)
    #29 base::RunLoop::Run() base/run_loop.cc:35:10 (content_browsertests+0x000002c95dde)
    #30 base::Thread::Run(base::RunLoop*) base/threading/thread.cc:228:13 (content_browsertests+0x000002cc73bb)
    #31 content::BrowserThreadImpl::IOThreadRun(base::RunLoop*) content/browser/browser_thread_impl.cc:243:11 (content_browsertests+0x00000204239f)
    #32 content::BrowserThreadImpl::Run(base::RunLoop*) content/browser/browser_thread_impl.cc:278:14 (content_browsertests+0x0000020425c2)
    #33 base::Thread::ThreadMain() base/threading/thread.cc:301:3 (content_browsertests+0x000002cc78f0)
    #34 base::(anonymous namespace)::ThreadFunc(void*) base/threading/platform_thread_posix.cc:71:13 (content_browsertests+0x000002cc0168)

  Previous write of size 8 at 0x7d1c0000b210 by main thread:
    #0 operator new(unsigned long) <null> (content_browsertests+0x00000050f8f2)
    #1 content::RenderProcessHostImpl::RegisterMojoInterfaces() content/browser/renderer_host/render_process_host_impl.cc:1189:7 (content_browsertests+0x00000242879a)
    #2 content::RenderProcessHostImpl::Init() content/browser/renderer_host/render_process_host_impl.cc:821:3 (content_browsertests+0x000002425969)
    #3 InitRenderView content/browser/frame_host/render_frame_host_manager.cc:1862:40 (content_browsertests+0x0000021be2ef)
    #4 content::RenderFrameHostManager::ReinitializeRenderFrame(content::RenderFrameHostImpl*) content/browser/frame_host/render_frame_host_manager.cc:2039 (content_browsertests+0x0000021be2ef)
    #5 content::RenderFrameHostManager::Navigate(GURL const&, content::FrameNavigationEntry const&, content::NavigationEntryImpl const&, bool) content/browser/frame_host/render_frame_host_manager.cc:241:10 (content_browsertests+0x0000021bd28a)
    #6 content::NavigatorImpl::NavigateToEntry(content::FrameTreeNode*, content::FrameNavigationEntry const&, content::NavigationEntryImpl const&, content::NavigationController::ReloadType, bool, bool, bool, scoped_refptr<content::ResourceRequestBodyImpl> const&) content/browser/frame_host/navigator_impl.cc:360:44 (content_browsertests+0x00000219908a)
    #7 content::NavigatorImpl::NavigateToPendingEntry(content::FrameTreeNode*, content::FrameNavigationEntry const&, content::NavigationController::ReloadType, bool) content/browser/frame_host/navigator_impl.cc:448:10 (content_browsertests+0x000002199cc5)
    #8 content::NavigationControllerImpl::NavigateToPendingEntryInternal(content::NavigationController::ReloadType) content/browser/frame_host/navigation_controller_impl.cc:1876:36 (content_browsertests+0x000002184ab6)
    #9 content::NavigationControllerImpl::NavigateToPendingEntry(content::NavigationController::ReloadType) content/browser/frame_host/navigation_controller_impl.cc:1818:18 (content_browsertests+0x00000217dd5b)
    #10 LoadEntry content/browser/frame_host/navigation_controller_impl.cc:448:3 (content_browsertests+0x00000217fb81)
    #11 content::NavigationControllerImpl::LoadURLWithParams(content::NavigationController::LoadURLParams const&) content/browser/frame_host/navigation_controller_impl.cc:778 (content_browsertests+0x00000217fb81)
    #12 LoadURLForFrame content/shell/browser/shell.cc:199:34 (content_browsertests+0x0000028188ee)
    #13 content::Shell::LoadURL(GURL const&) content/shell/browser/shell.cc:191 (content_browsertests+0x0000028188ee)
    #14 NavigateToURLBlockUntilNavigationsComplete content/public/test/content_browser_test_utils.cc:47:11 (content_browsertests+0x0000027b2e10)
    #15 content::NavigateToURL(content::Shell*, GURL const&) content/public/test/content_browser_test_utils.cc:84 (content_browsertests+0x0000027b2e10)
    #16 content::MediaBrowserTest::RunTest(GURL const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) content/browser/media/media_browsertest.cc:57:3 (content_browsertests+0x00000070fb7a)
    #17 content::MediaBrowserTest::RunMediaTestPage(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool) content/browser/media/media_browsertest.cc:47:29 (content_browsertests+0x00000070f883)
    #18 content::MediaTest::PlayMedia(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool) content/browser/media/media_browsertest.cc:98:5 (content_browsertests+0x000000713f0e)
    #19 PlayVideo content/browser/media/media_browsertest.cc:82:5 (content_browsertests+0x000000710194)
    #20 RunTestOnMainThread content/browser/media/media_browsertest.cc:117 (content_browsertests+0x000000710194)
    #21 non-virtual thunk to content::MediaTest_VideoBearSilentTheora_Test::RunTestOnMainThread() content/browser/media/media_browsertest.cc:116 (content_browsertests+0x000000710194)
    #22 content::ContentBrowserTest::RunTestOnMainThreadLoop() content/public/test/content_browser_test.cc:136:3 (content_browsertests+0x0000027b27d0)
    #23 content::BrowserTestBase::ProxyRunTestOnMainThreadLoop() content/public/test/browser_test_base.cc:334:3 (content_browsertests+0x0000027b8964)
    #24 Invoke<content::BrowserTestBase *> base/bind_internal.h:214:12 (content_browsertests+0x0000027b9565)
    #25 MakeItSo<void (content::BrowserTestBase::*const &)(), content::BrowserTestBase *> base/bind_internal.h:283 (content_browsertests+0x0000027b9565)
    #26 RunImpl<void (content::BrowserTestBase::*const &)(), const std::__1::tuple<base::internal::UnretainedWrapper<content::BrowserTestBase> > &, 0> base/bind_internal.h:346 (content_browsertests+0x0000027b9565)
    #27 base::internal::Invoker<base::internal::BindState<void (content::BrowserTestBase::*)(), base::internal::UnretainedWrapper<content::BrowserTestBase> >, void ()>::Run(base::internal::BindStateBase*) base/bind_internal.h:324 (content_browsertests+0x0000027b9565)
    #28 Run base/callback.h:388:12 (content_browsertests+0x0000028412fb)
    #29 content::ShellBrowserMainParts::PreMainMessageLoopRun() content/shell/browser/shell_browser_main_parts.cc:197 (content_browsertests+0x0000028412fb)
    #30 content::BrowserMainLoop::PreMainMessageLoopRun() content/browser/browser_main_loop.cc:942:13 (content_browsertests+0x0000020302e9)
    #31 Invoke<content::BrowserMainLoop *> base/bind_internal.h:214:12 (content_browsertests+0x000002033f65)
    #32 MakeItSo<int (content::BrowserMainLoop::*const &)(), content::BrowserMainLoop *> base/bind_internal.h:283 (content_browsertests+0x000002033f65)
    #33 RunImpl<int (content::BrowserMainLoop::*const &)(), const std::__1::tuple<base::internal::UnretainedWrapper<content::BrowserMainLoop> > &, 0> base/bind_internal.h:346 (content_browsertests+0x000002033f65)
    #34 base::internal::Invoker<base::internal::BindState<int (content::BrowserMainLoop::*)(), base::internal::UnretainedWrapper<content::BrowserMainLoop> >, int ()>::Run(base::internal::BindStateBase*) base/bind_internal.h:324 (content_browsertests+0x000002033f65)
    #35 Run base/callback.h:388:12 (content_browsertests+0x000002587cf1)
    #36 content::StartupTaskRunner::RunAllTasksNow() content/browser/startup_task_runner.cc:45 (content_browsertests+0x000002587cf1)
    #37 content::BrowserMainLoop::CreateStartupTasks() content/browser/browser_main_loop.cc:832:25 (content_browsertests+0x00000202e2da)
    #38 content::BrowserMainRunnerImpl::Initialize(content::MainFunctionParams const&) content/browser/browser_main_runner.cc:140:17 (content_browsertests+0x000002034661)
    #39 ShellBrowserMain(content::MainFunctionParams const&, std::__1::unique_ptr<content::BrowserMainRunner, std::__1::default_delete<content::BrowserMainRunner> > const&) content/shell/browser/shell_browser_main.cc:23:32 (content_browsertests+0x000002819cea)
    #40 content::ShellMainDelegate::RunProcess(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, content::MainFunctionParams const&) content/shell/app/shell_main_delegate.cc:295:16 (content_browsertests+0x00000280bb33)
    #41 content::RunNamedProcessTypeMain(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, content::MainFunctionParams const&, content::ContentMainDelegate*) content/app/content_main_runner.cc:405:35 (content_browsertests+0x000001f3cf86)
    #42 content::ContentMainRunnerImpl::Run() content/app/content_main_runner.cc:785:12 (content_browsertests+0x000001f3de5d)
    #43 content::ContentMain(content::ContentMainParams const&) content/app/content_main.cc:20:28 (content_browsertests+0x000001f33fce)
    #44 content::BrowserTestBase::SetUp() content/public/test/browser_test_base.cc:307:3 (content_browsertests+0x0000027b85e4)
    #45 content::ContentBrowserTest::SetUp() content/public/test/content_browser_test.cc:93:20 (content_browsertests+0x0000027b2656)
    #46 HandleExceptionsInMethodIfSupported<testing::Test, void> testing/gtest/src/gtest.cc:2458:12 (content_browsertests+0x0000029b2b5d)
    #47 testing::Test::Run() testing/gtest/src/gtest.cc:2470 (content_browsertests+0x0000029b2b5d)
    #48 testing::TestInfo::Run() testing/gtest/src/gtest.cc:2656:11 (content_browsertests+0x0000029b3cdd)
    #49 testing::TestCase::Run() testing/gtest/src/gtest.cc:2774:28 (content_browsertests+0x0000029b45b8)
    #50 testing::internal::UnitTestImpl::RunAllTests() testing/gtest/src/gtest.cc:4647:43 (content_browsertests+0x0000029bda82)
    #51 HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> testing/gtest/src/gtest.cc:2458:12 (content_browsertests+0x0000029bd436)
    #52 testing::UnitTest::Run() testing/gtest/src/gtest.cc:4255 (content_browsertests+0x0000029bd436)
    #53 RUN_ALL_TESTS testing/gtest/include/gtest/gtest.h:2237:46 (content_browsertests+0x0000027e9624)
    #54 base::TestSuite::Run() base/test/test_suite.cc:246 (content_browsertests+0x0000027e9624)
    #55 content::ContentTestLauncherDelegate::RunTestSuite(int, char**) content/test/content_test_launcher.cc:105:48 (content_browsertests+0x0000027b7a4b)
    #56 content::LaunchTests(content::TestLauncherDelegate*, int, int, char**) content/public/test/test_launcher.cc:517:31 (content_browsertests+0x0000027d4f8a)
    #57 main content/test/content_test_launcher.cc:131:10 (content_browsertests+0x0000027b79d2)

  Location is heap block of size 104 at 0x7d1c0000b210 allocated by main thread:
    #0 operator new(unsigned long) <null> (content_browsertests+0x00000050f8f2)
    #1 content::RenderProcessHostImpl::RegisterMojoInterfaces() content/browser/renderer_host/render_process_host_impl.cc:1189:7 (content_browsertests+0x00000242879a)
    #2 content::RenderProcessHostImpl::Init() content/browser/renderer_host/render_process_host_impl.cc:821:3 (content_browsertests+0x000002425969)
    #3 InitRenderView content/browser/frame_host/render_frame_host_manager.cc:1862:40 (content_browsertests+0x0000021be2ef)
    #4 content::RenderFrameHostManager::ReinitializeRenderFrame(content::RenderFrameHostImpl*) content/browser/frame_host/render_frame_host_manager.cc:2039 (content_browsertests+0x0000021be2ef)
    #5 content::RenderFrameHostManager::Navigate(GURL const&, content::FrameNavigationEntry const&, content::NavigationEntryImpl const&, bool) content/browser/frame_host/render_frame_host_manager.cc:241:10 (content_browsertests+0x0000021bd28a)
    #6 content::NavigatorImpl::NavigateToEntry(content::FrameTreeNode*, content::FrameNavigationEntry const&, content::NavigationEntryImpl const&, content::NavigationController::ReloadType, bool, bool, bool, scoped_refptr<content::ResourceRequestBodyImpl> const&) content/browser/frame_host/navigator_impl.cc:360:44 (content_browsertests+0x00000219908a)
    #7 content::NavigatorImpl::NavigateToPendingEntry(content::FrameTreeNode*, content::FrameNavigationEntry const&, content::NavigationController::ReloadType, bool) content/browser/frame_host/navigator_impl.cc:448:10 (content_browsertests+0x000002199cc5)
    #8 content::NavigationControllerImpl::NavigateToPendingEntryInternal(content::NavigationController::ReloadType) content/browser/frame_host/navigation_controller_impl.cc:1876:36 (content_browsertests+0x000002184ab6)
    #9 content::NavigationControllerImpl::NavigateToPendingEntry(content::NavigationController::ReloadType) content/browser/frame_host/navigation_controller_impl.cc:1818:18 (content_browsertests+0x00000217dd5b)
    #10 LoadEntry content/browser/frame_host/navigation_controller_impl.cc:448:3 (content_browsertests+0x00000217fb81)
...

I'll fix this. Access to the collection of filters must always be guarded by a lock, but there are now several places where it isn't.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2432550219bea14d12a8445ae92617e2b8d6acc0

commit 2432550219bea14d12a8445ae92617e2b8d6acc0
Author: rockot <rockot@chromium.org>
Date: Wed Aug 17 23:29:01 2016

Fix data races in MojoShellConnectionImpl

Corrects some data being improperly guarded by locks.

connection_filters_ must always be accessed under lock. This change
will lead to deadlock if any filter tries to add or remove filters
during OnConnect, but 1) nobody is doing that yet and 2) it is
currently unsafe to do this already. With guaranteed deadlock at
least it's now impossible to do by accident.

Refines the thread checking in ConnectionFilterImpl
to allow for destruction on any thread if OnConnect has never
been called. This guards against ConnectionFilterImpls being
added after shutdown is initiated. This state may not be reachable
in practice since we shouldn't bring up new RPHIs during shutdown,
but it seems reasonable to express more accurate constraints here
and the change is trivial.

Finally, this also allows the IOThreadContext's internal
MessageLoopObserver (nee "Obs"?) to be cleaned up when the
IOThreadContext is shut down, rather than leaving all of them
around until shutdown. This is worthwhile since incognito
BrowserContexts may be created and destroyed arbitrarily many
times in a normal browser session, accumulating many such
observers which would otherwise live indefinitely.

BUG= 638581 
R=ben@chromium.org

Review-Url: https://codereview.chromium.org/2245333005
Cr-Commit-Position: refs/heads/master@{#412690}

[modify] https://crrev.com/2432550219bea14d12a8445ae92617e2b8d6acc0/content/browser/renderer_host/render_process_host_impl.cc
[modify] https://crrev.com/2432550219bea14d12a8445ae92617e2b8d6acc0/content/common/mojo/mojo_shell_connection_impl.cc

Issue 638867 has been merged into this issue.

Users experienced this crash on the following builds:

Mac Canary 54.0.2831.0 -  0.88 CPM, 4 reports, 4 clients (signature content::MojoShellConnectionImpl::IOThreadContext::RemoveConnectionFilterOnIOThread)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas