New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 759205 link

Starred by 3 users

Issue metadata

Status: WontFix
Owner:
please use my google.com address
Closed: Sep 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 1
Type: Bug-Security



Sign in to add a comment

multithreaded use-after-free in mojo

Reported by jeffwalt...@gmail.com, Aug 25 2017

Issue description

UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36

Steps to reproduce the problem:
We ran a dynamic tool for detecting multithreaded use-after-free bugs in Chromium trunk (commit a0b867b844d0e05d5b5699016468870581a12ad4), the tool reported several use-after-frees. 

What is the expected behavior?

What went wrong?

------- #8 free call stack  
  base/synchronization/lock_impl.h:70
    buildtools/third_party/libc++/trunk/include/memory:2570
      mojo/edk/system/ports/node.cc:671
        mojo/edk/system/ports/node.cc:387
          buildtools/third_party/libc++/trunk/include/memory:2582
                              mojo/edk/system/channel_posix.cc:320
                                  base/message_loop/message_pump_libevent.cc:97
                                    base/third_party/libevent/event.c:381
                                      base/message_loop/message_pump_libevent.cc:224

------- #0 use call stack  
buildtools/third_party/libc++/trunk/include/__hash_table:2309
  mojo/edk/system/ports/node.cc:171
    mojo/edk/system/ports/node.cc:211
      mojo/edk/system/core.cc:381
        mojo/edk/embedder/incoming_broker_client_invitation.cc:41
          mojo/public/cpp/system/handle.h:157
            content/child/child_thread_impl.cc:392
              content/gpu/gpu_child_thread.cc:178
                buildtools/third_party/libc++/trunk/include/memory:2582
                  buildtools/third_party/libc++/trunk/include/vector:444

------- #0 use call stack  
  base/atomic_ref_count.h:37
    mojo/edk/system/core.cc:385
      mojo/edk/embedder/incoming_broker_client_invitation.cc:41
        mojo/public/cpp/system/handle.h:157
          content/child/child_thread_impl.cc:392
            content/gpu/gpu_child_thread.cc:178
              buildtools/third_party/libc++/trunk/include/memory:2582
                buildtools/third_party/libc++/trunk/include/vector:444

------- #8 free call stack  
 base/bind_internal.h:484
   base/callback_internal.cc:82
     base/callback.h:92
       buildtools/third_party/libc++/trunk/include/vector:644
         base/message_loop/message_loop.cc:528
           base/message_loop/message_pump_libevent.cc:220
              crtstuff.c:?
               base/run_loop.cc:124
                 base/threading/thread.cc:256
                   base/synchronization/lock.h:26
                     base/threading/platform_thread_posix.cc:77

------- #0 use call stack  
/archive/jeff/git/chromium/src/out/tsan3/../../mojo/edk/embedder/scoped_platform_handle.h:29
 buildtools/third_party/libc++/trunk/include/tuple:226
   mojo/edk/embedder/scoped_platform_handle.h:21
         content/child/child_thread_impl.cc:392
           content/gpu/gpu_child_thread.cc:178
             buildtools/third_party/libc++/trunk/include/memory:2582
               buildtools/third_party/libc++/trunk/include/vector:444

Did this work before? N/A 

Chrome version: 51.0.2704.103  Channel: n/a
OS Version: OS X 10.11.5
Flash Version: Shockwave Flash 26.0 r0

We could not confirm if they are real vulnerabilities or how serious they are, but perhaps worth taking a look by Chromium developers.
 

Comment 1 by ta...@google.com, Aug 26 2017

Components: Internals>Mojo
Labels: Security_Severity-High Security_Impact-Beta Needs-Feedback
Do you happen to have a repro testcase for these crashes?
Unfortunately we do not have test cases. The dynamic tool is modified from Tsan2, it collects traces from the instrumented browser at runtime and analyzes the traces offline. We did a number of browser actions related to PDF and printing, and opened tabs for Youtube and Facebook. 
Project Member

Comment 3 by sheriffbot@chromium.org, Aug 26 2017

Cc: ta...@google.com
Labels: -Needs-Feedback
Thank you for providing more feedback. Adding requester "tanin@google.com" to the cc list and removing "Needs-Feedback" label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 4 by sheriffbot@chromium.org, Aug 26 2017

Labels: M-61
Project Member

Comment 5 by sheriffbot@chromium.org, Aug 26 2017

Labels: ReleaseBlock-Stable
This is a serious security regression. If you are not able to fix this quickly, please revert the change that introduced it.

If this doesn't affect a release branch, or has not been properly classified for severity, please update the Security_Impact or Security_Severity labels, and remove the ReleaseBlock label. To disable this altogether, apply ReleaseBlock-NA.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 6 by sheriffbot@chromium.org, Aug 26 2017

Labels: -Pri-2 Pri-1
Labels: -Security_Impact-Beta -ReleaseBlock-Stable -M-61 Security_Impact-Stable M-62
Report mentions 51, not 61 - assuming that's correct changing to Security_Impact-Stable

Comment 8 by ta...@google.com, Aug 28 2017

Cc: -ta...@google.com thestig@chromium.org jam@chromium.org
Owner: yzshen@chromium.org
Status: Assigned (was: Unconfirmed)
yzshen@, I wonder if you can help route this to the right person.

Comment 9 by yzshen@chromium.org, Aug 28 2017

Cc: roc...@chromium.org
Hi, Ken.

Do you think you have some cycles to look into this one? :)
Cc: -roc...@chromium.org yzshen@chromium.org
Owner: roc...@chromium.org
I can try to look into this more, but I have some concerns:

- This seems like potentially many different bugs which should be investigated separately.
- The logs given are difficult to understand: which uses correspond with which frees?
- We have no ability to repro these reports or even verify that they're valid.
- At least one of these looks like a UAF on a singleton object that we never delete, which is obviously impossible.

Thanks for reply Ken!

I asked mostly because I haven't got any good idea when I read the report. Didn't mean to add to your already-heavy workload, if it doesn't seem useful to you. :)
Project Member

Comment 12 by sheriffbot@chromium.org, Sep 12 2017

rockot: Uh oh! This issue still open and hasn't been updated in the last 14 days. This is a serious vulnerability, and we want to ensure that there's progress. Could you please leave an update with the current status and any potential blockers?

If you're not the right owner for this issue, could you please remove yourself as soon as possible or help us find the right one?

If the issue is fixed or you can't reproduce it, please close the bug. If you've started working on a fix, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Status: WontFix (was: Assigned)
See also:  Issue 759203 .

Without repro cases, we have a very hard time diagnosing the problem, and no way of knowing if we've really fixed it even if we managed to come up with some kind of fix.

We've had reports like this in the past, in which non-deterministic fuzzers turn up things that look like bugs but which we can't act on. If there's a way to change your tool to produce deterministic repro cases, that could be quite useful indeed.
@palmer thank you for looking into this report and for the suggestion. We will continue improving the tool to create reproducible test cases for the detected user-after-frees.  Will update here once we have more results.
Project Member

Comment 15 by sheriffbot@chromium.org, Dec 29 2017

Labels: -Restrict-View-SecurityTeam allpublic
This bug has been closed for more than 14 weeks. Removing security view restrictions.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
We updated the tool with a better offline symbolizer. Just did a test 

TSAN_OPTIONS="atexit_sleep_ms=200 flush_memory_ms=2000 $TSAN_OPTIONS" out/tsan3/base_unittests --no-sandbox  2>&1 | tee log

The tool reported a bunch of new UaFs. See 
https://github.com/parasol-aser/UFO/blob/master/ufo-predict/chrome/uaf_list.txt

Here is a sample:

------- #6 use call stack  
~scoped_refptr @ git/chromium/src/base/memory/ref_counted.h:534 :9 pc: 0x10c9107
   (inlined by) base::SequencedWorkerPool::Worker::Run() @ src/base/threading/sequenced_worker_pool.cc:613 :0 pc: 0x10f4a7d
    operator= @ git/chromium/src/base/memory/ref_counted.h:554 :18 pc: 0x10eec1d
       (inlined by) base::SimpleThread::ThreadMain() @ chromium/src/base/threading/simple_thread.cc:69 :0 pc: 0x10f0246
         (inlined by) ~basic_string @ git/chromium/src/buildtools/third_party/libc++/trunk/include/string:1909 :0 pc: 0x10eee90
          __is_long @ git/chromium/src/buildtools/third_party/libc++/trunk/include/string:1224 :22 pc: 0x10f4db7
            base::(anonymous namespace)::ThreadFunc(void*) @ src/base/threading/platform_thread_posix.cc:77 :3 pc: 0x10ecbee
              __tsan_thread_start_func @ llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:882 :0 pc: 0x4bffbe

------- #0 free call stack  
 (inlined by) ~SequencedWorkerPool @ git/chromium/src/base/threading/sequenced_worker_pool.cc:1494 :0 pc: 0x545732
   (inlined by) ~unique_ptr @ git/chromium/src/buildtools/third_party/libc++/trunk/include/memory:2539 :0 pc: 0x10c9278
     (inlined by) reset @ git/chromium/src/buildtools/third_party/libc++/trunk/include/memory:2585 :0 pc: 0x10f4b91
      operator() @ git/chromium/src/buildtools/third_party/libc++/trunk/include/memory:2272 :5 pc: 0x10eedb9
        base::DeleteHelper<base::SequencedWorkerPool>::DoDelete(void const*) @ chromium/src/base/sequenced_task_runner_helpers.h:25 :3 pc: 0x10f3f4e
          base::internal::Invoker<base::internal::BindState<void (*)(void const*), void const*>, void ()>::RunOnce(base::internal::BindStateBase*) @ git/chromium/src/base/bind_internal.h:310 :3 pc: 0x10f3f23
             (inlined by) base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) @ chromium/src/base/debug/task_annotator.cc:59 :0 pc: 0x10f0dee
              Run @ git/chromium/src/base/callback.h:92 :3 pc: 0x10f362a
                 (inlined by) base::MessageLoop::RunTask(base::PendingTask*) @ chromium/src/base/message_loop/message_loop.cc:411 :0 pc: 0x10f49c8
                   (inlined by) begin @ git/chromium/src/base/observer_list.h:133 :0 pc: 0x10b1636
                    empty @ git/chromium/src/buildtools/third_party/libc++/trunk/include/vector:644 :23 pc: 0x103e9bb
                      base::MessageLoop::DoWork() @ chromium/src/base/message_loop/message_loop.cc:528 :13 pc: 0x1076073
                        base::MessagePumpDefault::Run(base::MessagePump::Delegate*) @ git/chromium/src/base/message_loop/message_pump_default.cc:33 :31 pc: 0x1076c6d
                          non-virtual thunk to base::MessageLoop::Run() @ chromium/src/base/message_loop/message_loop.cc:0 :0 pc: 0x10791e2
                            base::RunLoop::Run() @ git/chromium/src/base/run_loop.cc:124 :13 pc: 0x1075a08
                               (inlined by) base::SequencedWorkerPoolOwner::~SequencedWorkerPoolOwner() @ src/base/test/sequenced_worker_pool_owner.cc:29 :0 pc: 0x10aeb2c
                                ~Lock @ git/chromium/src/base/synchronization/lock.h:25 :12 pc: 0x116040e
                                  base::SequencedWorkerPoolOwner::~SequencedWorkerPoolOwner() @ src/base/test/sequenced_worker_pool_owner.cc:23 :55 pc: 0x116048a
                                    base::(anonymous namespace)::SequencedWorkerPoolTest_FlushForTesting_Test::TestBody() @ git/chromium/src/base/threading/sequenced_worker_pool_unittest.cc:1032 :7 pc: 0xe02cae
                                      testing::TestInfo::Run() @ git/chromium/src/third_party/googletest/src/googletest/src/gtest.cc:0 :11 pc: 0x101602e

OK, that looks unrelated to the original bug report. It looks like a test-only leak in base_unittests. And FYI it looks like SequencedWorkerPool is being removed very soon[1] so we probably don't care much if its tests leak.

I'm curious if you still see any leaks in Mojo when running e.g. mojo_system_unittests or mojo_public_bindings_unittests.

[1] https://chromium-review.googlesource.com/c/chromium/src/+/881882
Labels: Needs-Feedback
Also two more things to note:

- some leaky tests were just fixed in mojo_system_unittests, so syncing first would be a good idea
- we are merging many mojo test suites (including mojo_system_unittests and mojo_public_bindins_unittests) into a single mojo_unittests. It hasn't landed yet, but it might by the time you read this and/or sync :)

Thanks!
Thanks. We just tried mojo_system_unittests and mojo_public_bindins_unittests (after gclient sync), no UAFs found! 

We don't see the single mojo_unittests (guess it hasn't landed yet).

Sign in to add a comment