based on findit results, assigning to sigbjornf@ - Could you please take a look at the issue and assign it to concerned developer if your changes are not responsible?

Suspected CLs	The result is a list of CLs that change the crashed files.

Author: sigbjornf
Project: chromium
Changelist: https://chromium.googlesource.com/chromium/src//+/7de35e892724a0307ddd6440c69a2223e0fd4755
Time: Mon Apr 25 11:13:06 2016
File ThreadState.cpp is changed in this cl (and is part of stack frame #2, "blink::ThreadState::freePersistentNode")
Minimum distance from crash line to modified line: 3. (file: ThreadState.cpp, crashed on: 1354, modified: 1351).

Suspected Project: chromium
Suspected Component: Blink>MemoryAllocator

The problem here is that RTCCertificateGeneratorRequest passes an WebRTCCertificateObserver object along to a worker thread for async cert generation, the main thread is then detached and shut down. The worker thread completes and then tries to post back to the main thread. And fails when trying to release the Persistent<> that WebRTCCertificateObserver.

I don't quite understand why GenerateCertificateOnWorkerThread() gets to destruct the WebRTCCertificateObserver it is passed, 

 https://cs.chromium.org/chromium/src/content/renderer/media/rtc_certificate_generator.cc?q=GenerateCertificateOnWorkerThread&sq=package:chromium&l=74 

but perhaps it ends up being copied via Passed()? (Cc:ing hiroshige@ , who might have insights) If it is doing so and this is considered intended, then using Persistent<> is inappropriate (CrossThreadPersistent<> would be needed), but there seems to be a bigger shutdown coordination problem here -- accessing objects in the main thread's heap after it has been detached is not going to work.

Re-assigning to hbos@ due to this touching on rtc_peer_connection.cc details.

#c2: Agree. This looks a shutdown ordering problem to me.
The worker thread is stopped in PeerConnectionDependencyFactory::WillDestroyCurrentMessageLoop, which is called after the MessageLoop stop accepting tasks and after Blink shutdown.
I think the thread should shutdown earlier.

The WebRTCCertificateObserver is not copied, it is a pointer to it that is being Passed with a unique_ptr. It will be deleted when the message is destroyed (message has been executed or loop is shutdown).
As pointed out by tzik@, because it is holding a Persistent<> of a Blink object the observer must not be destroyed after Blink shutdown. tzik@'s CL (https://codereview.chromium.org/2156483002/) I think should fix this. Good job.

I think the change looks good. But be aware of that the change your are reverting was made to fix https://bugs.chromium.org/p/chromium/issues/detail?id=434972

Can you please test that #2 is not broken by this. 

Hmm, looks like my CL may cause the same issue that  http://crbug.com/434972 .

On my quick scan, WebRTC system's |signaling_thread| and |worker_thread| have to outlive the main thread, since proxy classes defined by BEGIN_{,SIGNALING_}PROXY_MAP() depend on these threads, and since they are RefCountInterface, their lifetime can be unpredictably long.
OTOH, Blink's GCed object need to be destroyed before Blink heap is terminated, which is before the main thread shutdown.

There's two separate issues that causes the crash:
1. blink::Persistent has to be destroyed before blink::ThreadState shutdown. However, some blink::Persistent can be left in the message queue, and deleted on base::MessageLoop destruction after blink::ThreadState shutdown.
2. blink::Persistent can't be brought to the other thread than the original thread, though WebRTCCertificateObserver takes a Persistent to the WebRTC worker thread. In normal case, the instance moves back to the main thread, but if it failed to go back to the main thread, it's destroyed on the worker thread.

https://codereview.chromium.org/2164503002/ will fix (1) part, and https://codereview.chromium.org/2159123002/ will fix (2).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/edbd386e6c520cf327e530f65988587eea25d65c

commit edbd386e6c520cf327e530f65988587eea25d65c
Author: tzik <tzik@chromium.org>
Date: Mon Aug 15 15:12:12 2016

Do not delete blink::WebRTCCertificateCallback on non-main thread

blink::WebRTCCertificateCallback has a blink::Persistent, which can not
be destroyed on the original thread. However, RTC implementation brings
it to a worker thread and occasionally deletes it on that thread, in
case the main thread is already gone.

This CL replaces the deleter of the std::unique_ptr that holds
WebRTCCertificateCallback with OnTaskRunnerDeleter, which ensures the
objects is deleted on the original thread.

BUG= 627004 

Review-Url: https://codereview.chromium.org/2164503002
Cr-Commit-Position: refs/heads/master@{#411965}

[modify] https://crrev.com/edbd386e6c520cf327e530f65988587eea25d65c/base/BUILD.gn
[modify] https://crrev.com/edbd386e6c520cf327e530f65988587eea25d65c/base/base.gyp
[modify] https://crrev.com/edbd386e6c520cf327e530f65988587eea25d65c/base/sequenced_task_runner.cc
[modify] https://crrev.com/edbd386e6c520cf327e530f65988587eea25d65c/base/sequenced_task_runner.h
[add] https://crrev.com/edbd386e6c520cf327e530f65988587eea25d65c/base/sequenced_task_runner_unittest.cc
[modify] https://crrev.com/edbd386e6c520cf327e530f65988587eea25d65c/content/renderer/media/rtc_certificate_generator.cc

ClusterFuzz has detected this issue as fixed in range 408378:408381.

Detailed report: https://cluster-fuzz.appspot.com/testcase?key=4572050421448704

Fuzzer: inferno_twister
Job Type: mac_asan_content_shell
Platform Id: mac

Crash Type: UNKNOWN READ
Crash Address: 0x000000000010
Crash State:
  blink::ThreadState::freePersistentNode
  blink::WebRTCCertificateObserver::~WebRTCCertificateObserver
  base::internal::BindState<void
  
Regressed: https://cluster-fuzz.appspot.com/revisions?job=mac_asan_content_shell&range=389432:389450
Fixed: https://cluster-fuzz.appspot.com/revisions?job=mac_asan_content_shell&range=408378:408381

Minimized Testcase (0.17 Kb): https://cluster-fuzz.appspot.com/download/AMIfv97UDcVcHQbNRnCcQBEY7X5uDhv1Pf6gWVrGJFPOx4j5olLGNsA2zVCU9zidznFnlp9rEJPKMGkCZSdZFZyMrIrNb2LQ-W2bDuKKpVaNi-PpnQtNIqJ_4KLO3uRDBwtXg7E0EyoKr6ad-0TxQcVddlaSOF6JUw?testcase_id=4572050421448704

See https://dev.chromium.org/Home/chromium-security/bugs/reproducing-clusterfuzz-bugs for more information.

If you suspect that the result above is incorrect, try re-doing that job on the test case report page.

ClusterFuzz testcase is verified as fixed, closing issue.

If this is incorrect, please add ClusterFuzz-Wrong label and re-open the issue.

Removing EditIssue view restrictions from ClusterFuzz filed bugs. If you believe that this issue should still be restricted, please reapply the label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot