New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 760938 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Cc:
EstimatedDays: ----
NextAction: ----
OS: Linux , Mac
Pri: 2
Type: Bug

Blocked on:
issue 765656



Sign in to add a comment

CHECK failure: !Allocator::IsObjectResurrectionForbidden() in Vector.h

Project Member Reported by ClusterFuzz, Aug 31 2017

Issue description

Detailed report: https://clusterfuzz.com/testcase?key=6640919901372416

Fuzzer: inferno_twister
Job Type: mac_asan_chrome
Platform Id: mac

Crash Type: CHECK failure
Crash Address: 
Crash State:
  !Allocator::IsObjectResurrectionForbidden() in Vector.h
  void WTF::Vector<blink::Member<blink::Event>, 0ul, blink::HeapAllocator>::Append
  blink::MediaStream::ScheduleDispatchEvent
  
Sanitizer: address (ASAN)

Regressed: https://clusterfuzz.com/revisions?job=mac_asan_chrome&range=478940:478956

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=6640919901372416

Additional requirements: Requires HTTP

Issue filed automatically.

See https://github.com/google/clusterfuzz-tools for more information.
 
Cc: msrchandra@chromium.org
Labels: M-62 Test-Predator-Wrong
Owner: mek@chromium.org
Status: Assigned (was: Untriaged)
This issue looks similar to Bug ID -- 756248. So assigning to the same owner.

@mek -- Could you please look into the issue, kindly re-assign if this is not related to your changes.
Thank You.
Project Member

Comment 2 by ClusterFuzz, Sep 4 2017

Labels: OS-Linux

Comment 3 by mek@chromium.org, Sep 12 2017

Owner: ----
Status: Available (was: Assigned)
The change of mine being blamed in 756248 should not have caused any changes whatsoever. It only made a method callable, but didn't actually add any calls to that method.

Comment 4 by mek@chromium.org, Sep 12 2017

Owner: hbos@chromium.org
assigning to hbos@ as https://codereview.chromium.org/2902733003 seems to be the only change in the given range that actually touched the code in question.

Comment 5 by mek@chromium.org, Sep 12 2017

Status: Assigned (was: Available)

Comment 6 by hbos@chromium.org, Sep 13 2017

Cc: haraken@chromium.org
Status: Started (was: Assigned)
I'm guessing this has something to do with the GC, not caused by my CL but my CL did significantly refactor related code, but I'll take a look and see if I can repro on ToT.

Comment 7 by hbos@chromium.org, Sep 13 2017

Cc: hbos@chromium.org
Labels: -Reproducible
Owner: haraken@chromium.org
Status: Assigned (was: Started)
I am unable to repro this. I tried with both tip of tree and https://codereview.chromium.org/2902733003.

haraken@ do you know what this crash means?
Cc: keishi@chromium.org
Owner: hbos@chromium.org
The problem is that  RTCPeerConnection's pre-finalizer is doing  complex things.

A pre-finalizer is not allowed to resurrect a reference to a dead object, because the marking phase is already done. If you resurrect a reference to a dead object, the object won't be traced and thus will get collected. In this case, the problem is that Vector's push_back reallocates the backing buffer and resurrect some dead object.

In short, you should not do any complex operations in a pre-finalizer.

See the following stack:

[57277:775:0822/015352.629672:FATAL:Vector.h(1598)] Check failed: !Allocator::IsObjectResurrectionForbidden().
0   Chromium Framework                  0x000000010c8eb05e base::debug::StackTrace::StackTrace(unsigned long) + 46
1   Chromium Framework                  0x000000010c948406 logging::LogMessage::~LogMessage() + 758
2   Chromium Framework                  0x0000000118a86c9f WTF::Vector<blink::Member<blink::Event>, 0ul, blink::HeapAllocator>::ReserveCapacity(unsigned long) + 607
3   Chromium Framework                  0x0000000118a8698d void WTF::Vector<blink::Member<blink::Event>, 0ul, blink::HeapAllocator>::AppendSlowCase<blink::Event*&>(blink::Event*&&&) + 141
4   Chromium Framework                  0x000000011b50afe4 blink::MediaStream::ScheduleDispatchEvent(blink::Event*) + 644
5   Chromium Framework                  0x000000011b50e243 blink::MediaStream::RemoveTrackByComponent(blink::MediaStreamComponent*) + 1571
6   Chromium Framework                  0x000000011c4274a1 content::RemoteMediaStreamImpl::OnChanged(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::unique_ptr<content::WebRtcMediaStreamTrackAdapterMap::AdapterRef, std::__1::default_delete<content::WebRtcMediaStreamTrackAdapterMap::AdapterRef> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::unique_ptr<content::WebRtcMediaStreamTrackAdapterMap::AdapterRef, std::__1::default_delete<content::WebRtcMediaStreamTrackAdapterMap::AdapterRef> > > > >) + 497
7   Chromium Framework                  0x000000011c426f35 content::RemoteMediaStreamImpl::~RemoteMediaStreamImpl() + 485
8   Chromium Framework                  0x000000011c45ae86 std::__1::__tree<std::__1::__value_type<webrtc::MediaStreamInterface*, std::__1::unique_ptr<content::RemoteMediaStreamImpl, std::__1::default_delete<content::RemoteMediaStreamImpl> > >, std::__1::__map_value_compare<webrtc::MediaStreamInterface*, std::__1::__value_type<webrtc::MediaStreamInterface*, std::__1::unique_ptr<content::RemoteMediaStreamImpl, std::__1::default_delete<content::RemoteMediaStreamImpl> > >, std::__1::less<webrtc::MediaStreamInterface*>, true>, std::__1::allocator<std::__1::__value_type<webrtc::MediaStreamInterface*, std::__1::unique_ptr<content::RemoteMediaStreamImpl, std::__1::default_delete<content::RemoteMediaStreamImpl> > > > >::destroy(std::__1::__tree_node<std::__1::__value_type<webrtc::MediaStreamInterface*, std::__1::unique_ptr<content::RemoteMediaStreamImpl, std::__1::default_delete<content::RemoteMediaStreamImpl> > >, void*>*) + 166
9   Chromium Framework                  0x000000011c43c141 content::RTCPeerConnectionHandler::~RTCPeerConnectionHandler() + 1105
10  Chromium Framework                  0x000000011c43c96d content::RTCPeerConnectionHandler::~RTCPeerConnectionHandler() + 29
11  Chromium Framework                  0x000000011b5ffb2a blink::RTCPeerConnection::InvokePreFinalizer(void*) + 266


Comment 9 by hbos@chromium.org, Sep 13 2017

Thanks haraken. I should own this.

The peer connection being GC'd results in objects being removed and destroyed, including media streams which schedules to fire events that tracks are being removed, this causes the resurrecting objects.

Firing events doesn't make any sense after the peer connection is PC'd. A possible fix to this is to ignore events fired after InvokePreFinalizer.

Comment 10 by hbos@chromium.org, Sep 13 2017

Labels: -Pri-1 Pri-2
Actually I don't understand why this is an object resurrection... Both the media streams and media stream tracks should be kept alive. They should be referenced by the RTCPeerConnection.

The AdapterRefs have WebMediaStream and WebMediaStreamTrack which should keep the corresponding blink MediaStream and MediaStreamTracks alive.

The AdapterRef is destroyed during RTCPeerConnection::InvokePreFinalizer, not at a later stage, so during this call it doesn't make sense for the blink objects to be dead?

Comment 11 by hbos@chromium.org, Sep 13 2017

re #9: It does make sense for the event to fire after the PC is destroyed because the event is fired on the stream, not on the PC. Ignoring events after destruction is not an option, assuming the stream/tracks aren't dead.

Comment 12 by hbos@chromium.org, Sep 13 2017

I should try to investigate and make sure the WebMediaStream/WebMediaStreamTracks really do keep the objects alive.
The problem is vector.push_back which you're calling at MediaStream::ScheduleDispatchEvent. It may cause a reallocation of the internal vector buffer. This is unsafe because 1) the reallocation creates new objects and 2) the constructor may mutate the object graph in an non-acceptable way.

Either way, the pre-finalizer is designed to do only a couple of very simple operations; e.g., clear pointers. We shouldn't do anything complex like dispatching events.


Comment 14 by hbos@chromium.org, Sep 14 2017

OK. I'll update the RemoteMediaStreamImpl (now refactored -> RemoteWebRtcMediaStreamAdapter with same flaw) destructor not to do anything complex like dispatch events or add or remove tracks from the stream.

That should revert what caused this regression.

Comment 15 by hbos@chromium.org, Sep 15 2017

I'm unable to repro this by invoking gc() because of a bug that keeps the RTCPeerConnection alive https://crbug.com/765656. I can try to fix both bugs to get test coverage or I can speculatively make the destructor non-complex without a corresponding test case.

Comment 16 by hbos@chromium.org, Sep 18 2017

Blockedon: 765656
Project Member

Comment 17 by ClusterFuzz, Oct 14 2017

ClusterFuzz has detected this issue as fixed in range 508789:508862.

Detailed report: https://clusterfuzz.com/testcase?key=6640919901372416

Fuzzer: inferno_twister
Job Type: mac_asan_chrome
Platform Id: mac

Crash Type: CHECK failure
Crash Address: 
Crash State:
  !Allocator::IsObjectResurrectionForbidden() in Vector.h
  void WTF::Vector<blink::Member<blink::Event>, 0ul, blink::HeapAllocator>::Append
  blink::MediaStream::RemoveTrackByComponent
  
Sanitizer: address (ASAN)

Regressed: https://clusterfuzz.com/revisions?job=mac_asan_chrome&range=478940:478956
Fixed: https://clusterfuzz.com/revisions?job=mac_asan_chrome&range=508789:508862

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=6640919901372416

Additional requirements: Requires HTTP

See https://github.com/google/clusterfuzz-tools for more information.

If you suspect that the result above is incorrect, try re-doing that job on the test case report page.
Project Member

Comment 18 by ClusterFuzz, Oct 14 2017

Labels: ClusterFuzz-Verified
Status: Verified (was: Assigned)
ClusterFuzz testcase 6640919901372416 is verified as fixed, so closing issue as verified.

If this is incorrect, please add ClusterFuzz-Wrong label and re-open the issue.

Comment 19 by hbos@chromium.org, Oct 16 2017

Labels: ClusterFuzz-Wrong
Status: Unconfirmed (was: Verified)
The fixed range includes https://chromium-review.googlesource.com/657639, which is the only relevant CL in that range.

I suspect this isn't fixed, but I can't repro. Likely this bug can only trigger under normal circumstances when the RTCPeerConnection can be garbage collected. https://crbug.com/765656

For now this is an unlikely low-prio bug that I can take a look at later.
Status: Assigned (was: Unconfirmed)
Assigning to hbos. 

Sign in to add a comment