New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 906027 link

Starred by 8 users

Issue metadata

Status: Verified
Owner:
Closed: Dec 21
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux , Windows , Mac
Pri: 1
Type: Bug


Participants' hotlists:
WebRTC-Unified-Plan

Show other hotlists

Other hotlists containing this issue:
WebRTC-1.0-Spec-Compliance


Sign in to add a comment

Too many transceivers degrade video quality

Reported by marten.k...@googlemail.com, Nov 16

Issue description

UserAgent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0

Steps to reproduce the problem:
1. open attached website
2. click 'Start' to establish a set of PeerConnections
3. stable 60fps video stream is visible
4. Click 'Add passive transceiver ...' 4 times and 'Start Signaling' once to have a total of 5 transceivers attached to the PeerConnections
5. Video quality remains the same (stable framerate and resolution)
6. Click 'Add passive transceiver ...' and 'Start signaling' once more for a 6th transceiver
7. Video quality degrades a lot (lower resolution, lower framerate)

What is the expected behavior?
Video quality remains the same as long as no additional streams are transmitted - meaning non-sending or non-receiving transceivers should not have any impact on video quality

What went wrong?
Video quality degrades once 6th transceiver is added. Also the googAvailableSendBandwidth from bweforvideo in chrome://webrtc-internals drops significantly.

Did this work before? N/A 

Does this work in other browsers? Yes

Chrome version: 70.0.3538.102 (Official Build) (64-bit)  Channel: stable
OS Version: 70.0.3538.102 (Official Build) (64-bit)
Flash Version:
 
index.html
5.6 KB View Download
Components: Blink>WebRTC>Video
Labels: Needs-Triage-M70
Cc: vamshi.kommuri@chromium.org
Labels: Triaged-ET Target-72 M-72 FoundIn-71 FoundIn-70 FoundIn-72 OS-Mac OS-Windows
Status: Untriaged (was: Unconfirmed)
Thanks for filing the issue!

Able to reproduce the issue on reported chrome version 70.0.3538.102 and on the latest canary 72.0.3615.0 using Ubuntu 17.10, Windows 10 and Mac 10.14.1

Our Observations:
-----------------
On M60(60.0.3112.0) - After clicking "start", browser isn't asking for camera access and we couldn't establish any connections. This behaviour is seen till M68(68.0.3440.134)
In M69(69.0.3497.100) we could access the camera and establish the connections without any issues, but the second screen is going black/dark.

Hence marking it as Untriaged and requesting someone from "Blink>WebRTC>Video" team to have a look into it and help in further triaging it.
Thanks for the feedback. As you can see in line 51 we forced the PeerConnection to use 'unified-plan', hence the video doesn't turn on in <=M69.
Owner: vasanthakumar@chromium.org
Status: Assigned (was: Untriaged)
Vasanthakumar, could you help getting text- and event logs from this?
Owner: terelius@chromium.org
Here you go Terelius.


webrtc_internals_dump (23).txt
1.0 MB View Download
event_log_20181126_1052_1561_11.log
451 KB View Download
event_log_20181126_1052_1561_12.log
376 KB View Download
event_log_20181126_1052_1561_14.log
173 KB View Download
event_log_20181126_1052_1561_13.log
252 KB View Download
Cc: jansson@chromium.org rantonysamy@chromium.org
Cc: hbos@chromium.org crodbro@chromium.org srte@webrtc.org
Sorry for the delayed update.

I've figured out what happens, but it is not clear to me what the correct behavior is.

When WebRTC sends RTP packets (using send-side bandwidth estimation), it expects to receive RTCP feedback that signals which sequence numbers have arrived. The RTCP feedback is then fed to the bandwidth estimator which computes the network delay for the packet by looking up the sequence number in the send time history. Since each packet is only supposed to be used once, the packet is also removed from the send time history.

What happens with multiple transceivers is that the RTCP packet is passed to each VideoSendStream, and each VideoSendStream forwards it to the congestion controller. The first call for each RTCP packet will find the correct send times and prune them from the history, so subsequent calls will fail. This is interpreted as the feedback being so delayed that the packets have timed out from the send time history. This triggers another piece of logic in the bandwidth estimator; after receiving kMaxConsecutiveFailedLookups delayed feedbacks, the bandwidth estimator reduces the send rate to 50% in an attempt to recover from severe network congestion. kMaxConsecutiveFailedLookups is currently 5, so with 6 transceivers and for each RTCP, we'll get one successful lookup with a normal BWE update, then 5 failure causing a 50% reduction. This very quickly converges to the minimum bitrate.
Labels: -Pri-2 Pri-1
Thanks, I added it to our hotlists. This is important for the Unified Plan rollout.
Imo, the root cause is that each RTCP packet is duplicated for each send stream. This might require a bit of a redesign to fix though.

As a quick fix, we could disable the "backoff on delayed feedback" which we don't believe is critical.
Cc: mflodman@chromium.org holmer@chromium.org
https://webrtc-review.googlesource.com/c/src/+/114165 implements the "quick fix". I'll update once I have verified that it solves the issue.
Cc: huib@chromium.org
Cc: youe...@gmail.com
Status: Started (was: Assigned)
Any update here? I see terelius@ is OOO for christmas but this is something we might want to merge into M-72.
The #12 CL has been merged last Friday (the 14th).
srte@ you reviewed the change, can you take a look if we can merge this?
I've filed https://bugs.chromium.org/p/chromium/issues/detail?id=916151 to merge to M72. I've verified that at the sample page works with at least 20 transceivers on tip-of-tree Chromium.
Just verified with Chrome Dev 73.0.3642.0 myself, and quality is stable with up to 24 passive transceivers. 

Thanks for the fix
Status: Verified (was: Started)
Awesome, I'll mark this as verified.  https://crbug.com/916151  was approved and this has been merged into M72. Possible follow-up work may be tracked in https://crbug.com/webrtc/10125.

Sign in to add a comment