New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 775450 link

Starred by 2 users

Issue metadata

Status: WontFix
Owner:
Last visit > 30 days ago
Closed: May 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

WebRTC: One way media when video is used through SBC

Reported by boazsel...@gmail.com, Oct 17 2017

Issue description

Chrome Version       : 59.0.3065.0 and up
OS version               : Windows 7 64-bit
Audio/Video format (if applicable): Opus and VP8

Behavior in Firefox (if known): Works

Video issue, Audio issue, both, neither? both

Flash or HTML5?  <right-clicking most players will either reveal some text
with “Flash”; otherwise likely HTML5>

If the browser or renderer crashed (“Aw, Snap”), please add any crash
IDs from chrome://crashes  (possibly after enabling crash reporting per
http://support.google.com/chrome/bin/answer.py?hl=en&answer=96817)

What steps will reproduce the problem?
(1)
(2)
(3)

What is the expected result?

What is the actual result?

Any additional information (anything else which may help us debug the
issue)?
I'm trying to make a webrtc call via SBC.
For some reason since this version I have one way voice and video.
I can hear and see all the time on the calling side.
On Firefox video and audio on the answering side open after ~2 min.

I'm attaching logs of old version and new.

Regards,
Boaz

Please attach the HTML5/JavaScript code or audio/video files as well as
screenshot and/or videos (if applicable)

 
log_not_working_59.0.3065.0.txt
126 KB View Download
log_working_59.0.3064.0.txt
94.9 KB View Download
Components: Blink>WebRTC

Comment 2 by guidou@chromium.org, Oct 18 2017

Components: -Internals>Media
Labels: Needs-Feedback
Can you provide a more specific description of the problem? Most of the description and summary are template text unrelated to the bug
Does this issue reproduce with a current Chrome version, including stable and perhaps dev/canary?
Can you provide reproduction steps together with a script (jsfiddle or similar) or at least a URL that we can use to try to reproduce the problem on our end?
Excuse me. I'm not experienced with bug reporting here.

* It reproduces with canary.
* The scenario is:

JSSip -- SBC -- JSSip

URL: https://tryit.jssip.net
Click the Settings icon.
SIP URI: 301@<your ip address> (in the other participant pick 302)
WebSocket URI: wss://<sbc-address>
Name: Whatever

Create 2 tabs (or 2 Chrome users) with different URIs, then call from one to the other.

We tested nightly builds and found that build 462348 works and 462358 and up are broken (the scenario works in very rare cases on these versions). The corresponding commit range that broke this according to our tests (which might be inaccurate) is:
dd3c9a967d856c4dc8b41fc1738caa4c035c6b82 - good
b1a1d1ecc123788db5696b3efaf9464e75460957 - bad

We executed some calls from Chrome 58 to Canary (64), and extracted the logs of the failing side (Canary) alone. They are attached. Both logs are from the same version, one time it worked (good) and one time it didn't (bad).

Notice that in the good log there is the following:
INFORMATION	5972	7220	09:34:35-331		0	Received first video RTP packet
INFORMATION	5972	4736	09:34:35-345		0	Initializing decoder with payload type '96'.
INFORMATION	5972	7220	09:34:35-351		0	SetOutputVolume() to 0 for recv stream with ssrc 1205191274
INFORMATION	5972	7220	09:34:35-351		0	SetOutputVolume() to 1 for recv stream with ssrc 1205191274

While in the bad log it looks like this:
INFORMATION	5972	7220	09:27:47-727		0	Received first video RTP packet
WARNING	5972	9940	09:27:47-927		0	No decodable frame in 200 ms, requesting keyframe.
VERBOSE	5972	7220	09:27:47-942		0	Jingle:Conn[000000000D6905D0:audio:0xTUTQQq:1:0:local:udp:172.17.137.x:59049->9yIydt3t:1:2130706431:local:udp:172.17.137.x:6540|CRWS|0|0|9115038255648079870|5]: UpdateState(), ms since last received response=1047, ms since last received data=3, rtt=100, pings_since_last_response=
...
VERBOSE	5972	7220	09:27:47-945		0	Jingle:Conn[000000000D691060:video:79n2j3XV:1:0:local:udp:172.17.137.x:59050->Q9Y+ILEz:1:2130706431:local:udp:172.17.137.x:6550|CRWS|0|0|9115038255648079870|6]: set_state
WARNING	5972	9940	09:27:48-127		0	No decodable frame in 200 ms, requesting keyframe.

and so on...

When it doesn't work, both audio and video are one-way.

* I currently can't provide a testing environment, but I might be able to run the system on Amazon in a few days and give you access to it.

If possible, please change the subject of this bug report to something like WebRTC: One way media when video is used through SBC

Any help debugging this will be appreciated.

Thanks!
chrome64log_good.txt
936 KB View Download
chrome64log_bad.txt
258 KB View Download
Project Member

Comment 4 by sheriffbot@chromium.org, Oct 18 2017

Cc: guidou@chromium.org
Labels: -Needs-Feedback
Thank you for providing more feedback. Adding requester "guidou@chromium.org" to the cc list and removing "Needs-Feedback" label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 5 by guidou@chromium.org, Oct 18 2017

Components: -Blink>WebRTC Blink>WebRTC>Network

Comment 6 Deleted

Comment 7 Deleted

Comment 8 by guidou@chromium.org, Oct 18 2017

Thanks for the updated report.
r462351 is the only WebRTC roll in the range reported by #3. No other Chromium commits in the range look related.

Deleted #6 to avoid confusion with incorrect commit.
Components: Blink>WebRTC>Audio Blink>WebRTC>Video
Summary: WebRTC: One way media when video is used through SBC (was: <Replace this with a summary. This form is for audio/video issue (HTML5).>)
Hmm, I don't see anything suspicious in that range. Adding Audio/Video components so that people more familiar with debugging "one way media" issues can advise on next steps.

The fact that "Initializing decoder with payload type '96'." appears in one log but not the other seems to indicate that the RED packet itself isn't processed correctly.

Comment 10 by org...@gmail.com, Oct 18 2017

Hi,

We're not positive about the commit range. The bug is non deterministic.

What we do know for sure is that 58 always works, and 59 works rarely. We did our best to bisect by the builds, but we can't know for sure.
Hi there,
We manage to bring our setup to the public internet.
Here are the steps for reproduce:
* First please run Chrome with flag "-ignore-certificate-errors"
* we need to open 2 users
  available users are:
  - user1: webrtctest1 password: 1234512345
  - user2: webrtctest2 password: 1234512345
* Let's log-in our users (need to do it for each user):
  - Open the page https://tryit.jssip.net/
  - Click on advanced option (Gear wheel)
    - SIP URI: <user>@<yourIpAdd> (like webrtc1@192.168.0.1)
    - SIP password: 1234512345
    - WebSocket URI: wss://52.40.162.88:443
    - Click on OK
  - Your Name: test 1 or something like it
  - click on the arrow or enter.
  - Wait for green status.
* Do the same for the other user.
* Call from one user to the other.

You can take Chrome version 57.0.2987.133 - in this version video and audio should work.
After it try Chrome version 60.0.3112.78 - in this case you should have audio 2 ways and video on way.

Thanks you so much for your help

Regards

Comment 12 by org...@gmail.com, Nov 6 2017

Hi,

Can you please have a look? We're quite stuck here with this.

Thanks ahead :)
Any chance you can take a look on it?
Please let us know if you are encounter with any problem.

Thanks,
Boaz
Owner: sprang@chromium.org
Status: Assigned (was: Unconfirmed)
sprang, can you or ilya help investigate what's going on here?
Cc: sprang@chromium.org
Owner: philipel@chromium.org
Looks like there's some kind of issue with the received video stream. In the one that doesn't work there's a number of warning about PacketBuffer expanding, and also a number of "No decodable frame in 3000 ms, requesting keyframe.".

If you're doing any kind of selective relaying and packet mangling in the SBC, can you please make sure there are no gaps/jumps in sequence number or picture id in the outgoing stream?

Otherwise we'll need to do an rtp recording and look deeper.

Assigning to philipel@ as the jitter buffer expert.
Hi,
We double checked it and we don't have any gaps or jumps in the seq numbers.

I appreciate your help,
10x Boaz

Comment 17 by org...@gmail.com, Dec 3 2017

Hi,

Any progress with this?

Comment 18 by org...@gmail.com, Dec 17 2017

ping
Have you checked that there are no gaps/jumps in the picture id as well?
Yes. We double checked it and all looks ok with the picture id.
I think if you want this investigated you'll have to provide us with an RTP recording (wireshark / tcpdump) on the receiver side when the problem happens.
Hi,
As you requested I'm sending you rtp captures.
I also added console and logger for each side of the call.
All rtp traffic is encrypted, in the file call_details.txt you can find all the encryption keys.

Files too big for upload.
Link to the files:
https://drive.google.com/file/d/1xa7MaVVZoIgQCd-6fZzSkdrApzo2QPbf/view?usp=sharing

Thanks
Hi,
Did you had a chance to take a look on it?
Sorry, I have not had time to look at this yet.

Unfortunately our tools do not support the pcapng format, we only suppport rtpdump and pcap. It would be very useful if you could produce an unencrypted rtpdump/pcap file.
Hi,
Attach decrypted pcap file of the callee side (on this side we have the problem with the video (video picture freeze after couple of seconds)
decrypted.zip
3.3 MB Download
Do you use WebRTC on the sender side? I had a look at the RTP dump and it looks like packet 14928 is a RED packet that has encapsulated a RED packet.
I had a closer look at the payload types present in the rtpdump in #25, and they look seemingly random. Could you verify that the payload in the rtpdump is in fact decrypted?

Comment 28 by org...@gmail.com, Mar 20 2018

Hi,

Sorry for the late response.

Here are decrypted rtpdumps of audio and video.

orig is the call originator (the side that sent the initial INVITE), term is the terminator (the other party).
rtpdumps.zip
2.9 MB Download

Comment 29 by org...@gmail.com, May 3 2018

Hi,

It appears that the problem was that we did not preserve SSRCs, but generated new ones for the outgoing leg. We also had some bugs with RTCP forwarding. It seems to work as expected now. Thank you.
Status: WontFix (was: Assigned)

Sign in to add a comment