Thank you for the webrtc_internals_dump file!

Follow up questions for the original reporter:

a) Chrome 49.0.2623.75 was pushed to the stable channel on March 2, 2016 [1] and was updated about a week later to 49.0.2623.87 on March 8, 2016 [2]. Can you try the same test with Chrome 49 and let us know if you still see the same issue? If you do, can you try to collect an RTP header dump? You can get that via chrome://webrtc-internals, starting with Chrome 49, by clicking the "Enable diagnostic packet and event recording" checkbox [3]. Please also collect a new webrtc_internals_dump.

b) From what I can see in ssrc_1250082313_recv, it looks like Chrome received > 300 kbps for video for about the first 20 seconds of the call, then that gradually dropped down to ~100 kbps from about 40s onwards. Yet despite that drop in received bitrate, the received video resolution (googFrame[Height|Width]Received) remained at 640x480 up through the 150s mark, after which it dropped to 320x240. googFrameRateReceived stayed around 25+ fps for most of the call, except for a brief dip between 142s and 151s. VGA resolution with ~100 kbps is doable, but not always ideal. Do those stats on the Chrome side match what you saw from the media server?

c) As for the PLIs, I only see 5 sent from 0s through 26s, with no new ones sent for the remainder of the call. Did you see other PLIs during this call?

d) "After a long time (a couple of minutes), it seems that video starts playing smoothly."
At least for this call, did the smooth video playback seem to align with the time Chrome started to receive QVGA from the media server?


[1] http://googlechromereleases.blogspot.com/2016/03/stable-channel-update.html
[2] http://googlechromereleases.blogspot.com/2016/03/stable-channel-update_8.html
[3] https://groups.google.com/forum/#!msg/discuss-webrtc/mcApW-3YADI/2SxLihQuGgAJ

Also please ensure that your media server adds continuous picture ids to the outgoing vp8 frames, otherwise webrtc can't determine continuity if there are lost FEC packets and will have to NACK them.

Friendly ping to the original reporter to answer the questions/requests in comments #2 and #3.

It's been two weeks since the questions in #2 and #3 were posted, and we haven't received a response. Therefore, I'm closing this bug.

Hi guys
Sorry for the delay in answering you, I've been out last few weeks and I could test what you proposed.

Current settings:
  * Chrome version 49.0.2623.110 (64-bit) Linux
  * Media server sends FEC packets every 11 media packets
  * Chrome and media server in the same machine with no losses
  * Vp8payloader using 15-bit picture id

Result of the tests:
  Video keeps freezing and in some intervals plays smoothly until it freezes again. It seems that video never recovers and never plays smoothly. There is not packets lost, no Nacks sent but some PLIs are still being sent. Server side stats matches the ones displayed on Chrome.

I've attached both, webrtc internals and header packets got from chrome.

Regards

Deleted: webrtc_internals_dump.txt 1.6 MB

webrtc_internals_dump.txt 1.6 MB View Download

Deleted: webrtc_log.17874.event_log.13 1.4 MB

webrtc_log.17874.event_log.13 1.4 MB Download

Hi guys.
Will this issue definitely not fixed?

If this doesn't reproduce between two Chrome browsers, I'd recommend that you first figure out what the difference is between Chrome's FEC and the FEC produced by your media server. That way we could determine if the different behavior is something that Chrome should handle in a better way or not.

Checking Chrome's FEC packages is not that easy so I must force a lot of loses on the network until I get a FEC packet sent from Chrome, furthermore, under  such conditions, It seems that Chrome does not send FECs packages regularly, so it complicates searching a common pattern to discriminate errors by comparing those packets with the ones packets sent by the media server.
All I can see is that FEC packets (the ones sent by chrome and the ones sent by the media server) use FEC Level 0 header and payload, mask varies in chrome depending on the packets to be protected, on the other hand, packets sent from media server keeps the same so it is configured to protect packets every 11 media packets, so it enables first 11 bits of the 16 bits mask in the FEC packet.
To discard possible malformed FEC packets in the server side, I tested that the server is capable of recovering media packets using FEC packets generated by both, other media server and chrome, and in both cases the media server is capable of recovering media without any problems.
Furthermore, I configured the server to send only one FEC packets, and it seems that chromes does not get affected then. The problem seems to happen when the media server keeps sending packets as a regular basis without loses on the network.
Is there any way I can configure chrome to force FEC packets to be sent more frecuently. Any help hint will be more than appreciated.

Regards

If you build Chrome for source you can enforce a fixed FEC rate by changing this line: https://code.google.com/p/chromium/codesearch#chromium/src/third_party/webrtc/modules/rtp_rtcp/source/rtp_sender_video.cc&l=239

and pass in a FecProtectionParams with an fec_rate which is greater than zero all the time.

Another option would be to send a fake RTCP RR from the receiver with a non-zero loss report.

Are you sending FEC packets in the middle of frames, or between frame boundaries? I think you may run into problems with Chrome if you do the former, e.g., for a frame with 2 packets you are sending:
[seq=1] [seq=2 fec] [seq=3 m-bit]

If the FEC packet is lost Chrome will have to wait for a retransmit to know that the frame is complete. For similar reasons the picture ids must be set correctly for FEC to be useful.

Actually, the media server is not checking wheter FEC packets are being sending in the middle of a frame or between frame boundaries. It just sends a new FEC packet every 11 media packets, that's all.
Please, could you tell me if this is a special requirement for VP8 or this restriction could also apply for other codecs, eg. h264?. Looking at RFCs I find no documentation about this stuff, FEC is too lax regarding this subject.
I'll make the suggested changes and I'll check everything again.
Thanks a lot for your comments

Any case, tests I've made does not include loses from server side to chrome, so there is never a packet lost between them.

It's a limitation with the ulpfec scheme since the FEC packets are sharing the same sequence number space as media packets. We've worked around the issue by always sending FEC packets between frames, and used picture ids to ensure we have all the necessary frames needed for continuity.