Thanks for the report,more updates in: 699269

anyway I can get read access to 699269?

It's not clear to me that this is a duplicate of 699269 (a crash in what appears to be to be an unrelated area). 

Stefan, can you attach the webrtc-internals output and additional detail that we discussed in email?

Stefan,could you please also try to repro this bug on latest M59 Beta 59.0.3071.36? You can download it from https://www.chromium.org/getting-involved/dev-channel. Thank you.

Here is a screen cap of the webrtc internals during during the incident

http://prnt.sc/f13tk1

and the corresponding dump attached

Deleted: webrtc_internals_dump (1).txt 1.1 MB

webrtc_internals_dump (1).txt 1.1 MB View Download

an easier way to reproduce the issue is to use our quality selector to change the stream to a different resolution which causes a mid-stream change of of the resolution the server is sending:

http://prntscr.com/f5bt4l

again this works very well in chrome 57, but does not work in chrome 58 or even in canary (60)

i wasnt not able to download 59.  selecting the dev-channel gave me chrome 60

sometime with the quality selector it will switch just fine, other times it will show that you are continuing to receive packets but the dropped frames is equal to the frame rate thus the video is not updating.  other times it will show a very low frame rate, only showing every 10th frame.  if you dont see it the first time just change the selector between various resolutions

After some quick debugging I don't think this is related to the new jitter buffer.

Frames are continuously sent to the decoder and decoded just fine, but sometimes the video freeze while the audio keep playing. By just switching back and forth between some other tab will make the video render again, in sync with the audio.

Also, I got this behavior on linux (60.0.3095.0) so I guess this is a problem for all OSes?

Just to clarify #11, the video sometimes freeze when I switch video quality.

philipel@, did you verify that frames gets decoded or did you only look at webrtc-internals? When I repro this googFrameRateDecoded doesn't drop, but it stays at exactly the same value if had before - this normally mean the value isn't updated at all.

I verified that frames are continuously decoded and that the decoder doesn't return any error codes (from within WebRTC, not looking at webrtc-internals).

Also, if I leave the stream frozen for ~10 seconds and then switch between the tabs, the video instantly resumes playing without any artifacts and synced to the audio. This implies that frames were continuously decoded.

here's another capture of when the frame rate was extremely low (perhaps 1 frame every few seconds)

http://prntscr.com/f5vind

I'm starting to think something is wrong on the sender side when changing resolution. I get different behaviour when I try this repeatedly. Look at webrtc-internals for this case where I start the video, wait 20 secs, change resolution, let it freeze for 20s. We receive 20 Mbps and 800 fps...

Deleted: Screenshot from 2017-05-09 09:38:44.png 70.6 KB

	Screenshot from 2017-05-09 09:38:44.png 70.6 KB View Download

It could be that there are two issues at play here. 1) the frame rate gets really low, 2) rendering stops and only resumes after switching tabs back and forth, here we do see frames being decoded as expected.

We've seen both happen. What's interesting with #1 is that you can see the target jitter buffer level, which usually are around 10 seconds on this page, plummets to 0, and seems to jump a bit up and down when this happens.

Philip, I think it would be good if you looked more into that issue tomorrow.

Niklas, can you also reproduce problem #2? Would it be possible for someone in your team to take a look and see if they can figure out what that might be about?

That's an interesting observation, and I can also see this happening.

Emircan, since Qiang is out, can you take a look at what happens in Chrome during a freeze?

Similar to #16, I also see 20 Mbps and 800 fps received after the freezing that happens due to the send resolution changing. In my case, however, there are no frames decoded, there's a target delay on the order of seconds, and PLIs are continuously being sent.

When switching resolution does not lead to a freeze, I quite often see decoding artifacts that last for seconds to minutes. Not sure why this would happen if IDRs are sent every 3 seconds?

Repro'd on 60.0.3094.0.

Deleted: Untitled 2.png 143 KB

	Untitled 2.png 143 KB View Download

Deleted: IMG_20170509_184011.jpg 4.2 MB

	IMG_20170509_184011.jpg 4.2 MB View Download

As the earlier screen shots from #20 and #16, I see a big jump in frameRateReceived(>500fps) during the freeze. Switching tabs doesn't help for me most of the time. There is clearly an issue in the sender side here. We can dig deeper if the issue happens even when there is ~30 fps.

I tried to understand why we do not see any frame updates during the freeze where hundreds of frames are incoming. It looks like it coincides with the logic in WebMediaPlayerMSCompositor where they detect if render callbacks stop. As a result, all the pendign frames are released.
https://cs.chromium.org/chromium/src/content/renderer/media/webmediaplayer_ms_compositor.cc?rcl=03c804e825b7c11870ce21da2e98ff5f70cd8462&l=251

Adding qiangchen@ who knows more about this algorithm. Also, this looks a bit like  issue 705679  which is already addressed.

Deleted: Screenshot from 2017-05-09 13:32:43.png 104 KB

	Screenshot from 2017-05-09 13:32:43.png 104 KB View Download

you should not see the remote client sending more than the frame rate, that's certainly unusual.  If this were the case I would expect to see the packets/s shoot up and the bitrate which doesn't happen. the behavior I see most consistently is when I switch resolutions chrome misses the first idr frame, sometimes many of them but will often start playing again.  other times it will never start playing as is the case for this that I reproduced this morning using a new version that only starts sending the new stream when it receives an IDR frame to ensure it's a clean switch:

http://prntscr.com/f68jl4

I still can reproduce the frameRateReceived shooting to 500ms+. Mostly when I switch directly to 320p.

I reproduced one without frameRateReceived going to high values as observed before. I printed the incoming height followed by elapsed timestamp[0] to understand why the rendering fails. It looks like when there is a resize, there is a huge fall in the incoming timestamps. To be specific, they are much less than before and the calculated elapsed timestamp falls into negative domain which the algorithm doesn't seem the handle properly. 
- Sender can investigate why there is such a jump between timestamps of different sizes.
- We should handle these jumps more gracefully. The fix on  issue 705679  does not seem to help in this case. I am assigning it to qiangchen@.

Turning off the smoothness algorithm(start chrome with --disable-rtc-smoothness-algorithm) fixes it for me. Can you please also try that?

[0] https://cs.chromium.org/chromium/src/content/renderer/media/webrtc/media_stream_remote_video_source.cc?rcl=7ff54f9912ee3dd2132e0fc7c1e9bdadc06b46b1&l=112

I will investigate the timestamps discontinuity and try to repro without the smoothness algorithm.. thanks for the additional information

I fixed the timestamp discontinuity and this fixed the freezing after switching streams.  Thanks for the insight on this.

On the original 3 issues however we are still regularly experiencing these on Chrome 58.  Again these issues are:
1.) Stream wont't load (blue pulsing ball indefinitely)
2.) Stream stops after some random period of time (audio continues)
3.) Stream has a high number of dropped frames

I was able to reproduce #1 this morning again on https://beam.pro/FTL-TestStream2.  Im getting pli's requests but again there was 0 packet loss for a large period of time and I am sending key frames every 3 seconds

http://prntscr.com/f856pl

also just reproduced the random stopping of the stream using the same sequence:

http://prntscr.com/f85uey

so i guess my question is, why arent the IDR frames being 'seen' since you are continuing to send pli's even though there is no packet loss

cc philipel, are these IDR frames sent as SEI NAL units? See https://bugs.chromium.org/p/chromium/issues/detail?id=719095 

no, there are no SEI nalu's in the IDR...there is an sps, pps and then one or more FU-A nalu's that contain the actual IDR frame

is there anything i can get on my end to help?  a network capture between my pc and the server?

Assigning over to Philip since the internals graph shows us being stuck in a mode where we request key frames continuously.

this stream probably wont be live for very long but it loads in canary and edge but not chrome 58.0.3029.110 (Official Build) (64-bit)

i captured the stream coming from the client and am analyzing now

https://beam.pro/Hollywood2DF

Re #35. that stream loads for me but playout freezes regularly. Looking at webrtc-internals I see one strange thing, the number of NACKs exceed the number of lost packets roughly a factor 10. Stefan, have you verified that your sender honors the NACK requests?

We don't have a massive nack buffer as generally our rtt to our cdn is low.  I have seen that behavior before also.  do you continually ask for missing packets?  I have also see the case where there is no packet loss at all but constant pli's as below

http://prntscr.com/f94s85

as mentioned I captured the stream to our ingest server.  here is a tcpdump capture along with the corresponding elementary streams:

https://www.dropbox.com/sh/4wnwt92lpkpgw6s/AAB1U3Dg5dQaiIaU1K9cvFGFa?dl=0

meaning, do you continually ask for the same missing packet over and over?

i setup a stream that consistently reproduces the 'wont start' scenario where chrome v58.0.3029.110 (Official Build) (64-bit) just continues to receive packets and wont show video.  Sometimes it will show a frame and start playing audio, most of the time it wont show anything.  it's difficult to predict if you will receive packet loss, but even when there is 0 packet loss it wont load.  Again, there is no problem in Canary or Edge.

https://beam.pro/FTL-TestStream6

here is another one I stood up for you:

https://beam.pro/ftl-teststream7

here is one that if you can get it to load will exhibit the very low frame rate issue, appear to only be decoding the IDR frames and nothing else (although i can be certain they are the key frames)

https://beam.pro/FTL-TestStream8

one more...this one will stop randomly from time to time

https://beam.pro/FTL-TestStream9

We have made a few changes regarding H264 and I tested it on ToT, things work better now but low fps video still occurs. I will continue to look into it.

I have spent a few hours testing stream 6 to 9, and at ToT I can find two things, one with major and one with minor impact.

The major thing is that the SFU does not seem to respond to NACK, which means any packet loss will cause the stream to halt until a new keyframe is received. Using a 1% packet loss makes the stream unwatchable.

The minor thing is that we sometimes drop frames due to them being late to the decoder, but this happens only rarely, and the impact is a small (very) small stutter.

I could not get a stream to fail to start (given you don't refresh the tab, because then it never restarts and it looks like no data is sent) and sometimes there was some initial stutter that went away fairly quickly.

thanks!  if there is anything you need to help repro, etc.  please don't hesitate to ask

those test streams will not respond to nacks, they are canned streams that are being sent 'as is' to replicate these issue.  In general packet loss is expected and having the video stop until the next IDR is an acceptable behavior.

on the stuttering, my expectation would be the jitterbuffer should grow to minimize 'late' frames (within reason obviously).  is this the case?

There seems to be a large difference between M58 and tip-of-tree. With M58, the streams often freeze completely and the client starts sending non-stop PLI. I haven't been able to reproduce this or any other clear bug using tip-of-tree. The streams will stutter a bit but the stuttering seems to correlate with lost packets, so the behaviour is expected. I guess our next step should be to investigate whether M59 behaves as M58 or as tip-of-tree.

Could you clarify what the server does? It is a pre-recorded video, but you're saying that the video isn't treated as a normal camera- or screen-capture? Then the packets are sent over a PeerConnection to the client, but the server ignores all RTCP?

Restarting the stream by reloading the tab doesn't work even in current canary. To reload the page, I have to close the tab and open a new one. I have no idea why.

Could you check whether you experience any of the other issues in current canary (60.0.3111.0 or later)?

the original stream is pushed to our ingest server then relayed to all subscribed distribution servers and then onto the client. if there is loss between the distribution server and the client those packets will be nacked, but if there is a packet missing at the origin, as is the case in these streams, requesting a nack will obviously not succeed as the packet didnt exist in the first place

sslivins@, the jitter buffer is tuned to favor low delay rather than smooth video.

stefan@, the SFU not having a way to recover lost packets sounds like it could easily lead to a large amount of freezes for the viewers. If you stream 200 packets per second with a 1% packet loss you would expect the stream to freeze half a second after the last keyframe.

Should this be duped with 719095?

Regarding #49, definitely seems like the sender responding to NACKs from the server would be the best way to handle this - otherwise a loss on the sender->server link will cause a freeze for all viewers.

these streams do respond to nacks at the server, what i am saying is they are captures of problematic streams and if there is packet loss in that capture nacks for those packets it will obviously not be able to provide them

so i dont want to get side tracked, what i am trying to understand here is what is it about these streams that cause this behavior.  obviously packet loss, out of order packets, jitter, etc are normal and expected.  is there something non-compliant about these streams or something i can do differently to prevent chrome 58 from behaving the way it does?  This behavior obviously has a huge impact on our viewers.  our typical session is over an hour and users are experiencing permanent freezes often within 10 minutes. our telemetry showed a huge decrease in session length as soon as chrome 58 dropped.  we continue to see chrome 58 has the lowest session length of all browsers.  Since chrome 58 is the majority at the moment for chrome browsers, i'd love to know any way I can to mitigate this

sslivins@

I don't know exactly what is causing the problems with certain streams, but I can tell you what problems I have seen so you can avoid those.

The first is the problems with SEI nalus. In a stream where frames starts with a SEI nalu (which is a valid stream) we will interpret that frame as a delta frame. This has caused streams with SEIs in them to never start since we don't get a keyframe.

Another problem I have seen is sending SPS/PPS mid-stream (again, it's valid to do so). We only expect to receive SPS/PPS before a keyframe, and because of this and a few other reasons these packets were not inserted into the packet buffer. This in turn cause the stream to become non-continuous (packets are missing!) and we can't decode it.

Both of these problems have been fixed and we have a merge request here:  crbug.com/719095 . It would be good if you comment in that bug explaining the impact these problems cause.

Another bug that is this one: https://codereview.chromium.org/2899713002/. This one is not that common and also cause a tight loop which cause the tab to hang. It has already been merged to M59.

The last problem I know of is related to packet-loss/reordering. In a H264 stream, if the last packet of the frame is received before some other packet of that frame there is a risk that we consider that frame to be received and we create a frame from those packets. This will cause the stream to freeze since there is a gap in the packet sequence.

A note on #55, we are working on improving loss/reordering robustness for H264 streams.

thank you for the details, it's greatly appreciated

so we should never send SEI messages nor should be send an sps/pps outside of an IDR frame so i dont think those issues apply in my case.  the last one is obviously extremely common in an environment with packet loss and out of order packets.  do you expect the stream would freeze indefinitely when this situation happens?

#57

We will request a keyframe if we have not been able to deliver a decodable frame to the decoder in 3 seconds.

Also,  crbug.com/719095  has now been merged to M59.

#59

so the vast majority of our streams have a key frame interval of 3 seconds or less.  so in the case where a stream freezes as is the case of the ones mentioned in #39-42, whats preventing it from starting again?  is that related to the bug mentioned in #55 where the marker bit packet is arriving before other packets?

also i was curious on the dropped frames stat.  i see this often where there is a spike in drpped frames on a stream that is 60fps but the video looks completely fluid with no noticeable dropped frames and im pretty sensitive to this stuff by now so im saying with a high degree of certainty that it isnt dropping display frames, so what constitutes a dropped frame?

so it appears the problem is primarily due to out or order packets when there are multiple packets per frame.  at least the behavior i am seeing is it freeze unnecessarily.  if there is a single slice per frame and FU-A is used it doesnt freeze even whem there are out of order packets.  perhaps the right packet out of order or lost is enough to cause a deadlock in the decoder

Hopefully we will have a CL up this week which will improve the H264 reordering/lost packets situation.

The reason it works with FUA is because one NALU is one frame, so in that case we will not create incomplete frames.

Any update on this Philip?

A fix is ready but got stuck in CQ. Should land tomorrow and be in Canary on Wednesday. With that it would be great to get it verified as soon as possible so we can request a merge to M60.

Great to hear.  we'll get you feedback asap, probably within a couple days

@stefan @philipel -

I was seeing the same problem with packetloss causing long pauses in H264 streams. I'm able to reproduce it very easily using simulated packetloss.

I applied philipel's patch - https://codereview.chromium.org/2926083002/ - against a local development build of chromium, and it fixed the problem.


I have a question... is it understood why/how this regression happened? As mentioned previously, this was not an issue prior to Chrome 58.

The following revision refers to this bug:
  https://chromium.googlesource.com/external/webrtc.git/+/2c9f9f2bc9dbbb6f8abeb92615b42ce0ddfccb39

commit 2c9f9f2bc9dbbb6f8abeb92615b42ce0ddfccb39
Author: philipel <philipel@webrtc.org>
Date: Tue Jun 13 09:47:28 2017

Only create H264 frames if there are no gaps in the packet sequence number.

In the case of H264 we can't know which packet that is the fist packet of a
frame. In order to avoid creating incomplete frames we keep track of which
packets that we haven't received, and if there is a gap in the packet sequence
number leading up to this frame then a frame wont be created.

BUG= chromium:716558 

Review-Url: https://codereview.webrtc.org/2926083002
Cr-Commit-Position: refs/heads/master@{#18559}

[modify] https://crrev.com/2c9f9f2bc9dbbb6f8abeb92615b42ce0ddfccb39/webrtc/modules/video_coding/packet_buffer.cc
[modify] https://crrev.com/2c9f9f2bc9dbbb6f8abeb92615b42ce0ddfccb39/webrtc/modules/video_coding/packet_buffer.h
[modify] https://crrev.com/2c9f9f2bc9dbbb6f8abeb92615b42ce0ddfccb39/webrtc/modules/video_coding/video_packet_buffer_unittest.cc
[modify] https://crrev.com/2c9f9f2bc9dbbb6f8abeb92615b42ce0ddfccb39/webrtc/video/rtp_video_stream_receiver.cc

Yes, it's understood. It was an issue introduced with our new video jitter buffer implementation which we unfortunately didn't catch until it reached stable.

Okay, thanks for the info. Fingers crossed that this can be merged into Chrome 60.

getting this into 60 would be fantastic.  you would make a lot of people in our community very very happy.

btw is this in Canary now?

It should be in Canary by now, but hasn't reached Stable.

Any news on whether this will be merged into 60? Thanks.

Since this has been verified to fix the issue, and since it's been in Dev for some time without issues, I'll request a merge to M60 now.

Correction, it has been in Canary for some time without issues. It still hasn't made its way to Dev (missed the latest Dev build by a few hours), but it will be in this week's Dev cut.

This bug requires manual review: M60 has already been promoted to the beta branch, so this requires manual review
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), josafat@(ChromeOS), bustamante@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

bustamente, what's the status of this merge review?

Completed.  This change meets the bar and is approved for merge into M60 (build 3112).

The following revision refers to this bug:
  https://chromium.googlesource.com/external/webrtc.git/+/86c2c61b5099eaa8107b4bfd5635495055844a12

commit 86c2c61b5099eaa8107b4bfd5635495055844a12
Author: philipel <philipel@webrtc.org>
Date: Thu Jun 22 09:26:45 2017

Only create H264 frames if there are no gaps in the packet sequence number.

In the case of H264 we can't know which packet that is the fist packet of a
frame. In order to avoid creating incomplete frames we keep track of which
packets that we haven't received, and if there is a gap in the packet sequence
number leading up to this frame then a frame wont be created.

BUG= chromium:716558 
R=holmer@google.com, stefan@webrtc.org

Review-Url: https://codereview.webrtc.org/2926083002
Cr-Original-Commit-Position: refs/heads/master@{#18559}
Review-Url: https://codereview.webrtc.org/2947373002 .
Cr-Commit-Position: refs/branch-heads/60@{#5}
Cr-Branched-From: c61bf947b4ac31f3500858ffcae6fee39d799930-refs/heads/master@{#18252}

[modify] https://crrev.com/86c2c61b5099eaa8107b4bfd5635495055844a12/webrtc/modules/video_coding/packet_buffer.cc
[modify] https://crrev.com/86c2c61b5099eaa8107b4bfd5635495055844a12/webrtc/modules/video_coding/packet_buffer.h
[modify] https://crrev.com/86c2c61b5099eaa8107b4bfd5635495055844a12/webrtc/modules/video_coding/video_packet_buffer_unittest.cc
[modify] https://crrev.com/86c2c61b5099eaa8107b4bfd5635495055844a12/webrtc/video/rtp_stream_receiver.cc

I see this marked as M-59 as well. Will this be back ported to M-59 as well?

If this could land in a M-59 update, that would be great.

I think this is quite a significant bug, and the issue is affecting us and our clients in a major way, currently. Video will freeze for many seconds (sometimes indefinitely) even if a single packet is lost, and a single lost packet is a fairly common occurrence.

"Just use firefox", "Download Chrome Canary" (Soon "download Chrome beta") has become a common refrain for us, these days :/

So i spent the last week looking at quite a few streams with Canary and all issues introduced in Chrome 58 do appear to be resolved.  The one issues that was introduced in Chrome 59 which is a grey image on part or all of the image is still present.  I have not investigated the root cause as of yet but my guess is there is a corrupt or missing IDR frame which is then used for reference by the decoder resulting in the grey image until the next IDR frame comes in.  Since this issues have been marked as fixed should i open a new ticket to track this?

I believe I've seen the grey image issue  that @ssliv mentioned once or twice as well. I'm going to keep an eye out for it and see if I can collect any chromium logs.

Yes, please open a new issue for tracking that problem.