Currently, when an error is reported from HW decoder, we Destroy the VDA class in Chrome[0]. Than the error is passed onto WebRTC's generic encoder and boils down to nothing[1]. There is no logic in place to reset the encoder.

[0] https://code.google.com/p/chromium/codesearch#chromium/src/content/renderer/media/rtc_video_decoder.cc&rcl=1458143321&l=466
[1] https://code.google.com/p/chromium/codesearch#chromium/src/third_party/webrtc/modules/video_coding/generic_decoder.cc&rcl=1458127931&l=152

Digging down I found this CL that removes Reset() functionality from rtc_video_decoder. Can we bring this back to webrtc::VideoDecoder interface? pbos@ WDYT?
https://codereview.chromium.org/1695583002

Reset() shouldn't be necessary here. A new Decode() attempt from a new keyframe should try to bring the VDA back up (re-initialize?). This recovery should be done inside RTCVideoDecoder I think, and shouldn't require a new interface.

This error boiling down to ~nothing should be fixed when H.264 supports software decode here: https://code.google.com/p/chromium/codesearch#chromium/src/third_party/webrtc/video/video_decoder.cc&sq=package:chromium&type=cs&l=104&rcl=1458143321

I'm not sure how this is wired up at the moment, it should work for VP8 and VP9, but I think we didn't wire it up to H.264 because there was no such decoder in software at that point.

I uploaded a CL trying to solve it within RTCVideoDecoder as pbos@ described. PTAL: https://codereview.chromium.org/1817573004/.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d73415ce0c9e0d6c334612f3c3395b8162093586

commit d73415ce0c9e0d6c334612f3c3395b8162093586
Author: emircan <emircan@chromium.org>
Date: Wed Mar 30 17:44:04 2016

Handle HW decode failures in RTCVideoDecoder

This CL adds functionality to reset underlying HW decoder implementation
after an error happens.

BUG= 594161 
TEST=Tested on Mac using VT HW decoder, injecting error frames in intervals.

Review URL: https://codereview.chromium.org/1817573004

Cr-Commit-Position: refs/heads/master@{#384019}

[modify] https://crrev.com/d73415ce0c9e0d6c334612f3c3395b8162093586/content/renderer/media/rtc_video_decoder.cc
[modify] https://crrev.com/d73415ce0c9e0d6c334612f3c3395b8162093586/content/renderer/media/rtc_video_decoder.h
[modify] https://crrev.com/d73415ce0c9e0d6c334612f3c3395b8162093586/content/renderer/media/rtc_video_decoder_unittest.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d73415ce0c9e0d6c334612f3c3395b8162093586

commit d73415ce0c9e0d6c334612f3c3395b8162093586
Author: emircan <emircan@chromium.org>
Date: Wed Mar 30 17:44:04 2016

Handle HW decode failures in RTCVideoDecoder

This CL adds functionality to reset underlying HW decoder implementation
after an error happens.

BUG= 594161 
TEST=Tested on Mac using VT HW decoder, injecting error frames in intervals.

Review URL: https://codereview.chromium.org/1817573004

Cr-Commit-Position: refs/heads/master@{#384019}

[modify] https://crrev.com/d73415ce0c9e0d6c334612f3c3395b8162093586/content/renderer/media/rtc_video_decoder.cc
[modify] https://crrev.com/d73415ce0c9e0d6c334612f3c3395b8162093586/content/renderer/media/rtc_video_decoder.h
[modify] https://crrev.com/d73415ce0c9e0d6c334612f3c3395b8162093586/content/renderer/media/rtc_video_decoder_unittest.cc

Is this bug fixed now with the above patch?

I will have a followup CL that dynamically resets the error counter, i.e. if we start decoding frames successfully, reset the counter so that periodic failures can be handled as well. Currently, this handles the first 5 errors and falls back to SW.

SGTM :)

Do the tests cover that video actually resumes playing?
I would assume that an effective reset would also have to involve sending an RTCP message to the sender to say "send me an IFrame", but I don't see any such code here.

Has this been tested with a live stream, so that you actually see the recovery happen?

hta@, I tested on a Mac using VideoToolbox decoder: via AppRTC loopback by injecting errors in code, and via Cisco WebEx where we currently have errors happening-3 people session-. It is able to recover from VDA Error multiple times.

I am not sure about the RTCP message since Chrome code doesn't know about it. Currently, we are sending WEBRTC_VIDEO_CODEC_ERROR to WebRTC. That would be handled inside WebRTC's generic_decoder. I just took pbos@'s advice from #2. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e3de5b456a41997ef6026b06bf70801277a58df9

commit e3de5b456a41997ef6026b06bf70801277a58df9
Author: emircan <emircan@chromium.org>
Date: Wed Apr 06 20:23:30 2016

Dynamic error tracking in RTCVideoDecoder

This CL dynamically resets the error counter, i.e. if we
start decoding frames successfully, reset the counter so
that periodic failures can be handled as well.

BUG= 594161 
TEST=Tested on Mac using VT HW decoder, injecting error frames in intervals.

Review URL: https://codereview.chromium.org/1850063003

Cr-Commit-Position: refs/heads/master@{#385544}

[modify] https://crrev.com/e3de5b456a41997ef6026b06bf70801277a58df9/content/renderer/media/rtc_video_decoder.cc
[modify] https://crrev.com/e3de5b456a41997ef6026b06bf70801277a58df9/content/renderer/media/rtc_video_decoder.h
[modify] https://crrev.com/e3de5b456a41997ef6026b06bf70801277a58df9/content/renderer/media/rtc_video_decoder_unittest.cc

emircan@ I'm not too sure what the (manual) steps would be to verify this fix in M51 .. can you please explain?

Yeah this bug only happened in a specific Webex-WebRTC scenario. We have tested that this fix resolves the problem, so I'll mark it as verified.