Moved issue webrtc:8192 to now be  issue chromium:761336 .

Tested the issue using #60.0.3112.116 on Android 6.0.1 - Samsung Galaxy S6 and was able to reproduce the issue as per the steps mentioned in comment #0. 

Observed the same behavior since M59. Hence marking it as Untriaged to get more input's on this.

Please navigate to below link for the logs.

go/chrome-androidlogs/761336

Thanks!!

Philip, can you take a look at this?

For some reason the packet containing the SPS/PPS has its own timestamp.

Got SPS 0 in packet 26187
Got PPS 0 in packet 26187
Got packet 26187 (1329136532)  <-- Note the timestamp differs from the rest of the frame.
Got packet 26188 (1329137052)
Got packet 26189 (1329137052)
Got packet 26190 (1329137052)
Got packet 26191 (1329137052)
Decoding [26188 26191] (key)  <-- Fails because packet 26187 is not part of the frame.

Also, later on when a new keyframe is requested we don't get new a SPS/PPS, only the IDR.

Got packet 26280 (1329197387)  <-- First packet of keyframe, does not contain SPS/PPS
Got packet 26281 (1329197387)
Got packet 26282 (1329197387)
Got packet 26283 (1329197387)
Got packet 26284 (1329197387)
Got packet 26285 (1329197387)
Got packet 26286 (1329197387)
Got packet 26287 (1329197387)
Got packet 26288 (1329197387)
Got packet 26289 (1329197387)
Got packet 26290 (1329197387)
Got packet 26291 (1329197387)
Got packet 26292 (1329197387)
Got packet 26293 (1329197387)
Decoding [26280 26293] (key)

magjed@, do you know if there has been any changes regarding H264 encoding on Android that could explain this?

It seems like a bug that we don't get new SPS/PPS from the encoder when we request a key frame.

There hasn't been any code changes in H264 encoding on Chrome Android that I'm aware of. There might have been a change in what H264 profiles we use though, can someone paste the SDP negotiation here?

@magjed please see attached offer from Chrome on Android and Answer from Chrome on Windows 7.

Tested with:
https://appr.tc/?vrc=H264&vsc=H264

Offer - Chrome Version 60.0.3112.116 Galaxy S6, SM-G920F Build/NRD90M, Android 7.0
Answer - Chrome Version 61.0.3163.79 (Official Build) (64-bit), Windows 7.

Deleted: offer.txt 3.0 KB

offer.txt 3.0 KB View Download

Deleted: answer.txt 3.0 KB

answer.txt 3.0 KB View Download

Same issue:
https://bugs.chromium.org/p/webrtc/issues/detail?id=8234

@magjed I don't think this issue gets the attention it deserves.

H264 is broken between Android and Desktop.
iOS 11 will be released four days from now and it only supports H264.

Sorry, I have been busy.

We negotiate H264 Constrained Baseline.  If Chrome on Linux can't decode it, I guess the problem is with the Android hardware encoder.

What Android device are you using? Maybe it's a problem with the specific device model?

I tested with two Android devices:
Samsung Galaxy S6
Sony Xperia Z5

Do you have an Android device that doesn't have this issue?

Android H264 HW encoder used to work a while back at least, but now it sounds like it's broken. I don't work with this anymore.

Brave - can you take a look?

It still works on Sep 7, as https://bugs.chromium.org/p/webrtc/issues/detail?id=8187#c4 shows. But in later M62 and newer, H264 is not even in the SDP. I track down to https://cs.chromium.org/chromium/src/third_party/webrtc/pc/mediasession.cc?sq=package:chromium&l=1877, where the |filtered_codecs| still has H264 in the list. Then something is wrong at adding content to SDP in mediasession. 

Need more time for bisecting. 

PS: Android H264 HW encoder hasn't been touched for quite a while. So I guess the problem is in the SDP exchanged, such as some parameter mismatching?

@braveyao do you suggest that SDP munging can be a workaround? I don't think that's the case.

This appears as a series of H264 compatibility issues.

First of all, the missing H264 in SDP since Chrome M62:

Since magjed@'s cl, https://codereview.chromium.org/2985263002, the profile level id is set to "profile-level-id=42001f" in Chrome on Android, while it's "profile-level-id=42e01f" from desktop Chrome. Then when doing tests between Chrome Android and Chrome desktops,
if Chrome Android is the caller, Chrome desktop won't add H264 into ANSWER;
if Chrome desktop is the caller, Chrome Android won't add H264 into ANSWER.

Does this mean Chrome Android only supports H264 Baseline and Chrome desktop only supports H264 Constrained Baseline. And they can't talk to each other? Which doesn't sound good..

magjed@, do you know more about this or aware who may know it?

(Besides above issue, there are others. Simply to say, HW H264 is broken in chrome android recently.)

Regarding Baseline vs Constrained Baseline; Chrome is reporting the Android HW codec to be Baseline (media::H264PROFILE_BASELINE https://cs.chromium.org/chromium/src/media/base/video_codecs.h?l=56), and my CL:s is just passing on that information to WebRTC. This is working as intended. It's Chrome's responsibility to advertise all codecs it supports and advertise them with the correct profile. This is an issue that needs to be fixed in Chromium and not WebRTC. The native WebRTC library on Android is working without interop problems. In the native library, we are advertising the Android HW encoder as Constrained Baseline, because that seems to be the bitstream it's producing.

This issue should be owned by someone working in Chromium, so you either you Brave, or emircan@.

- Chrome desktop uses SW for encode and decode. Encode(OpenH264) supports only Constrained_Baseline and decode supports all. In SDP we only offer Constrained_Baseline.
- Chrome Android have HW encode. HW encode is Baseline(+Main +High to be added). In SDP we only offer Baseline.
- Other HW implementations(Mac, Win, CrOS) have both HW and SW encode. In SDP we only offer both Constrained_Baseline and Baseline. However, when we talk to SW only(Linux desktop), Constrained_Baseline is picked always. i.e., I just tested that Linux desktop and Mac H264 call falls back to Sw on Mac.
Therefore, as it is now, we cannot make any match between Linux desktop and Hw implementations. In Android it is visible because there is no SW fallback.

- I don't think the answer is to add Constrained_Baseline as a supported profile in Hw encode in Chrome. That would be basically lying because we support Baseline. 
- A good solution would be to refactor SDP so that you can distinguish profiles you can encode vs decode, but I am not even sure if it is possible with current spec.
- Is there any way we can change the profile matching logic such that each side does whatever the highest profile it can. We would assume that decode is present for every profile on both sides. Or, allow matching Constrained_Baseline with Baseline like before. magjed@ WDYT?

Some correctness: if Mac (or other HW implementations) is the caller, it will offer Baseline. And Linux will omit H264 in the ANSWER due to profile mismatching and video call will be only established with VPx codec.
Which means NO H264 video call is available from HW implementation Chrome to SW implementation Chrome.

> - I don't think the answer is to add Constrained_Baseline
> as a supported profile in Hw encode in Chrome. That would
> be basically lying because we support Baseline.
I have talked with Qualcomm about what their HW encoders produces, and it does not produce any bitstream which is outside of CBP tool set. That is why we are negotiating Constrained Baseline for the HW codecs in the Android native library. It's not lying, even though MediaCodec documentation says Baseline.

> - A good solution would be to refactor SDP so that you can
> distinguish profiles you can encode vs decode, but I am not
> even sure if it is possible with current spec.
That's not possible. We use sendrecv codecs which means we must be able to both send and recv it. See https://bugs.chromium.org/p/webrtc/issues/detail?id=5187 for more info.

> - Is there any way we can change the profile matching logic such
> that each side does whatever the highest profile it can. We would
> assume that decode is present for every profile on both sides. Or,
> allow matching Constrained_Baseline with Baseline like before.
What we should do is this; whenever we can encode Constrained Baseline, which is the case for the SW encoder (and e.g. Qualcomm HW encoder), and can decode Baseline, Constrained Baseline, Main, and High profile, we should negotiate all of Baseline, Constrained Baseline, Main, and High profile. The reason this works and is according to the spec, is because even if we negotiate e.g. H.264 High profile, it's legal to send with H.264 Constrained Baseline, because that's a true subset of High. So Chrome on desktops should support all these different profiles.

Per the H264 RFC 6184, you can't match profiles with different constraints outside of a moderately confusing list of "X with constraint Y is the same as A with constraint B".  See https://tools.ietf.org/html/rfc6184#section-8.1 table 5.

If you want to handle multiple profiles (beyond the limited matching in table 5), you have to offer multiple profiles (on different payloads).  When answering, you need to examine the offered profiles and match against what you can send (and receive) -- not just against the highest profile you're capable of.  I.e. when processing a CBP request, if your encoder supports High profile, don't reject because the offer isn't High; accept CBP and configure the encoder to do CBP.  If you're not doing this, it's on you - this is how you're supposed to do offer/answer for H264.  You can reject it if you like, but don't.

If offering, and you do High profile, offer payloads with lower profiles too for best results, and (for interactive video) at least also offer CBP, which is pretty much the standard for interactive and the lowest common denominator.

> I have talked with Qualcomm about what their HW encoders produces, and it does not produce any bitstream which is outside of 
> CBP tool set. That is why we are negotiating Constrained Baseline for the HW codecs in the Android native library. It's not 
> lying, even though MediaCodec documentation says Baseline.
I didn't know about this case in Android. braveyao@ can you take a look at this and change the supported profile in Android from
Baseline to Constrained_Baseline then?
However in Mac and Win, even if we say we support both Constrained_Baseline and Baseline, underlying implementations would be the same as there is only Baseline is available in their APIs. That's what I meant by lying workaround.  

> What we should do is this; whenever we can encode Constrained Baseline, which is the case for the SW encoder 
> (and e.g. Qualcomm HW encoder), and can decode Baseline, Constrained Baseline, Main, and High profile, we should negotiate all 
> of Baseline, Constrained Baseline, Main, and High profile. The reason this works and is according to the spec, is because even 
> if we negotiate e.g. H.264 High profile, it's legal to send with H.264 Constrained Baseline, because that's a true subset of High. 
> So Chrome on desktops should support all these different profiles.
This sounds good, then each side encodes whatever the highest profile it can. Can you change this in webrtc side? I will get some HW
and test&add other profiles supported soon.

rjesup@ - Yes, you are right, and this is what I propose Chromium should do; each profile should be negotiated in a separate payload type.

emircan@ - Have you looked into what bitstreams the HW encoders on Win and Mac actually produce? It's sad if we can't offer CBP, because it's the most compatible profile.

> emircan@ - Have you looked into what bitstreams the HW encoders on Win and Mac actually produce? 
> It's sad if we can't offer CBP, because it's the most compatible profile.
I double checked the SPS profile output today. 
- On Mac, we get |profile_idc:66 constraint_set0_flag:0 constraint_set1_flag:0| from SPS and there seems to be no documented API to get CBP. 
- On Win, Baseline profile has the same output as Mac above. Although not documented, I tried plugging in eAVEncH264VProfile_ConstrainedBase as MF_MT_MPEG2_PROFILE and it worked; it had SPS output |profile_idc:66 constraint_set0_flag:1 constraint_set1_flag:1|. We can add CBP as another supported profile there with some checks.
- I sent the SPS parser patch to braveyao@ to check Android output.
[0] https://msdn.microsoft.com/en-us/library/windows/desktop/dd318776(v=vs.85).aspx

> rjesup@ - Yes, you are right, and this is what I propose Chromium should do; each profile should be 
> negotiated in a separate payload type.
From what I understand, we agree that each profile should be separately negotiated. The question here is to pick the lowest common profile or the highest in negotiation. AFAICT rjesup@ suggests picking the lowest and configuring the higher encoder to do the lowest. But then comes issues: we need to always have CBP in Android(although OpenH264 isn't available) that caused this bug and falling to OpenH264 CBP in most cases in Mac. 

If we negotiate each profile separately in different payloads, we just need to order the list of supported profiles in order of preference, i.e. High, Main, etc., and that should be enough for negotiation. WebRTC takes care of the matching. When e.g. the profile High is negotiated and and sent to the SW encoder, it's ok for the encoder to produce CBP.

Any update?

Yes, see https://bugs.chromium.org/p/webrtc/issues/detail?id=8317 for details. We added support for Baseline using the SW codecs, so interop with Android Chrome should work again.

There are two rendering issue fixed at Android side. Now VPx is back to work.
The cl to fix the H264 SPS/PPS from HW H264 on Android is uploaded for review. Plus the working for webrtc::8317, H264 interop will be back soon.

@braveyao Do you have bug or commit ids for the android rendering issues that fixed VPx? Thanks.

https://bugs.chromium.org/p/chromium/issues/detail?id=772899 has all the info inside.

@braveyao thanks. Any chance of the h264 related fixes making it to M62?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d1eba968e852c442c23ed0ac8f31ee878f2e1f05

commit d1eba968e852c442c23ed0ac8f31ee878f2e1f05
Author: Weiyong Yao <braveyao@chromium.org>
Date: Fri Oct 20 16:28:25 2017

Android HW H264: append SPS/PPS at the start of key frame

Android MediaCodec will generate a seperate config frame containing
SPS/PPS at beginning of encoding H264 stream and the following key
frames won't have SPS/PPS any more, while other SW/HW H264 encoder
implementations will generate key frames with SPS/PPS always.
Recently WebRTC requires the H264 IDR/keyframe to contain SPS/PPS to
start decoder. Then the H264 intercommunication is broken between
Android and other platforms.

To fix this, append SPS/PPS at the start of H264 keyframe on Android.

Bug:  761336 
Change-Id: Ifb6b27f4448186dc4d7411921c556cf57d74068e
Reviewed-on: https://chromium-review.googlesource.com/710454
Reviewed-by: Frank Liberato <liberato@chromium.org>
Commit-Queue: Weiyong Yao <braveyao@chromium.org>
Cr-Commit-Position: refs/heads/master@{#510459}
[modify] https://crrev.com/d1eba968e852c442c23ed0ac8f31ee878f2e1f05/media/base/android/BUILD.gn
[modify] https://crrev.com/d1eba968e852c442c23ed0ac8f31ee878f2e1f05/media/base/android/java/src/org/chromium/media/MediaCodecBridge.java
[add] https://crrev.com/d1eba968e852c442c23ed0ac8f31ee878f2e1f05/media/base/android/java/src/org/chromium/media/MediaCodecEncoder.java

Will request merge back to M63 next week.

Your change meets the bar and is auto-approved for M63. Please go ahead and merge the CL to branch 3239 manually. Please contact milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), gkihumba@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/47a3feab3a614dd63b6e8c4767a347e98673f595

commit 47a3feab3a614dd63b6e8c4767a347e98673f595
Author: Weiyong Yao <braveyao@chromium.org>
Date: Wed Oct 25 17:14:34 2017

Android HW H264: append SPS/PPS at the start of key frame

Android MediaCodec will generate a seperate config frame containing
SPS/PPS at beginning of encoding H264 stream and the following key
frames won't have SPS/PPS any more, while other SW/HW H264 encoder
implementations will generate key frames with SPS/PPS always.
Recently WebRTC requires the H264 IDR/keyframe to contain SPS/PPS to
start decoder. Then the H264 intercommunication is broken between
Android and other platforms.

To fix this, append SPS/PPS at the start of H264 keyframe on Android.

TBR=braveyao@chromium.org

(cherry picked from commit d1eba968e852c442c23ed0ac8f31ee878f2e1f05)

Bug:  761336 
Change-Id: Ifb6b27f4448186dc4d7411921c556cf57d74068e
Reviewed-on: https://chromium-review.googlesource.com/710454
Reviewed-by: Frank Liberato <liberato@chromium.org>
Commit-Queue: Weiyong Yao <braveyao@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#510459}
Reviewed-on: https://chromium-review.googlesource.com/737545
Reviewed-by: Weiyong Yao <braveyao@chromium.org>
Cr-Commit-Position: refs/branch-heads/3239@{#221}
Cr-Branched-From: adb61db19020ed8ecee5e91b1a0ea4c924ae2988-refs/heads/master@{#508578}
[modify] https://crrev.com/47a3feab3a614dd63b6e8c4767a347e98673f595/media/base/android/BUILD.gn
[modify] https://crrev.com/47a3feab3a614dd63b6e8c4767a347e98673f595/media/base/android/java/src/org/chromium/media/MediaCodecBridge.java
[add] https://crrev.com/47a3feab3a614dd63b6e8c4767a347e98673f595/media/base/android/java/src/org/chromium/media/MediaCodecEncoder.java

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fc65df9ad4c8d96de5e22b90e22c62895a2c78af

commit fc65df9ad4c8d96de5e22b90e22c62895a2c78af
Author: Weiyong Yao <braveyao@chromium.org>
Date: Thu Oct 26 17:27:18 2017

[Android] exclude a DCHECK for HW encoder

Capture timestamps is not available on Android at present. So the DCHECK
will be hitted and stop Dbg bulding from running when HW codec is selected.

Bug:  761336 
Change-Id: Ic00cc2ed57f10a523066478b35f0158a3363a530
Reviewed-on: https://chromium-review.googlesource.com/738644
Reviewed-by: Emircan Uysaler <emircan@chromium.org>
Commit-Queue: Weiyong Yao <braveyao@chromium.org>
Cr-Commit-Position: refs/heads/master@{#511861}
[modify] https://crrev.com/fc65df9ad4c8d96de5e22b90e22c62895a2c78af/content/renderer/media/gpu/rtc_video_encoder.cc

magjed@ Chrome 64 Dev on Samsung Android tablet has only Base profile in the SDP: profile-level-id=42001f.
It should also have Constrained Baseline profile-level-id=42e01f?

When using SFU, I need to mangle the SDP in Android to make it view Chrome desktop stream:
msg.sdp = msg.sdp.replace('42e01f', '42001f');

When Android is both receiving and sending a video stream, the Android stream freezes.
When Android is only sending a video stream and Chrome on desktop refresh the page, the stream freeze.

Is there a chance to get Constrained Baseline work on all OS as it used to be few months ago?

Android does not ship with SW H264, so it will only have the HW codecs, which Chrome reports as Baseline. If Android <-> Android H264 does not work, it sounds like something is wrong with HW encoding/decoding.

Chrome on Android used to advertise CBP (42e01f).
We had all browsers on all OS working with our SFU (Chrome/Firefox/Edge/Safari on Windows, Android, Linux, Mac, iOS).
Chrome on Android changed something around version 60 and broke interop.
Are you saying that Chrome on Android won't follow the WebRTC standard and won't support CBP?

There are issues with Chrome Windows <-> Chrome Android H264 with multiple video tracks and when changing tracks dynamically.

The codec Android advertises is the HW codec. This has always been the case. The Android HW codec has always been reported as H264PROFILE_BASELINE in Chrome 
(there is not even the notion of CPB in Chrome https://cs.chromium.org/chromium/src/media/base/video_codecs.h).

Chrome used to map all it's H264 codecs from media::VideoCodecProfile to webrtc::Profile::kProfileConstrainedBaseline. I.e. it would map media::VideoCodecProfile::H264PROFILE_HIGH -> webrtc::Profile::kProfileConstrainedBaseline, media::VideoCodecProfile::H264PROFILE_BASELINE -> webrtc::Profile::kProfileConstrainedBaseline, etc.

This made it impossible to add support for anything other than CBP, and it obviously violates the spec. People have been requesting High profile support for a long time, so this mapping has now been fixed so that e.g. media::VideoCodecProfile::H264PROFILE_HIGH is mapped to webrtc::Profile::kProfileHigh and media::VideoCodecProfile::H264PROFILE_BASELINE is mapped to webrtc::Profile::kProfileBaseline. This is where the change you see in SDP comes from.

I work with WebRTC and not Chrome, so I can't answer the other questions. If the Android HW encoder/decoder has problems, or we need support for SW CBP on Android, it's up to the Chrome team. braveyao@ is the assigned owner of this issue.

The H264 profile negotiated is meaningless to Android at present. No matter what profile is negotiated, MediaCodec will encode H264 in its own way. So just munge the SDP to get the negotiation done, i.e. force it to be BP or CBP bidirectionally.

 Issue 773345  has been merged into this issue.