Tagging with current canary milestone for further triaging.

Tested on mac os 10.11.6 ,win7 and ubuntu 14.04 using chrome latest stable and latest canary M55 #55.0.2883.75 &M57 #57.0.2944.0 and issue is reproduced.

Issue is seen from M40 and is a non-regression issue . 

Hence , marking it as untriaged.

Thanks !

FYI:
* happening in Chrome 56 and 57 on Mac.
* it doesn't happen in Firefox - it deals fine with that camera.
* it didn't used to happen with a C920, but is now, basically on all high res web cams.
* when you start a getusermedia in Firefox or even facetime before loading the camera in Chrome, the framerate is normal.
* it is not related to forcing a framerate, but forcing a high def video resolution seems to fix it.
* a guess: it came in with recent WebUSB changes.

The report was for  53.0.2785.143, but #4 mentions 56 and 57 (I'm
assuming Canaries) in Mac and Windows, so, if confirmed, it's 
quite a big regression. 

If it's not too much asking, could you try and repro with e.g.
http://simpl.info/getusermedia/constraints/
?  If so, perhaps we can start a bisect.

Also I'd like to see if the stuttering is in the capture itself
(e.g. timestamps) or in the playback (<video> as in WebMediaPlayerMS).

Also, if it repros during a call it would be great to get a screenshot of webrtc-internals video send graphs. Silvia, does it happen all the time for you?

qiangchen@ can you take a look to see if it is related to  issue 660883 ?

It's worth taking a look but it *should* not be related since local rendering doesn't use the video frame timestamps.

Possibly, but it is better for me to have such a camera for further investigation. Where can I borrow one?

hdodda@ can you elaborate on what you actually tested and observed? Since all googlers use Logitech cameras I doubt we wouldn't have heard about this if it repros all the time on all OSes. 

From a user report I just found with Chrome 54.0.2840.87:
"We recently upgraded our MacMini to macOS Sierra. Ever since, we are seeing jerky video feed from our own webcam (Logitech C920 HD) to the video screen when using Chrome"
Switching to Firefox seems to have helped.

Re#10: Not all logitech cameras, but only C920 C930 behave weird.

http://simpl.info/getusermedia/constraints/ shows the problem quite well in 56 and 57 on Mac (El Capitan).
[incidentally, that demo needs fixing: after the first getusermedia, it gets an error: Uncaught TypeError: window.stream.stop is not a function]

Webrtc-internals only shows the constraints and isn't very useful - this happens on a getusermedia already even before feeding into the PeerConnection.

Logitech C920 or C930e both show the problem and these are the most popular Web cams, so it's quite an urgent problem for us. We see it on all our Macs, including MacMini and Macbook. I wasn't able to replicate on windows.

Another aside: hot-plugged web cams don't seem to be recongized by Chrome any more unless I restart the computer :-(.

jansson@ can you give this a try? We just tested with C920 and 930 and we're unable to reproduce.

Just found an old MacMini with chrome 54 (updates constraint by IT support) and it doesn't show the problem. Hmmm...

We've taken the framerates in webrtc-internals now, see attached. It just shows that it feeds the low framerate captured by getusermedia also into the PeerConnection.

This only happens if you don't specify width and height constraints, or if the resolution is roughly below 800x600.

Deleted: Screen Shot 2016-12-16 at 9.56.27 am.png 26.3 KB

	Screen Shot 2016-12-16 at 9.56.27 am.png 26.3 KB View Download

Deleted: Screen Shot 2016-12-16 at 9.56.31 am.png 28.5 KB

	Screen Shot 2016-12-16 at 9.56.31 am.png 28.5 KB View Download

I've just updated my mac to 10.12 and tested on apprtc with forced low resolution constraint, but it looks smooth.

What is the step we can control the constraint on appear.in?

Thanks.

qiangchen: 640x480 (no framerate constraint any longer) is the default. If you want to use a different resolution, try ?lowData (which will give you 320x240 @ 15fps) or ?hd (which will give you 720p)

Using ?hd fixed the issue for a user which confirms what Silvia said in #17

I've tested for both hd and ld case on Mac 10.12, but I cannot reproduce the issue. 

Could that be related to your corporate setting of Mac? Do you have any personal computer to try?

I couldnt reproduce it either on a Macbook Air 2013 running 10.12.2 using C920. I tested both https://simpl.info/getusermedia/constraints/ with all settings and appear.in with lowData/hd. They seem to have stable frame rate in all cases. Is there any way to reproduce this?

Make sure you actually get access to the C920 camera, because current MacOS Sierra (10.12.1) doesn't actually recognize a newly plugged in camera, so I'm having to reboot my computer to test this.

I've also been able to reproduce the issue on my MBP 2013 (El Capitan) with a C910. I wasn't able to reproduce on Linux (Fedora 24)/Chrome 55.

I think I can explain why we don't see the problem at higher resolutions. In media/capture/video/mac/video_capture_device_avfoundation_mac.mm there's a threshold for preferring the use of MJPEG for larger resolutions if it's available. The threshold is 640x480. If we're using MJPEG we remove the AVCaptureStillImageOutput from the capture session. Doing that regardless of codec results in a stable frame rate for my C910.

I've found that by flipping the order that the AVCaptureVideoDataOutput (captureVideoDataOutput_) and AVCaptureStillImageOutput (stillImageOutput_) are initialized and added to the AVCaptureSession in setCaptureDevice, we achieve the same result. i.e. add the AVCaptureStillImageOutput first. I'm not familiar with AV Foundation so I can't explain why that would be the case, but it is inline with the observation that starting a new stream from e.g. firefox fixes the problem in that there's probably some shared state in AV Foundation at the root of the issue.

This works for me anyway. Hopefully not a red herring. Here's the diff of above in case it helps.

diff --git a/media/capture/video/mac/video_capture_device_avfoundation_mac.mm b/media/capture/video/mac/video_capture_device_avfoundation_mac.mm
index 0145616..f57c778 100644
--- a/media/capture/video/mac/video_capture_device_avfoundation_mac.mm
+++ b/media/capture/video/mac/video_capture_device_avfoundation_mac.mm
@@ -260,7 +260,13 @@ - (BOOL)setCaptureDevice:(NSString*)deviceId {
     return NO;
   }
   [captureSession_ addInput:captureDeviceInput_];
-
+
+  // Create and plug the still image capture output. This should happen in
+  // advance of the actual picture to allow for the 3A to stabilize.
+  stillImageOutput_.reset([[AVCaptureStillImageOutput alloc] init]);
+  if (stillImageOutput_ && [captureSession_ canAddOutput:stillImageOutput_])
+    [captureSession_ addOutput:stillImageOutput_];
+
   // Create a new data output for video. The data output is configured to
   // discard late frames by default.
   captureVideoDataOutput_.reset([[AVCaptureVideoDataOutput alloc] init]);
@@ -277,11 +283,6 @@ - (BOOL)setCaptureDevice:(NSString*)deviceId {
                                   DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)];
   [captureSession_ addOutput:captureVideoDataOutput_];

-  // Create and plug the still image capture output. This should happen in
-  // advance of the actual picture to allow for the 3A to stabilize.
-  stillImageOutput_.reset([[AVCaptureStillImageOutput alloc] init]);
-  if (stillImageOutput_ && [captureSession_ canAddOutput:stillImageOutput_])
-    [captureSession_ addOutput:stillImageOutput_];

   return YES;
 }

A quick note that the patch also seems to work on a problematic Mac Mini (10.10.5) with the C930e, and another Mac Mini (10.12.1) running the C920.

Tested on mac os 10.12.1 sierra using the latest chrome canary M57 #57.0.2955.0 and followed steps :

1. Navigated to the url given in comment #5 ., "http://simpl.info/getusermedia/constraints/" .
2. Clicked on the "QVGA, VGA and HD " buttons  and monitored the FPS rate as attached in screencast.

As the FPS rate is fluctuating between 30 to 5 or lesser , confirming the issue.

@could anyone please confirm if bisect is required.

Thanks!

Deleted: 670944_1.mp4 5.2 MB

	670944_1.mp4 5.2 MB View Download

hdodda@: yes, please run a bisect.

briely.marum@coviu.com: thanks for the patch! If you feel
up to contributing to Chromium, I can review that patch for
you ;) -- we just need to make sure the photo parts still 
work fine, e.g. takePhoto() using [1] (with 
chrome://flags/#enable-experimental-web-platform-features on).


[1] https://simpl.info/imagecapture/

Thank #23: I'll take a further look.

I still cannot reproduce the problem. I am making sure that C920 is used as described in #22. After the comments in #23, I tried to disable mjpeg completely to have stuttering but there isn't a resolution that causes it. C920 results in kCVPixelFormatType_422YpCbCr8 for lower resolutions and kCMVideoCodecType_JPEG_OpenDML for higher resolutions and works as expected for me.

I tried your patch that reorders addOutput calls and fixes the issue for you. It doesn't make a difference as far as I tested for both video and image capture, things work with stable frame rate like before. There is no Apple documentation telling that AVCaptureSession::addOutput order matters, but it is worth adding as it doesn't make things worse. I put the patch for review: https://codereview.chromium.org/2585383002/

fippo@, can you verify that this patch fixes your problems as well?

Maybe it's because you're on MacOS 10.12.2 ? Maybe Apple fixed something?

Is this scheduled as a hot patch for current Chrome or just for release in about 12 weeks? (Need to know if we need to put client side temporary fix in.)

I couldn't repro before upgrading to 10.12.2 Sierra either unfortunately. This patch can be be merged to 56 once we have a verification that it works in canary. 56 is scheduled for Jan 31 release.

I assume Silvia means something like 10.12.1, which you never had.

I just reproduced the issue.
I have a Mac Air with MacOS 10.12.2,
I used C920.

I will look further into the details.

I ran tracing during the bug is reproduced. 

The most upstream is the |captureOutput: didOutputSampleBuffer: fromConnection| function, which is a callback we provide to MacOS for receiving the captured data.

Actually, the camera is giving us many frames (on the order of 30FPS), but in a weird manner. Instead of distribute those frames evenly across the time line as normal case does, it suddenly gives us 6 frame (within 1 ms, and from 6 threads), and about 5 such pulses per second. By the later webrtc processing, only the first of those frames within 1ms can survive, others will be dropped, as explains why we observe 5 FPS through webrtc-internal. 

As |captureOutput: didOutputSampleBuffer: fromConnection| is already the most upstream function we have, I cannot know what really happens on the Mac side, and why this function is called in such weird way. 

Fortunately, I've verified codereview.chromium.org/2585383002 works well for both real time capture and snapshoting. Let's take this as a fix.

Deleted: Screen Shot 2016-12-21 at 2.23.05 PM.png 40.6 KB

	Screen Shot 2016-12-21 at 2.23.05 PM.png 40.6 KB View Download

Thanks qiangchen@ for the analysis. I will merge the fix to 56 as well.

Your change meets the bar and is auto-approved for M56 (branch: 2924)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/884ca9d87221f2416730c01c7512606d911fcf78

commit 884ca9d87221f2416730c01c7512606d911fcf78
Author: emircan <emircan@chromium.org>
Date: Thu Dec 22 23:57:31 2016

Merge 56: Fix low frame rate problems in high resolution USB cameras using AVFoundation

This CL addresses the issue reported in the below CL about low frame rate in high
resolution USB cameras using AVFoundation. It reorders output formats for
AVCaptureSession.

All credit for this patch goes to briely.marum@coviu.com, see
https://bugs.chromium.org/p/chromium/issues/detail?id=670944#c23

BUG= 670944 
TEST=Tested video output with C920.

Review-Url: https://codereview.chromium.org/2585383002
Cr-Commit-Position: refs/heads/master@{#439602}
(cherry picked from commit c2df103047f0246c7a909544112c76627e0359e8)
NOTRY=true
NOPRESUBMIT=true
TBR=mcasas@chromium.org

Review-Url: https://codereview.chromium.org/2595223003
Cr-Commit-Position: refs/branch-heads/2924@{#607}
Cr-Branched-From: 3a87aecc31cd1ffe751dd72c04e5a96a1fc8108a-refs/heads/master@{#433059}

[modify] https://crrev.com/884ca9d87221f2416730c01c7512606d911fcf78/media/capture/video/mac/video_capture_device_avfoundation_mac.mm