WebRtc* content_browsertests are slow |
|||||
Issue descriptionIn any normal rel bot (e.g. [1], but any), content_browsertests takes a long time and WebRtc* tests figure prominently in the top-10 (3 are WebRtc-prefixed) and top-100 (48 are WebRtc-prefixed). (For reference, there are 204 WebRtc-prefixed tests, taking each an average of 1.5 seconds. These tests run on CQ and are too slow. E.g. Top10: *WebRtcPausePlayBrowserTest.SurvivesPeerConnectionPausePlaying 11552 SitePerProcessBrowserTest.ScrollBubblingFromOOPIFTest 10752 SnapshotBrowserTest.SyncMultiWindowTest 9698 SnapshotBrowserTest.AsyncMultiWindowTest 9175 MediaSessionImplVisibilityBrowserTestInstances/MediaSessionImplVisibilityBrowserTest.TestEntryPoint/1 6814 *WebRtcBrowserTest.CanSetupCallAndSendDtmf 5769 *WebRtcGetUserMediaBrowserTest.GetUserMediaWithInvalidOptionalSourceID 5503 File/MediaTest.VideoTulipWebm/0 4978 Http/MediaTest.VideoTulipWebm/0 4978 SnapshotBrowserTest.SingleWindowTest 4454 *WebRtcBrowserTest.CallAndVerifyVideoMutingWorks 3672 Note that the first one takes > 10 s to execute (!!) [1] https://ci.chromium.org/buildbot/tryserver.chromium.linux/linux_chromium_rel_ng/609961 Top 100: WebRtcPausePlayBrowserTest.SurvivesPeerConnectionPausePlaying 11552 SitePerProcessBrowserTest.ScrollBubblingFromOOPIFTest 10752 SnapshotBrowserTest.SyncMultiWindowTest 9698 SnapshotBrowserTest.AsyncMultiWindowTest 9175 MediaSessionImplVisibilityBrowserTestInstances/MediaSessionImplVisibilityBrowserTest.TestEntryPoint/1 6814 WebRtcBrowserTest.CanSetupCallAndSendDtmf 5769 WebRtcGetUserMediaBrowserTest.GetUserMediaWithInvalidOptionalSourceID 5503 File/MediaTest.VideoTulipWebm/0 4978 Http/MediaTest.VideoTulipWebm/0 4978 SnapshotBrowserTest.SingleWindowTest 4454 WebRtcBrowserTest.CallAndVerifyVideoMutingWorks 3672 SRC_ClearKey/EncryptedMediaTest.Playback_AudioOnly_WebM/0 3410 SRC_ClearKey/EncryptedMediaTest.Playback_VideoAudio_WebM/0 3406 WebRtcBrowserTest.CanSetupDefaultVideoCall 3405 WebRtcBrowserTest.CanSetupVideoCallAndDisableLocalVideo 3405 WebRtcBrowserTest.NoCrashWhenConnectChromiumSinkToRemoteTrack 3405 WebRtcGetUserMediaBrowserTest.TwoGetUserMediaAndStop 3405 SRC_ClearKey/EncryptedMediaTest.Playback_AudioClearVideo_WebM/0 3405 SRC_ClearKey/EncryptedMediaTest.Playback_VideoClearAudio_WebM/0 3405 SRC_ClearKey/EncryptedMediaTest.Playback_VideoOnly_WebM_Fullsample/0 3146 MSE_ClearKey/EncryptedMediaTest.Playback_AudioOnly_WebM/0 3146 WebRtcGetUserMediaBrowserTest.GetVideoStreamAndStop 3145 SRC_ClearKey/EncryptedMediaTest.Playback_VideoOnly_WebM/0 3145 WebRtcBrowserTest.GetSettingsWhenRemoteDimensionsUnknown 3144 WebRtcBrowserTest.CanSetupAudioAndVideoCall 3143 SRC_ClearKey/EncryptedMediaTest.Playback_VideoOnly_WebM_Subsample/0 3143 MSE_ClearKey/EncryptedMediaTest.Playback_AudioClearVideo_WebM/0 3143 Http/MediaTest.VideoBearOpusWebm/0 3143 Http/MediaTest.AudioBearFlac192kHzMp4/0 3143 Http/MediaTest.VideoBearHighBitDepthMp4/0 3143 WebRtcBrowserTest.CanSetupDefaultVideoCallWithOldGetUserMedia 3142 WebRtcBrowserTest.CanSetupVideoCallWith1To1AspectRatio 3142 SRC_ClearKey/EncryptedMediaTest.Playback_VideoAudio_WebM_Opus/0 3142 SRC_ClearKey/EncryptedMediaTest.Playback_VideoClearAudio_WebM_Opus/0 3142 MSE_ClearKey/EncryptedMediaTest.Playback_VideoAudio_WebM/0 3142 MSE_ClearKey/EncryptedMediaTest.Playback_VideoOnly_WebM/0 3142 MSE_ClearKey/EncryptedMediaTest.Playback_VideoOnly_WebM_Fullsample/0 3142 MSE_ClearKey/EncryptedMediaTest.Playback_VideoOnly_WebM_Subsample/0 3142 MSE_ClearKey/EncryptedMediaTest.Playback_VideoClearAudio_WebM/0 3142 MSE_ClearKey/EncryptedMediaTest.Playback_AudioOnly_WebM_Opus/0 3142 MSE_ClearKey/EncryptedMediaTest.Playback_VideoAudio_WebM_Opus/0 3142 MSE_ClearKey/EncryptedMediaTest.Playback_VideoClearAudio_WebM_Opus/0 3142 File/MediaTest.VideoBearOpusWebm/0 3142 File/MediaTest.AudioBearFlacMp4/0 3142 File/MediaTest.AudioBearFlac192kHzMp4/0 3142 File/MediaTest.VideoBearHighBitDepthMp4/0 3142 Http/MediaTest.AudioBearFlacMp4/0 3142 WebRtcDataBrowserTest.CallWithSctpDataAndMedia 2887 WebRtcBrowserTest.CanForwardRemoteStream 2886 WebRtcBrowserTest.CallWithNewVideoMediaStream 2886 SitePerProcessBrowserTest.HiddenOOPIFWillNotGenerateCompositorFrames 2883 WebRtcBrowserTest.CanSetupVideoCallWith4To3AspectRatio 2882 WebRtcDataBrowserTest.CallWithDataAndMedia 2882 WebRtcBrowserTest.CanMakeAudioCallAndThenRenegotiateToVideo 2881 WebRtcBrowserTest.CallAndModifyStream 2881 Http/MediaTest.VideoBearMp4Vp9/0 2881 WebRtcBrowserTest.CanSetupVideoCallWith16To9AspectRatio 2880 WebRtcBrowserTest.CanSetupAudioAndVideoCallWithoutMsidAndBundle 2880 WebRtcBrowserTest.NegotiateOfferWithBLine 2880 WebRtcMediaRecorderTest.PausePreventsDataavailableFromBeingFired 2880 File/MediaTest.VideoBearOpusOgg/0 2880 UserMedia/WebRtcConstraintsBrowserTest.GetUserMediaConstraints/5 2880 WebRtcBrowserTest.CallInsideIframe 2629 WebRtcGetUserMediaBrowserTest.GetAudioStreamAndCheckMutingInitiallyUnmuted 2624 WebRtcDataBrowserTest.CanSetupLegacyCall 2623 WebRtcGetUserMediaBrowserTest.RenderSameTrackMediastreamAndStop 2621 UserMedia/WebRtcConstraintsBrowserTest.GetUserMediaConstraints/0 2621 WebRtcGetUserMediaBrowserTest.TwoGetUserMediaWithFirstHdSecondVga 2620 WebRtcGetUserMediaBrowserTest.TestGetUserMediaAspectRatio1To1 2620 WebRtcGetUserMediaBrowserTest.GetAudioAndVideoStreamAndStop 2619 Http/MediaTest.AudioBearFlacOgg/0 2619 UserMedia/WebRtcConstraintsBrowserTest.GetUserMediaConstraints/3 2619 ServiceWorkerVersionBrowserTest.TimeoutWorkerInEvent 2618 WebRtcBrowserTest.CanMakeEmptyCallThenAddStreamsAndRenegotiate 2618 WebRtcGetUserMediaBrowserTest.GetAudioStreamAndCheckMutingInitiallyMuted 2618 File/MediaTest.VideoBearHighBitDepthVP9/0 2618 File/MediaTest.VideoBear12DepthVP9/0 2618 File/MediaTest.VideoBearMp4Vp9/0 2618 File/MediaTest.AudioBearFlacOgg/0 2618 Http/MediaTest.VideoBearHighBitDepthVP9/0 2618 Http/MediaTest.VideoBear12DepthVP9/0 2618 UserMedia/WebRtcConstraintsBrowserTest.GetUserMediaConstraints/1 2618 UserMedia/WebRtcConstraintsBrowserTest.GetUserMediaConstraints/2 2618 GLAndSoftwareCompositing/CompositingRenderWidgetHostViewBrowserTestTabCaptureHighDPI.CopyToVideoFrame_ScaledResult/0 2371 WebRtcMediaRecorderTest.StartWithTimeSlice/4 2359 WebRtcGetUserMediaBrowserTest.TwoGetUserMediaAndCheckCallbackAfterFailure 2358 Http/MediaTest.VideoBearWavMulaw/0 2358 GLAndSoftwareCompositing/CompositingRenderWidgetHostViewBrowserTestTabCapture.CopyFromSurface_ForVideoFrame/0 2358 GLAndSoftwareCompositing/CompositingRenderWidgetHostViewBrowserTestTabCapture.CopyFromSurface_ForVideoFrame_Scaled/0 2358 WebRtcGetUserMediaBrowserTest.TwoGetUserMediaWithEqualConstraints 2357 GLAndSoftwareCompositing/CompositingRenderWidgetHostViewBrowserTestTabCaptureHighDPI.CopyToVideoFrame_EntireRegion/0 2357 ClearSiteDataThrottleBrowserTest.CacheIntegrationTest 2356 WebRtcGetUserMediaBrowserTest.RenderClonedMediastreamAndStop 2356 WebRtcGetUserMediaBrowserTest.TwoGetUserMediaWithSecondVideoCropped 2356 WebRtcGetUserMediaBrowserTest.TestGetUserMediaAspectRatio16To9 2356 WebRtcMediaRecorderTest.StartWithTimeSlice/0 2356 WebRtcMediaRecorderTest.StartWithTimeSlice/2 2356 WebRtcMediaRecorderTest.StartWithTimeSlice/3 2356 WebRtcMediaRecorderTest.StartWithTimeSlice/5 2356
,
Dec 15 2017
+ guidou@, phoglund@ as per OWNERs guidou@: Could you please take a look or delegate?
,
Dec 18 2017
Who decides what too slow is? WebRTC generally does need to play media for a while for us to determine that everything is working, so it's different than loading a webpage. Sure, we can look at making them faster, but we need something more specific to aim for than "make them faster".
,
Dec 18 2017
Also, the test that takes 10 seconds is a kind of stress test that we just added. There are a few more of those coming in. We have talked about maybe running it in our performance waterfall (so we can run it for even longer), so we can at least take that one off the CQ.
,
Dec 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/4d396a4e640a6e476f687aa7a71b05ac78cf3917 commit 4d396a4e640a6e476f687aa7a71b05ac78cf3917 Author: Kristoffer Erlandsson <kerl@google.com> Date: Mon Dec 18 18:37:23 2017 Change test to MANUAL. We aim to create a step to run this on the WebRTC bots instead. BUG=chromium:795422 Change-Id: I1a462ff75c3242b64acf51c9e40e466ad22f6606 Reviewed-on: https://chromium-review.googlesource.com/831950 Reviewed-by: Patrik Höglund <phoglund@chromium.org> Commit-Queue: Kristoffer Erlandsson <kerl@google.com> Cr-Commit-Position: refs/heads/master@{#524749} [modify] https://crrev.com/4d396a4e640a6e476f687aa7a71b05ac78cf3917/content/browser/webrtc/webrtc_pause_play_browsertest.cc
,
Dec 19 2017
kerl: actually, as it turns out, your tests already run under content_browsertests_manual on our bots (https://build.chromium.org/deprecated/chromium.webrtc/builders/Linux%20Tester/builds/33909). They are configured to run all WebRTC* manual content_browsertests. That takes the slowest tests out of the CQ. I'll leave making the rest of the tests faster to you, Guido.
,
Jan 16 2018
As this issue is related to Infra>Flakiness, unable to test and confirm this wrt added Needs-TestConfirmation label. Hence adding TE-NeedsTriageHelp for help in triaging this further. Thanks!
,
Jan 16 2018
I can report that the stress tests are all moved now, so SurvivesPeerConnectionPausePlaying has moved out.
,
Jan 23 2018
,
Jan 23 2018
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by mcasas@chromium.org
, Dec 15 2017