New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 909704 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Dec 20
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux , Chrome
Pri: 2
Type: Bug
Flaky-Test: WebRtcDisableEncryptionFlagBrowserTest.VerifyEncryption



Sign in to add a comment

WebRtcDisableEncryptionFlagBrowserTest.VerifyEncryption is flaky

Project Member Reported by Findit, Nov 28

Issue description


Flaky test: WebRtcDisableEncryptionFlagBrowserTest.VerifyEncryption
Sample failed build due to flakiness: https://ci.chromium.org/buildbot/chromium.memory/Linux%20ASan%20LSan%20Tests%20%281%29/53190
Test output log: https://chromium-swarm.appspot.com/task?id=417150cfe9c4eb10
Culprit (70.0% confidence): r611174
Analysis: https://findit-for-me.appspot.com/waterfall/flake?key=ag9zfmZpbmRpdC1mb3ItbWVywQELEhdNYXN0ZXJGbGFrZUFuYWx5c2lzUm9vdCKKAWNocm9taXVtLm1lbW9yeS9MaW51eCBBU2FuIExTYW4gVGVzdHMgKDEpLzUzMTkwL2Jyb3dzZXJfdGVzdHMvVjJWaVVuUmpSR2x6WVdKc1pVVnVZM0o1Y0hScGIyNUdiR0ZuUW5KdmQzTmxjbFJsYzNRdVZtVnlhV1o1Ulc1amNubHdkR2x2Ymc9PQwLEhNNYXN0ZXJGbGFrZUFuYWx5c2lzGAEM

Please revert the culprit, or disable the test and find the appropriate owner.

If the culprit above is wrong, please file a bug using this link:
https://bugs.chromium.org/p/chromium/issues/entry?status=Unconfirmed&labels=Pri-1,Test-Findit-Wrong&components=Tools%3ETest%3EFindit%3EFlakiness&summary=%5BFindit%5D%20Flake%20Analyzer%20-%20Wrong%20result%20for%20WebRtcDisableEncryptionFlagBrowserTest.VerifyEncryption&comment=Link%20to%20Analysis%3A%20https://findit-for-me.appspot.com/waterfall/flake?key=ag9zfmZpbmRpdC1mb3ItbWVywQELEhdNYXN0ZXJGbGFrZUFuYWx5c2lzUm9vdCKKAWNocm9taXVtLm1lbW9yeS9MaW51eCBBU2FuIExTYW4gVGVzdHMgKDEpLzUzMTkwL2Jyb3dzZXJfdGVzdHMvVjJWaVVuUmpSR2x6WVdKc1pVVnVZM0o1Y0hScGIyNUdiR0ZuUW5KdmQzTmxjbFJsYzNRdVZtVnlhV1o1Ulc1amNubHdkR2x2Ymc9PQwLEhNNYXN0ZXJGbGFrZUFuYWx5c2lzGAEM

Automatically posted by the findit-for-me app (https://goo.gl/Ot9f7N).
 
Components: Blink>WebRTC
Project Member

Comment 2 by bugdroid1@chromium.org, Nov 28

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a1185fcb981b2317bdc707990b28a065be2ddb38

commit a1185fcb981b2317bdc707990b28a065be2ddb38
Author: Ned Nguyen <nednguyen@google.com>
Date: Wed Nov 28 16:17:44 2018

Revert "Align resource timing buffer full processing to spec PR 168 (reland)"

This reverts commit 6dd060768d1c1183e0d29e422a2213f106d98299.

Reason for revert: suspect causing  WebRtcDisableEncryptionFlagBrowserTest.VerifyEncryption to be flaky (FindIt's report)

BUG: chromium:909704 

Original change's description:
> Align resource timing buffer full processing to spec PR 168 (reland)
> 
> This change implements the processing model from PR 168[1], when
> it comes to setResourceTimingBufferSize(), clearResourceTimings()
> and the firing of the resourcetimingbufferfull event.
> 
> This is a reland of
> https://chromium-review.googlesource.com/c/chromium/src/+/1345269
> (but with nicer tests).
> 
> [1] https://github.com/w3c/resource-timing/pull/168
> 
> Change-Id: I10431bfda7f79b484c7ee5c608bb8a360d905339
> Bug:  908181 ,  908414 
> Reviewed-on: https://chromium-review.googlesource.com/c/1350950
> Reviewed-by: Nicolás Peña Moreno <npm@chromium.org>
> Commit-Queue: Yoav Weiss <yoavweiss@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#611174}

TBR=npm@chromium.org,yoavweiss@chromium.org

Change-Id: Ia4bf26c6be1e7c4ae35b904bac25fe031f854604
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  908181 ,  908414 
Reviewed-on: https://chromium-review.googlesource.com/c/1354123
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#611722}
[modify] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/renderer/core/timing/performance.cc
[modify] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/renderer/core/timing/performance.h
[delete] https://crrev.com/617a8563d6b9a1bee6113d8ba35e4806f99593ac/third_party/blink/web_tests/external/wpt/resource-timing/buffer-full-add-after-full-event.html
[delete] https://crrev.com/617a8563d6b9a1bee6113d8ba35e4806f99593ac/third_party/blink/web_tests/external/wpt/resource-timing/buffer-full-add-entries-during-callback-that-drop.html
[delete] https://crrev.com/617a8563d6b9a1bee6113d8ba35e4806f99593ac/third_party/blink/web_tests/external/wpt/resource-timing/buffer-full-add-entries-during-callback.html
[delete] https://crrev.com/617a8563d6b9a1bee6113d8ba35e4806f99593ac/third_party/blink/web_tests/external/wpt/resource-timing/buffer-full-add-then-clear.html
[delete] https://crrev.com/617a8563d6b9a1bee6113d8ba35e4806f99593ac/third_party/blink/web_tests/external/wpt/resource-timing/buffer-full-decrease-buffer-during-callback.html
[delete] https://crrev.com/617a8563d6b9a1bee6113d8ba35e4806f99593ac/third_party/blink/web_tests/external/wpt/resource-timing/buffer-full-increase-buffer-during-callback.html
[delete] https://crrev.com/617a8563d6b9a1bee6113d8ba35e4806f99593ac/third_party/blink/web_tests/external/wpt/resource-timing/buffer-full-inspect-buffer-during-callback.html
[delete] https://crrev.com/617a8563d6b9a1bee6113d8ba35e4806f99593ac/third_party/blink/web_tests/external/wpt/resource-timing/buffer-full-set-to-current-buffer.html
[delete] https://crrev.com/617a8563d6b9a1bee6113d8ba35e4806f99593ac/third_party/blink/web_tests/external/wpt/resource-timing/buffer-full-store-and-clear-during-callback.html
[delete] https://crrev.com/617a8563d6b9a1bee6113d8ba35e4806f99593ac/third_party/blink/web_tests/external/wpt/resource-timing/buffer-full-then-increased.html
[delete] https://crrev.com/617a8563d6b9a1bee6113d8ba35e4806f99593ac/third_party/blink/web_tests/external/wpt/resource-timing/buffer-full-when-populate-entries.html
[add] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/web_tests/external/wpt/resource-timing/resource_timing_buffer_full_when_populate_entries.html
[add] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/web_tests/external/wpt/resource-timing/resource_timing_buffer_full_when_shrink_buffer_size-expected.txt
[add] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/web_tests/external/wpt/resource-timing/resource_timing_store_and_clear_during_callback-expected.txt
[add] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/web_tests/external/wpt/resource-timing/resource_timing_store_and_clear_during_callback.html
[delete] https://crrev.com/617a8563d6b9a1bee6113d8ba35e4806f99593ac/third_party/blink/web_tests/external/wpt/resource-timing/resources/buffer-full-utilities.js
[modify] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/web_tests/external/wpt/service-workers/service-worker/resources/performance-timeline-worker.js
[add] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/web_tests/http/tests/misc/cache-hit-resource-timing-buffer-full-crash-expected.txt
[add] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/web_tests/http/tests/misc/cache-hit-resource-timing-buffer-full-crash.html
[add] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/web_tests/http/tests/misc/finish-load-resource-timing-buffer-full-crash-expected.txt
[add] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/web_tests/http/tests/misc/finish-load-resource-timing-buffer-full-crash.html
[add] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/web_tests/http/tests/misc/stop-loading-on-resource-timing-buffer-full-crash-expected.txt
[add] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/web_tests/http/tests/misc/stop-loading-on-resource-timing-buffer-full-crash.html
[add] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/web_tests/http/tests/misc/webtiming-buffer-full-no-event.html
[modify] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/web_tests/http/tests/performance-timing/resource_timing_buffer_full_250.html
[modify] https://crrev.com/a1185fcb981b2317bdc707990b28a065be2ddb38/third_party/blink/web_tests/http/tests/workers/resources/performance-timeline-worker.js

Owner: grunell@chromium.org
Assign this to grunell@ (test author) in case the revert doesn't fix flakiness
Status: Fixed (was: Untriaged)

Comment 5 Deleted

 Issue 910216  has been merged into this issue.
Components: -Blink>WebRTC Blink>WebRTC>Network
Labels: -Pri-1 OS-Chrome OS-Linux Pri-2
Status: Assigned (was: Fixed)
Labels: -Sheriff-Chromium
Flakiness seems to have improved:
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=WebRtcDisableEncryptionFlagBrowserTest

Removing from sheriff queue as owner can take it from here.
The range in the link in comment #6 doesn't cover the revert commit (611722). Also, it's Chromium OS, not Linux. (Although it's flaky there too, see dupe in comment #7.)

When I analyze tip of tree in find it's analysis UI, it just stays at "Analyzing" seemingly forever.

The flakiness dashboard in comment #9 shows a few timeouts.

I can't verify the effect of reverting the CL. I think it can be relanded.
Cc: olka@chromium.org yoavweiss@chromium.org maxmorin@chromium.org ossu@chromium.org peah@chromium.org
Status: Started (was: Assigned)
From the log[1,2]:

[28196:28279:1210/184952.671292:WARNING:render_delay_buffer2.cc(342)] Applying total delay of 5 blocks.
[28196:28279:1210/184952.682111:WARNING:render_delay_buffer2.cc(314)] Receiving a first externally reported audio buffer delay of 0 ms.
[28196:28279:1210/184952.682309:WARNING:render_delay_buffer2.cc(342)] Applying total delay of 1 blocks.
[28196:28279:1210/184952.682394:WARNING:render_delay_buffer2.cc(239)] New max number api jitter observed at capture block 1:  2 blocks
[28196:28279:1210/184952.683070:WARNING:render_delay_buffer2.cc(239)] New max number api jitter observed at capture block 2:  3 blocks
[28196:28279:1210/184952.683133:WARNING:render_delay_buffer2.cc(257)] Render buffer underrun detected at block 2
[1:25:1210/184952.645861:FATAL:scoped_refptr.h(220)] Check failed: ptr_. 
#0 0x7fdd820f8f4d base::debug::StackTrace::StackTrace()
#1 0x7fdd81de9e5a base::debug::StackTrace::StackTrace()
#2 0x7fdd81e5bc3b logging::LogMessage::~LogMessage()
#3 0x7fdd76ed6902 scoped_refptr<>::operator->()
#4 0x7fdd7a10a922 content::WebRtcAudioDeviceImpl::RenderData()
#5 0x7fdd7a115322 content::WebRtcAudioRenderer::SourceCallback()
#6 0x7fdd7a114f19 content::WebRtcAudioRenderer::Render()
#7 0x7fdd74a8dbca media::AudioOutputDeviceThreadCallback::Process()
#8 0x7fdd74a421f7 media::AudioDeviceThread::ThreadMain()
#9 0x7fdd82134d2d base::(anonymous namespace)::ThreadFunc()
#10 0x7fdd55c00184 start_thread
#11 0x7fdd5145803d clone

The only place |renderer_| is used in WebRtcAudioDeviceImpl::RenderData() is for the early DCHECK. So |renderer_| has been invalidated (or never set) when that function is called. This could then be timing dependent.

The warnings before the CHECK is from AEC3. Per, can you explain what those mean? They could be normal, but I didn't see them in the around 15 runs I did locally (which all succeeded).

I don't see any suspicious CLs around the time when the pass rate dropped (revision 611174). Yoav, could it be that your CL changes the timing so that this issue is revealed with that?

Adding Olga and Max if you have any ideas.

[1] https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Tests%20%28dbg%29%281%29/76340
[2] https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8927472331082366912/+/steps/browser_tests/0/stdout
The warnings relate to how the audio pipeline outside of APM/AEC3 behaves and reflect how AEC3 perceives that (API call jitter/render underrun).

These warnings are fully safe for AEC3 should not cause any issues, apart from echo leakage/transparency loss if the external pipeline is behaving too badly.

Labels: Target-73
As far as I can tell there's no guarantee that |renderer_| holds a non-null pointer in this function, hence the lock which protects that variable among other. Fixing the DCHECK, and another DCHECK access that is not under lock.
Components: -Blink>WebRTC>Network Blink>WebRTC>Audio
Project Member

Comment 15 by bugdroid1@chromium.org, Dec 18

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c4580946bcf136a5e156c12d91d25d11dc05a8b8

commit c4580946bcf136a5e156c12d91d25d11dc05a8b8
Author: Henrik Grunell <grunell@chromium.org>
Date: Tue Dec 18 15:44:35 2018

Fix access of |renderer_| in WebRtcAudioDeviceImpl in DCHECKs.

* Always access under lock.
* Always check if non-null.

Fixes flaky crash in WebRtcDisableEncryptionFlagBrowserTest.VerifyEncryption.

Bug:  909704 
Change-Id: I06ca121f28916fc2c1401b7605cfee486e5f2c8c
Reviewed-on: https://chromium-review.googlesource.com/c/1382476
Commit-Queue: Henrik Grunell <grunell@chromium.org>
Reviewed-by: Max Morin <maxmorin@chromium.org>
Cr-Commit-Position: refs/heads/master@{#617500}
[modify] https://crrev.com/c4580946bcf136a5e156c12d91d25d11dc05a8b8/content/renderer/media/webrtc/webrtc_audio_device_impl.cc
[modify] https://crrev.com/c4580946bcf136a5e156c12d91d25d11dc05a8b8/content/renderer/media/webrtc/webrtc_audio_device_impl.h

Waiting for flakiness data before closing.
Status: Fixed (was: Started)
No failures since fix landed. One is marked as such, but looking at its log browser_tests is not run at all.

https://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=WebRtcDisableEncryptionFlagBrowserTest

Sign in to add a comment