New issue
Advanced search Search tips

Issue 921694 link

Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 2
Type: ----



Sign in to add a comment

network_service_content_browsertests failing on chromium.linux/Linux Tests (dbg)(1)(32)

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, Jan 14

Issue description

Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of kmarshall@chromium.org

network_service_content_browsertests failing on chromium.linux/Linux Tests (dbg)(1)(32)

Builders failed on: 
- Linux Tests (dbg)(1)(32): 
  https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Tests%20%28dbg%29%281%29%2832%29

Test log:
[ RUN      ] VideoCaptureBrowserTest.ReceiveFramesFromFakeCaptureDevice/22
Xlib:  extension "RANDR" missing on display ":99".

DevTools listening on ws://127.0.0.1:45480/devtools/browser/fded6f2d-f4a8-47de-90d8-bc2da303a816

GMOCK WARNING:
Uninteresting mock function call - returning directly.
    Function call: Opened(2, 1)
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/googlemock/docs/CookBook.md#knowing-when-to-expect for details.

GMOCK WARNING:
Uninteresting mock function call - returning directly.
    Function call: OnStarted(123)
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See https://github.com/google/googletest/blob/master/googlemock/docs/CookBook.md#knowing-when-to-expect for details.
[24277:24325:0114/104110.165485:ERROR:shared_memory_handle_posix.cc(43)] close: Bad file descriptor (9)
[24193:24266:0114/104110.298675:ERROR:browser_process_sub_thread.cc(209)] Waited 19 ms for network service
[24193:24276:0114/104110.302498:ERROR:process_posix.cc(329)] Unable to terminate process 24280: No such process (3)
[24193:24276:0114/104110.303058:WARNING:internal_linux.cc(64)] Failed to read /proc/24280/stat
[24193:24266:0114/104110.305189:WARNING:discardable_shared_memory_manager.cc(410)] Some MojoDiscardableSharedMemoryManagerImpls are still alive. They will be leaked.
[24193:24342:0114/104110.310827:WARNING:internal_linux.cc(64)] Failed to read /proc/24277/stat
BrowserTestBase received signal: Terminated. Backtrace:

 
Labels: -Sheriff-Chromium OS-Linux
Owner: chfremer@chromium.org
Status: Assigned (was: Available)
Project Member

Comment 2 by bugdroid1@chromium.org, Jan 14

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2acb596c77e8e48f567ac4ade4dad64b9d861e25

commit 2acb596c77e8e48f567ac4ade4dad64b9d861e25
Author: Kevin Marshall <kmarshall@chromium.org>
Date: Mon Jan 14 21:24:25 2019

Disable flaky test ReceiveFramesFromFakeCaptureDevice on Linux.

Observed a test flake on Linux when opening the video capture device,
so marking the test as disabled until the failure can be investigated.

TBR=chfremer@chromium.org

Bug: 921694
Change-Id: Iec57eef0c5c1ca54d7c9e3d757903fc27a92efc6
Reviewed-on: https://chromium-review.googlesource.com/c/1409902
Reviewed-by: Kevin Marshall <kmarshall@chromium.org>
Commit-Queue: Kevin Marshall <kmarshall@chromium.org>
Cr-Commit-Position: refs/heads/master@{#622607}
[modify] https://crrev.com/2acb596c77e8e48f567ac4ade4dad64b9d861e25/content/browser/renderer_host/media/video_capture_browsertest.cc

Comment 4 by chfremer@google.com, Jan 17 (5 days ago)

Note, that in all 8 failures from the two builds in #3 the test run times out and produces the following stack trace:

BrowserTestBase received signal: Terminated. Backtrace:
#0 0x0000f406dd75 base::debug::StackTrace::StackTrace()
#1 0x0000f3dc97ee base::debug::StackTrace::StackTrace()
#2 0x00005892185c content::(anonymous namespace)::DumpStackTraceSignalHandler()
#3 0x0000f771ecb0 ([vdso]+0xcaf)
#4 0x0000f771ecd7 ([vdso]+0xcd6)
#5 0x0000d1811178 pthread_join
#6 0x0000f40a33b9 base::PlatformThread::Join()
#7 0x0000f3fd0393 base::Thread::Stop()
#8 0x0000e7b7b82b media::AudioThreadImpl::Stop()
#9 0x0000e7b1fb1e media::AudioManager::Shutdown()
#10 0x0000ebdfadb0 content::BrowserMainLoop::ShutdownThreadsAndCleanUp()
#11 0x0000ebe06cfa content::BrowserMainRunnerImpl::Shutdown()

This seems to indicate that the browser is deadlocked during shutdown trying to join the "audio thread", which is actually more like a media thread, since it is also used by video capture.
Project Member

Comment 5 by bugdroid1@chromium.org, Jan 18 (5 days ago)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/003e572af5c0935ebc68007945cae07b895a61db

commit 003e572af5c0935ebc68007945cae07b895a61db
Author: Christian Fremerey <chfremer@chromium.org>
Date: Fri Jan 18 00:32:58 2019

Remove duplicate close in FakeJpegDecodeAccelerator

Running content_browsertests --gtest_filter=VideoCaptureBrowserTest.ReceiveFramesFromFakeCaptureDevice/19
produced log outputs complaining about trying to close an invalid file handle.
It appears the cause is that a SharedMemoryHandle is closed manually right after
having handed off ownership of it to a WritableUnalignedMapping.

This CL removes the manual call to close. It seems to have been added
by mistake in
https://chromium-review.googlesource.com/c/chromium/src/+/1117689

Note that the test case is currently disabled on Linux due to flakiness.
The issue fixed in this CL is probably not the cause for the flakiness.

Bug: 921694
Change-Id: I158fd243d4a62af6d786ab36a12ca74e04056de7
Reviewed-on: https://chromium-review.googlesource.com/c/1419260
Reviewed-by: Ken Rockot <rockot@google.com>
Reviewed-by: Dan Sanders <sandersd@chromium.org>
Commit-Queue: Christian Fremerey <chfremer@chromium.org>
Cr-Commit-Position: refs/heads/master@{#623926}
[modify] https://crrev.com/003e572af5c0935ebc68007945cae07b895a61db/media/gpu/fake_jpeg_decode_accelerator.cc

Comment 6 by chfremer@chromium.org, Jan 18 (4 days ago)

The deadlock I referred to in #4 is very likely due to the device thread being blocked in the destructor of class VideoCaptureJpegDecoderImpl [1]. This can be inferred from the combination of the following:

1.) The concrete test case where the timeout happens is always 
VideoCaptureBrowserTest.ReceiveFramesFromFakeCaptureDevice/22, which uses the following parameters:
use_mojo_service = 0
exercise_accelerated_jpeg_decoding = 1
resolution_to_use = 1280, 720
device_index_to_use = 2

This means class VideoCaptureJpegDecoderImpl is involved.
It also means that it runs in the Browser process, which is a requirement for causing a deadlock in there.

2.) VideoCaptureJpegDecoderImpl is the only class in media/capture/video involved in this test case that blocks a thread, which appears to be the Browser IO thread. 

3.) At the point of Browser shutdown corresponding to the stack from #4, the IO thread is already shut down.

[1] https://cs.chromium.org/chromium/src/media/capture/video/video_capture_jpeg_decoder_impl.cc?type=cs&g=0&l=48
Project Member

Comment 7 by bugdroid, Today (8 hours ago)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/14bc149e0ef309d67be1d4f9b763c41ed86b5e31

commit 14bc149e0ef309d67be1d4f9b763c41ed86b5e31
Author: Christian Fremerey <chfremer@chromium.org>
Date: Tue Jan 22 22:07:11 2019

[Video Capture] Fix deadlock on shutdown when using accelerated MJPEG decoding

The destructor of class VideoCaptureJpegDecoderImpl was blocking the
current thread, which could lead to a deadlock described in
https://bugs.chromium.org/p/chromium/issues/detail?id=921694#c6.
This deadlock has shown up as test flakiness.

This CL removes the need for blocking the thread in the destructor
by requiring that instances are always destroyed on the thread we
would otherwise have to wait for.

This CL also reenables test cases that were disabled because of the
flakiness.

Test: content_browsertests --gtest_filter=*ReceiveFramesFromFakeCaptureDevice*
Bug: 921694
Change-Id: I06b960c8e9cf749b94ce7785a3726884fd47a37d
Reviewed-on: https://chromium-review.googlesource.com/c/1423638
Commit-Queue: Christian Fremerey <chfremer@chromium.org>
Reviewed-by: Emircan Uysaler <emircan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#624957}
[modify] https://crrev.com/14bc149e0ef309d67be1d4f9b763c41ed86b5e31/content/browser/renderer_host/media/in_process_video_capture_device_launcher.cc
[modify] https://crrev.com/14bc149e0ef309d67be1d4f9b763c41ed86b5e31/content/browser/renderer_host/media/video_capture_browsertest.cc
[modify] https://crrev.com/14bc149e0ef309d67be1d4f9b763c41ed86b5e31/media/capture/BUILD.gn
[add] https://crrev.com/14bc149e0ef309d67be1d4f9b763c41ed86b5e31/media/capture/video/scoped_video_capture_jpeg_decoder.cc
[add] https://crrev.com/14bc149e0ef309d67be1d4f9b763c41ed86b5e31/media/capture/video/scoped_video_capture_jpeg_decoder.h
[modify] https://crrev.com/14bc149e0ef309d67be1d4f9b763c41ed86b5e31/media/capture/video/video_capture_device_client.cc
[modify] https://crrev.com/14bc149e0ef309d67be1d4f9b763c41ed86b5e31/media/capture/video/video_capture_jpeg_decoder_impl.cc
[modify] https://crrev.com/14bc149e0ef309d67be1d4f9b763c41ed86b5e31/media/capture/video/video_capture_jpeg_decoder_impl.h
[modify] https://crrev.com/14bc149e0ef309d67be1d4f9b763c41ed86b5e31/services/video_capture/device_media_to_mojo_adapter.cc

Sign in to add a comment