WebRtcCaptureFromElementBrowserTest.CaptureFromMediaElement/0/1/2/3 fails fairly consistently on Lollipop Low-end Tester |
||||||||||||
Issue descriptionHi mcasas, it seems you added some tests (WebRtcCaptureFromElementBrowserTest.CaptureFromMediaElement/*) in https://codereview.chromium.org/2104643002 which are failing (almost consistently) on the Lollipop Low-end Tester bot. See http://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=content_browsertests&builder=chromium.android%3ALollipop%20Low-end%20Tester for an overview of how much those tests are failing, it seems the tests were added in build 3578, i.e. https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Low-end%20Tester/builds/3578 Example log: I 3920.661s run_tests_on_device(AG86044202E68GC) [==========] Running 1 test from 1 test case. I 3920.661s run_tests_on_device(AG86044202E68GC) [----------] Global test environment set-up. I 3920.661s run_tests_on_device(AG86044202E68GC) [----------] 1 test from WebRtcCaptureFromElementBrowserTest, where TypeParam = I 3920.661s run_tests_on_device(AG86044202E68GC) [ RUN ] WebRtcCaptureFromElementBrowserTest.CaptureFromMediaElement/3 I 3920.661s run_tests_on_device(AG86044202E68GC) [WARNING:dns_config_service_posix.cc(316)] Failed to read DnsConfig. I 3920.661s run_tests_on_device(AG86044202E68GC) [ERROR:devtools_http_handler.cc(228)] Cannot start http server for devtools. Stop devtools. I 3920.661s run_tests_on_device(AG86044202E68GC) [WARNING:simple_synchronous_entry.cc(1054)] Could not open platform files for entry. I 3920.661s run_tests_on_device(AG86044202E68GC) [ERROR:shell_android.cc(70)] Not implemented reached in void content::Shell::PlatformSetTitle(const string16&): Media Capture from DOM Elements (video/audio) Browser Test I 3920.662s run_tests_on_device(AG86044202E68GC) ../../base/memory/ref_counted.h:322: T* scoped_refptr<T>::operator->() const [with T = base::SingleThreadTaskRunner]: assertion "ptr_ != NULL" failed I 3920.662s run_tests_on_device(AG86044202E68GC) I 3920.662s run_tests_on_device(AG86044202E68GC) DownloadContentTest.DownloadGZipWithNoContentPASSED I 3920.662s run_tests_on_device(AG86044202E68GC) WebRtcCaptureFromElementBrowserTest.CaptureFromMediaElement/3UNKNOWN Here's a log print from where CaptureFromMediaElement/3 doesn't crash I 9912.216s run_tests_on_device(AG86044208C42GC) Note: Google Test filter = WebRtcCaptureFromElementBrowserTest.CaptureFromMediaElement/3 I 9912.216s run_tests_on_device(AG86044208C42GC) [==========] Running 1 test from 1 test case. I 9912.216s run_tests_on_device(AG86044208C42GC) [----------] Global test environment set-up. I 9912.216s run_tests_on_device(AG86044208C42GC) [----------] 1 test from WebRtcCaptureFromElementBrowserTest, where TypeParam = I 9912.216s run_tests_on_device(AG86044208C42GC) [ RUN ] WebRtcCaptureFromElementBrowserTest.CaptureFromMediaElement/3 I 9912.216s run_tests_on_device(AG86044208C42GC) [WARNING:dns_config_service_posix.cc(316)] Failed to read DnsConfig. I 9912.216s run_tests_on_device(AG86044208C42GC) [ERROR:devtools_http_handler.cc(228)] Cannot start http server for devtools. Stop devtools. I 9912.216s run_tests_on_device(AG86044208C42GC) [WARNING:simple_synchronous_entry.cc(1054)] Could not open platform files for entry. I 9912.217s run_tests_on_device(AG86044208C42GC) [ERROR:shell_android.cc(70)] Not implemented reached in void content::Shell::PlatformSetTitle(const string16&): Media Capture from DOM Elements (video/audio) Browser Test I 9912.217s run_tests_on_device(AG86044208C42GC) referenceTable head length=54 1 I 9912.217s run_tests_on_device(AG86044208C42GC) [ OK ] WebRtcCaptureFromElementBrowserTest.CaptureFromMediaElement/3 (4271 ms) Crash from the logcat of the device of a failing test run: 6EAGC: 07-07 06:55:19.885 20979 21056 F libc : ../../base/memory/ref_counted.h:322: T* scoped_refptr<T>::operator->() const [with T = base::SingleThreadTaskRunner]: assertion "ptr_ != NULL" failed 6EAGC: 07-07 06:55:19.886 20979 21056 F libc : Fatal signal 6 (SIGABRT), code -6 in tid 21056 (mojo:media) 6EAGC: 07-07 06:55:19.891 549 3461 W libprocessgroup: failed to open /acct/uid_10089/pid_21020/cgroup.procs: No such file or directory 6EAGC: 07-07 06:55:19.891 549 3461 I ActivityManager: Process org.chromium.content_browsertests_apk:sandboxed_process0 (pid 21020) has died 6EAGC: 07-07 06:55:19.991 142 142 I DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** 6EAGC: 07-07 06:55:19.991 142 142 I DEBUG : Build fingerprint: 'google/4560MMX_b/4560MMX_b_sprout:5.1.1/LMY47W/1837160:userdebug/dev-keys' 6EAGC: 07-07 06:55:19.991 142 142 I DEBUG : Revision: '0' 6EAGC: 07-07 06:55:19.991 142 142 I DEBUG : ABI: 'arm' 6EAGC: 07-07 06:55:19.992 142 142 I DEBUG : pid: 20979, tid: 21056, name: mojo:media >>> org.chromium.content_browsertests_apk:test_process <<< 6EAGC: 07-07 06:55:19.992 142 142 I DEBUG : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr -------- 6EAGC: 07-07 06:55:20.045 142 142 I DEBUG : Abort message: '../../base/memory/ref_counted.h:322: T* scoped_refptr<T>::operator->() const [with T = base::SingleThreadTaskRunner]: assertion "ptr_ != NULL" failed' 6EAGC: 07-07 06:55:20.045 142 142 I DEBUG : r0 00000000 r1 00005240 r2 00000006 r3 00000000 6EAGC: 07-07 06:55:20.045 142 142 I DEBUG : r4 9c091dd8 r5 00000006 r6 00000000 r7 0000010c 6EAGC: 07-07 06:55:20.045 142 142 I DEBUG : r8 b79d2490 r9 b6ddcdd4 sl b79d249c fp 9c091dd0 6EAGC: 07-07 06:55:20.045 142 142 I DEBUG : ip 00005240 sp 9c091618 lr b6d97db5 pc b6dbe980 cpsr 60050010 6EAGC: 07-07 06:55:20.046 142 142 I DEBUG : 6EAGC: 07-07 06:55:20.046 142 142 I DEBUG : backtrace: 6EAGC: 07-07 06:55:20.046 142 142 I DEBUG : #00 pc 0003d980 /system/lib/libc.so (tgkill+12) 6EAGC: 07-07 06:55:20.046 142 142 I DEBUG : #01 pc 00016db1 /system/lib/libc.so (pthread_kill+52) 6EAGC: 07-07 06:55:20.046 142 142 I DEBUG : #02 pc 000179c7 /system/lib/libc.so (raise+10) 6EAGC: 07-07 06:55:20.046 142 142 I DEBUG : #03 pc 00014169 /system/lib/libc.so (__libc_android_abort+36) 6EAGC: 07-07 06:55:20.046 142 142 I DEBUG : #04 pc 000124f0 /system/lib/libc.so (abort+4) 6EAGC: 07-07 06:55:20.047 142 142 I DEBUG : #05 pc 0001549f /system/lib/libc.so (__libc_fatal+16) 6EAGC: 07-07 06:55:20.047 142 142 I DEBUG : #06 pc 000141ed /system/lib/libc.so (__assert2+20) 6EAGC: 07-07 06:55:20.047 142 142 I DEBUG : #07 pc 0048a6e5 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so 6EAGC: 07-07 06:55:20.047 142 142 I DEBUG : #08 pc 00b1ae31 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so 6EAGC: 07-07 06:55:20.047 142 142 I DEBUG : #09 pc 00a924fb /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so 6EAGC: 07-07 06:55:20.047 142 142 I DEBUG : #10 pc 00a9259d /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so 6EAGC: 07-07 06:55:20.048 142 142 I DEBUG : #11 pc 0284ff81 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so 6EAGC: 07-07 06:55:20.048 142 142 I DEBUG : #12 pc 02850061 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so 6EAGC: 07-07 06:55:20.048 142 142 I DEBUG : #13 pc 0056cdb7 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so 6EAGC: 07-07 06:55:20.048 142 142 I DEBUG : #14 pc 0056530f /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so 6EAGC: 07-07 06:55:20.048 142 142 I DEBUG : #15 pc 005655e5 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so 6EAGC: 07-07 06:55:20.048 142 142 I DEBUG : #16 pc 0095fd1d /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so 6EAGC: 07-07 06:55:20.048 142 142 I DEBUG : #17 pc 01fd75e3 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so 6EAGC: 07-07 06:55:20.049 142 142 I DEBUG : #18 pc 01fd7615 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so 6EAGC: 07-07 06:55:20.049 142 142 I DEBUG : #19 pc 02007db7 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so 6EAGC: 07-07 06:55:20.049 142 142 I DEBUG : #20 pc 0200431f /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so 6EAGC: 07-07 06:55:20.049 142 142 I DEBUG : #21 pc 0001659f /system/lib/libc.so (__pthread_start(void*)+30) 6EAGC: 07-07 06:55:20.049 142 142 I DEBUG : #22 pc 000144cb /system/lib/libc.so (__start_thread+6) 6EAGC: 07-07 06:55:20.406 549 649 D TaskPersister: removeObsoleteFile: deleting file=745_task.xml 6EAGC: 07-07 06:55:20.406 549 649 D TaskPersister: removeObsoleteFile: deleting file=745_task_thumbnail.png 6EAGC: 07-07 06:55:21.225 142 142 I DEBUG : 6EAGC: 07-07 06:55:21.225 142 142 I DEBUG : Tombstone written to: /data/tombstones/tombstone_09
,
Jul 7 2016
It seems to be failing indeed. Two questions: - do you know how can I symbolize the crash stack? (tried third_party/android_platform/development/scripts/stack against my ToT but didn't work). - is there any easy way to single out this platform and DISABLE the test(s)? It's also strange that these 3 tests fails whereas the neighbour VerifyCanvasCaptureColor works ok.
,
Jul 8 2016
,
Jul 8 2016
About the first question, I would guess that it is possible to pull the needed data from the actual bot (given that you probably can't repro locally?), not sure exactly what is needed atm, can try to investigate next week. As for the second question it should be possible I think, jbudorick@ do you know how to disable a test just for the Lollipop Low-end Tester bot? (do we really want to disable the test for that bot though? It might be revealing some timing/memory issue)
,
Jul 8 2016
Yeah, annotate the test w/ @DisableIf.Build(hardware_is = "sprout", message = "<crbug link>") e.g. https://codesearch.chromium.org/search/?q=hardware_is.*sprout&sq=package:chromium&type=cs
,
Jul 8 2016
#5: These tests are content_browsertests written in C++ [1], is there any disabling-annotation equivalent in that case? [1] https://cs.chromium.org/chromium/src/content/browser/media/webrtc/webrtc_capture_from_element_browsertest.cc?q=CaptureFromMediaElement&sq=package:chromium&l=64&dr=C
,
Jul 8 2016
#6: oops, should've looked more closely at #0. I don't think we have similar runtime test disabling abilities in c++.
,
Jul 11 2016
Downloaded the zip of build files used for run nr. 3623 (https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Low-end%20Tester/builds/3623) and ran addr2line on the addresses in the stack of that build: 0x0048e66d scoped_refptr<base::SingleThreadTaskRunner>::operator->() const /b/build/slave/android_build/build/src/out/Debug/../../base/memory/ref_counted.h:322 (discriminator 1) 0x00b22011 content::EmbeddedApplicationRunner::Instance::Quit() /b/build/slave/android_build/build/src/out/Debug/../../content/common/mojo/embedded_application_runner.cc:101 0x00a98fbb shell::ShellConnectionRefImpl::~ShellConnectionRefImpl() /b/build/slave/android_build/build/src/out/Debug/../../services/shell/public/cpp/lib/shell_connection_ref.cc:31 (discriminator 1) 0x00a9905d shell::ShellConnectionRefImpl::~ShellConnectionRefImpl() /b/build/slave/android_build/build/src/out/Debug/../../services/shell/public/cpp/lib/shell_connection_ref.cc:37 0x0285fe51 std::__1::default_delete<shell::ShellConnectionRef>::operator()(shell::ShellConnectionRef*) const /b/build/slave/android_build/build/src/out/Debug/../../third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libcxx/include/memory:2431 (discriminator 1) 0x0285ff31 media::ServiceFactoryImpl::~ServiceFactoryImpl() /b/build/slave/android_build/build/src/out/Debug/../../media/mojo/services/service_factory_impl.cc:52 0x0057281b base::Callback<void (), (base::internal::CopyMode)1>::Run() const /b/build/slave/android_build/build/src/out/Debug/../../base/callback.h:389 (discriminator 1) 0x0056ad73 base::Callback<void (), (base::internal::CopyMode)1>::Run() const /b/build/slave/android_build/build/src/out/Debug/../../base/callback.h:389 (discriminator 1) 0x0056b049 mojo::Connector::OnHandleReadyInternal(unsigned int) /b/build/slave/android_build/build/src/out/Debug/../../mojo/public/cpp/bindings/lib/connector.cc:225 0x009667dd base::Callback<void (unsigned int), (base::internal::CopyMode)1>::Run(unsigned int) const /b/build/slave/android_build/build/src/out/Debug/../../base/callback.h:389 (discriminator 2) 0x01fe3c6f base::MessageLoop::~MessageLoop() /b/build/slave/android_build/build/src/out/Debug/../../base/message_loop/message_loop.cc:174 (discriminator 7) 0x01fe3ca1 base::MessageLoop::~MessageLoop() /b/build/slave/android_build/build/src/out/Debug/../../base/message_loop/message_loop.cc:188 0x0201458f std::__1::default_delete<base::MessageLoop>::operator()(base::MessageLoop*) const /b/build/slave/android_build/build/src/out/Debug/../../third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libcxx/include/memory:2431 (discriminator 1) 0x02010af7 ThreadFunc /b/build/slave/android_build/build/src/out/Debug/../../base/threading/platform_thread_posix.cc:70 Adding a couple of ppl who seems to have been fiddling with code related to that call stack lately (e.g. in https://codereview.chromium.org/2118083002). rockot@ and ben@ do you know what could cause this crash stack?
,
Jul 11 2016
I see what I think is an obvious cause for this bug and it's my fault. Simple fix incoming, though my luck reproing is nil so I'll be going partially on faith.
,
Jul 11 2016
#9: what devices are you using to try to repro? I think we can dig up a sprout (i.e., what that bot is using) if you'd like to borrow one.
,
Jul 11 2016
Seems like a large time sink for what I think is an obvious fix. I'd rather just land https://codereview.chromium.org/2140673003 which I'm fairly certain resolves.
,
Jul 11 2016
(I only have various Nexus devices fwiw)
,
Jul 11 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/5265f8ef11e3205bdbbd0178f8c739e6e39973b7 commit 5265f8ef11e3205bdbbd0178f8c739e6e39973b7 Author: rockot <rockot@chromium.org> Date: Mon Jul 11 19:57:39 2016 Fix race in EmbeddedApplicationRunner::ShutDown Joins the application thread before resetting the stored TaskRunner reference. Avoids a nullptr dereference if ShutDown is called immediately after BindServiceRequest but before the posted task has run. BUG=626299 TBR=ben@chromium.org Review-Url: https://codereview.chromium.org/2140673003 Cr-Commit-Position: refs/heads/master@{#404713} [modify] https://crrev.com/5265f8ef11e3205bdbbd0178f8c739e6e39973b7/content/common/mojo/embedded_application_runner.cc
,
Jul 12 2016
Most recent build no longer shows these failures: https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Low-end%20Tester/builds/3640 Could only be fixed flakily, but given the stack of the failures I think it's fairly safe to say that r404713 actually does fix them.
,
Jul 12 2016
Yeah, it looks like in all of the previous runs at least one or two of the 4 CaptureFromMediaElement tests were failing (and in that run none of them did), thanks rockot@!
,
Jul 12 2016
,
Aug 26 2016
These seem to be failing again :( https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Low-end%20Tester?numbuilds=100
,
Aug 26 2016
Failed in https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Low-end%20Tester/builds/3851 did not fail in https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Low-end%20Tester/builds/3850 but this time is due to a Check: I 3361.086s run_tests_on_device(AG860440II6EAGC) [FATAL:embedded_application_runner.cc(72)] Check failed: !thread_. I 3361.087s run_tests_on_device(AG860440II6EAGC) #00 0xa11985d5 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so+0x006aa5d5 I 3361.087s run_tests_on_device(AG860440II6EAGC) #01 0xa135e2d5 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so+0x008702d5 I 3361.087s run_tests_on_device(AG860440II6EAGC) #02 0xa135e385 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so+0x00870385 I 3361.087s run_tests_on_device(AG860440II6EAGC) #03 0xa135e39b /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so+0x0087039b I 3361.087s run_tests_on_device(AG860440II6EAGC) #04 0xa1183a6f /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so+0x00695a6f I 3361.087s run_tests_on_device(AG860440II6EAGC) #05 0xa11ae5eb /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so+0x006c05eb I 3361.087s run_tests_on_device(AG860440II6EAGC) #06 0xa119dd91 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so+0x006afd91 I 3361.087s run_tests_on_device(AG860440II6EAGC) #07 0xa119e143 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so+0x006b0143 I 3361.087s run_tests_on_device(AG860440II6EAGC) #08 0xa119e3ed /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so+0x006b03ed I 3361.087s run_tests_on_device(AG860440II6EAGC) #09 0xa11d12e3 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so+0x006e32e3 I 3361.087s run_tests_on_device(AG860440II6EAGC) #10 0xa11cd2f7 /data/app/org.chromium.content_browsertests_apk-1/lib/arm/lib_content_browsertests__library.so+0x006df2f7 I 3361.087s run_tests_on_device(AG860440II6EAGC) #11 0xb6dfb59f /system/lib/libc.so+0x0001659f I 3361.087s run_tests_on_device(AG860440II6EAGC) #12 0xb6df94cb /system/lib/libc.so+0x000144cb I 3361.087s run_tests_on_device(AG860440II6EAGC) #13 0xffffffff <unknown> I 3361.087s run_tests_on_device(AG860440II6EAGC) And that's what's been happening all the time. Any idea what could have changed? The Check line is: https://cs.chromium.org/chromium/src/content/common/mojo/embedded_application_runner.cc?type=cs&q=embedded_application_runner&sq=package:chromium&l=72
,
Aug 26 2016
Seems like this would be possible if the runner's message loop has been shutdown and the QuitOnRunnerThread task is dropped, causing Instance to be destroyed on the application thread.
,
Sep 9 2016
These tests are still failing, any updates here? :)
,
Sep 21 2016
,
Sep 22 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/295a7beaf3b3610204405d9cbb7848e9c0b63da3 commit 295a7beaf3b3610204405d9cbb7848e9c0b63da3 Author: mcasas <mcasas@chromium.org> Date: Thu Sep 22 16:26:47 2016 Disable WebRtcCaptureFromElementBrowserTest.MAYBE_CaptureFromMediaElement in devices pre-Marshmallow These (4) tests time out in Lollipop Low-End bot; this CL disables this particular version of Android while investigations continue. BUG=626299 Review-Url: https://codereview.chromium.org/2362553002 Cr-Commit-Position: refs/heads/master@{#420368} [modify] https://crrev.com/295a7beaf3b3610204405d9cbb7848e9c0b63da3/content/browser/webrtc/webrtc_capture_from_element_browsertest.cc
,
Sep 23 2016
Verified that #22 stopped the test from running (and hence failing) in Lollipop Low-End.
,
May 29 2017
Not really ServiceManager component, but close enough - content's usage of Service Manager. I haven't had time to sort out what the bug actually is here, but I'm at least moving this to the pile of available bugs which are not actively being investigated.
,
Nov 1 2017
,
Oct 17
|
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by gsennton@chromium.org
, Jul 7 2016Status: Assigned (was: Untriaged)