Flaky native crash during PushMessagingTest#testPushPermissionDenied |
|||||||||||||||
Issue description"org.chromium.chrome.browser.push_messaging.PushMessagingTest#testPushPermissionDenied" is flaky. This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is infrastructure-related, please add Infra-Troopers label and change issue status to Untriaged. When done, please remove the issue from Sheriff Bug Queue by removing the Sheriff-Chromium label. We have detected 3 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyYAsSBUZsYWtlIlVvcmcuY2hyb21pdW0uY2hyb21lLmJyb3dzZXIucHVzaF9tZXNzYWdpbmcuUHVzaE1lc3NhZ2luZ1Rlc3QjdGVzdFB1c2hQZXJtaXNzaW9uRGVuaWVkDA. Flaky tests should be disabled within 30 minutes unless culprit CL is found and reverted. Please see more details here: https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/sheriffing-bug-queues#triaging-auto-filed-flakiness-bugs
,
Apr 4 2017
Issue 707994 has been merged into this issue.
,
Apr 4 2017
When it flakes, this test is crashing in exactly the same way as https://crbug.com/707994 . Again, no stack traces, just a notification in the output that native is crashing. I had a brief look in the log but didn't see anything relevant. I'll have to leave this one in the hands of the push folks.
,
Apr 4 2017
,
Apr 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/970a4c9930710171b68590310689da52ac6f02e7 commit 970a4c9930710171b68590310689da52ac6f02e7 Author: maxmorin <maxmorin@chromium.org> Date: Tue Apr 04 08:47:26 2017 Disable flaky PushMessagingTest#testPushPermissionDenied and NotificationPlatformBridgeTest#testPermissionGranted. TBR=peter@chromium.org BUG=707528, 707994 Review-Url: https://codereview.chromium.org/2789353002 Cr-Commit-Position: refs/heads/master@{#461663} [modify] https://crrev.com/970a4c9930710171b68590310689da52ac6f02e7/chrome/android/javatests/src/org/chromium/chrome/browser/notifications/NotificationPlatformBridgeTest.java [modify] https://crrev.com/970a4c9930710171b68590310689da52ac6f02e7/chrome/android/javatests/src/org/chromium/chrome/browser/push_messaging/PushMessagingTest.java
,
Apr 4 2017
,
Apr 4 2017
Could this somehow be related to https://codereview.chromium.org/2781413002/?
,
Apr 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f3800a0ecbf562fab539af5a06c69244ad09e990 commit f3800a0ecbf562fab539af5a06c69244ad09e990 Author: maxmorin <maxmorin@chromium.org> Date: Tue Apr 04 13:21:15 2017 Disable flaky NotificationPlatformBridgeTest#testPermissionDenied. Looks like this one is flaky too: https://build.chromium.org/p/chromium.linux/builders/Android%20Tests%20%28dbg%29/builds/41439 TBR=peter@chromium.org BUG=707528 Review-Url: https://codereview.chromium.org/2792853004 Cr-Commit-Position: refs/heads/master@{#461692} [modify] https://crrev.com/f3800a0ecbf562fab539af5a06c69244ad09e990/chrome/android/javatests/src/org/chromium/chrome/browser/notifications/NotificationPlatformBridgeTest.java
,
Apr 6 2017
Might have been my patch https://codereview.chromium.org/2757483002, will have a look when I get a chance.
,
Apr 6 2017
Actually probably not my patch, those tests wouldn't run through the PermissionPromptAndroid since it isn't enabled for them.
,
Sep 7 2017
,
Sep 8 2017
Just tried a local run and all the PushMessagingTests failed: C 213.231s Main Summary C 213.232s Main ******************************************************************************** C 213.232s Main [==========] 5 tests ran. C 213.232s Main [ PASSED ] 0 tests. C 213.232s Main [ FAILED ] 5 tests, listed below: C 213.232s Main [ FAILED ] org.chromium.chrome.browser.push_messaging.PushMessagingTest#testDefaultNotification C 213.232s Main [ FAILED ] org.chromium.chrome.browser.push_messaging.PushMessagingTest#testNotificationsPermissionDenied C 213.232s Main [ FAILED ] org.chromium.chrome.browser.push_messaging.PushMessagingTest#testPushAndShowNotification C 213.232s Main [ FAILED ] org.chromium.chrome.browser.push_messaging.PushMessagingTest#testPushPermissionDenied C 213.232s Main [ FAILED ] org.chromium.chrome.browser.push_messaging.PushMessagingTest#testPushPermissionGranted C 213.229s Main [FAIL] Example failure: org.chromium.chrome.browser.push_messaging.PushMessagingTest#testPushPermissionGranted: C 213.229s Main junit.framework.AssertionFailedError: Tab never selected/initialized. C 213.229s Main at junit.framework.Assert.fail(Assert.java:50) C 213.229s Main at junit.framework.Assert.assertTrue(Assert.java:20) C 213.229s Main at org.chromium.content.browser.test.util.CriteriaHelper.pollInstrumentationThread(CriteriaHelper.java:79) C 213.229s Main at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:117) C 213.229s Main at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:138) C 213.229s Main at org.chromium.chrome.test.ChromeActivityTestCommon.startMainActivityFromIntent(ChromeActivityTestCommon.j ava:385) C 213.229s Main at org.chromium.chrome.test.ChromeActivityTestCommon.startMainActivityWithURL(ChromeActivityTestCommon.java :351) C 213.229s Main at org.chromium.chrome.test.ChromeActivityTestRule.startMainActivityWithURL(ChromeActivityTestRule.java:240 ) C 213.229s Main at org.chromium.chrome.test.ChromeActivityTestRule.startMainActivityFromLauncher(ChromeActivityTestRule.jav a:231) C 213.229s Main at org.chromium.chrome.browser.notifications.NotificationTestRule.setUp(NotificationTestRule.java:52) C 213.229s Main at org.chromium.chrome.browser.notifications.NotificationTestRule.access$100(NotificationTestRule.java:34) C 213.229s Main at org.chromium.chrome.browser.notifications.NotificationTestRule$3.evaluate(NotificationTestRule.java:149) C 213.229s Main at org.chromium.chrome.test.ChromeActivityTestRule$1.evaluate(ChromeActivityTestRule.java:65) This was on an Android O device though, maybe that had something to do with it? I noticed it kept returning to the app launcher instead of the home screen between tests, maybe that is related?
,
Sep 11 2017
Indeed, just tested on KitKat phone and only had two failures: C 213.232s Main [ FAILED ] org.chromium.chrome.browser.push_messaging.PushMessagingTest#testDefaultNotification C 213.232s Main [ FAILED ] org.chromium.chrome.browser.push_messaging.PushMessagingTest#testNotificationsPermissionDenied Meanwhile 5/5 failed on O in a second test run..
,
Sep 11 2017
Hmm maybe they're all just flaky - over 4 test runs on the K phone I got 2, 0, 4, and 0 failures out of 5.
,
Sep 11 2017
Actually, I think the "Tab never selected/initialized" failure I get very commonly locally may be a different error from the one causing flakes on the try bots. This is corroborated by the description of https://chromium-review.googlesource.com/c/chromium/src/+/587373 . Once or twice locally I got the following error, which I think may the same as the flakes on the test bots (hard to tell without an original stack posted in this bug description though :/) C 278.851s Main [FAIL] org.chromium.chrome.browser.notifications.NotificationPlatformBridgeTest#testPermissionGranted: C 278.851s Main junit.framework.AssertionFailedError: Criteria not met in allotted time. C 278.851s Main at junit.framework.Assert.fail(Assert.java:50) C 278.851s Main at junit.framework.Assert.assertTrue(Assert.java:20) C 278.851s Main at org.chromium.content.browser.test.util.CriteriaHelper.pollInstrumentationThread(CriteriaHelper.java:79) C 278.851s Main at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:117) C 278.851s Main at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:138) C 278.851s Main at org.chromium.chrome.browser.notifications.NotificationPlatformBridgeTest.getInfobarBlocking(Notification PlatformBridgeTest.java:97) C 278.851s Main at org.chromium.chrome.browser.notifications.NotificationPlatformBridgeTest.testPermissionGranted(Notificat ionPlatformBridgeTest.java:218)
,
Sep 11 2017
Scaling up the timeout of CriteriaHelper.DEFAULT_MAX_TIME_TO_POLL by 10x massively reduces the flakiness, I did just got yet another error though just now (5x running O): C 145.146s Main ******************************************************************************** C 145.146s Main Detailed Logs C 145.146s Main ******************************************************************************** C 145.147s Main [FAIL] org.chromium.chrome.browser.push_messaging.PushMessagingTest#testDefaultNotification: C 145.147s Main org.junit.ComparisonFailure: expected:<s[etNotifyOnPush fals]e ok> but was:<s[ubscrib]e ok> C 145.147s Main at org.junit.Assert.assertEquals(Assert.java:115) C 145.147s Main at org.junit.Assert.assertEquals(Assert.java:144) C 145.147s Main at org.chromium.chrome.browser.push_messaging.PushMessagingTest.waitForTitle(PushMessagingTest.java:344) C 145.147s Main at org.chromium.chrome.browser.push_messaging.PushMessagingTest.runScriptAndWaitForTitle(PushMessagingTest. java:309) C 145.147s Main at org.chromium.chrome.browser.push_messaging.PushMessagingTest.testDefaultNotification(PushMessagingTest.j ava:258)
,
Sep 11 2017
So yes, increasing the timeout here stops it from timing out with the 'Tab never selected/initialized' error locally: https://cs.chromium.org/chromium/src/chrome/test/android/javatests/src/org/chromium/chrome/test/ChromeActivityTestCommon.java?l=385 (From adding logs it seems it was timing out while loading the chrome.cr native libary.) Re-enabling the tests and kicking off a tryjob here[1] showed some actual errors in NPBTest.pushPermissionGranted & Denied [2] - from looking at the post test screenshots I wonder if it is related to the infobar / permission prompt UI changes? (Or perhaps that made these disabled tests fail consistently rather than flakily..) [1] https://chromium-review.googlesource.com/c/chromium/src/+/660219 [2] https://00e9e64bacc9ae6f44ffc1a0525f8e19569b092de36378fcd0-apidata.googleusercontent.com/download/storage/v1/b/chromium-result-details/o/html%2Fchrome_public_test_apk_linux_android_rel_ng_382179_2017_09_11_T15_57_35-UTC?qk=AD5uMEsy_QIi_4WvnqXUXyfNvXVXTjPKarHRGiLAdP5-wHWywwNzUMcfPJjI0DkMLRmSP_ST0xZOjhiv_GCR5IbGr1CLBq5YWA2TLdWdqwPk3GWabjyfD-jY4BdR4kpC411j5B3viR6HDzZpD12A1yMEovdAfYo_o6ud2D3Qjxj4HeVYGQ-vSj7PUZOPP3N69_I1NdVYydErmqGMEtaod3bBQgMtyNtaGmgJLRLEX6UCR45OobWllwxUW_9HZ-mzeKqj0ILECA5QnZxKl7w6hYBKRMbWI7RZDahVo-m3xx4fos58rUcQW1LspUbnxqb1bd4V7PEjHWK361CFf-mhbWdzrJvfrpcuupbK6nbysMdOlRfYE6Wkhx1dGIWQ126V7aNLBCH_msQDrGMCEyvneZD3Zp6W2EOuNd83g257jO1pqTYSrrvVXHMXdm_fP17AWxvxjT5pS2V9M_Q_6o0G6J6jzJCB88IzHTEl1hvKWMn4ZnE4DUWA9T7kToLYrvFQ17UApDivfWQfP5WRRPr7LqAiCjlQejRckRraVD78FZmlMWnyKCVQUDF_EhKlfeiX2LuB9xhbe-cKUD-7-Iyg748i-GRr485BkBt9zUkH7ioaTwmujatzLHk-2aiFI7fR18YKeKItGBODZzU_MJ1qDyD4m5THuTxfCcJed0sOkBxHGfTVp5YYw3wvLSNkD2EUosGp9S8kAPIze_5GP5E2a0TZKz26eJMDtiAY0Ivxxmh1QBlu8Q9ZCZM79E2431r3qbzQzqYGiRdVkfjE9As6w7ceAa2oCkKzxksyli3wX1jVx9z2pybP5CuDCVC-Eg0gM9IUZTZsFSkurxZ17imTwfK2NP_WXqg-NweskjPd2r_DZlno0_d9NJ42mcxY1uTs4lor_0BIPflF junit.framework.AssertionFailedError: Criteria not met in allotted time. at junit.framework.Assert.fail(Assert.java:50) at junit.framework.Assert.assertTrue(Assert.java:20) at org.chromium.content.browser.test.util.CriteriaHelper.pollInstrumentationThread(CriteriaHelper.java:79) at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:117) at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:138) at org.chromium.chrome.browser.notifications.NotificationPlatformBridgeTest.getInfobarBlocking(NotificationPlatformBridgeTest.java:97) at org.chromium.chrome.browser.notifications.NotificationPlatformBridgeTest.testPermissionDenied(NotificationPlatformBridgeTest.java:158) junit.framework.AssertionFailedError: Criteria not met in allotted time. at junit.framework.Assert.fail(Assert.java:50) at junit.framework.Assert.assertTrue(Assert.java:20) at org.chromium.content.browser.test.util.CriteriaHelper.pollInstrumentationThread(CriteriaHelper.java:79) at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:117) at org.chromium.content.browser.test.util.CriteriaHelper.pollUiThread(CriteriaHelper.java:138) at org.chromium.chrome.browser.notifications.NotificationPlatformBridgeTest.getInfobarBlocking(NotificationPlatformBridgeTest.java:97) at org.chromium.chrome.browser.notifications.NotificationPlatformBridgeTest.testPermissionGranted(NotificationPlatformBridgeTest.java:218)
,
Sep 11 2017
,
Sep 12 2017
Okay, reading back up this bug looks like the original error was a native crash with no obvious stack trace. I think I have now managed to reproduce this on a trybot on my experimental CL (www.crrev.com/c/660219): https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/383118 There are a couple of (v wide!) native stacks in the tombstones - see attached - but I'm not sure how to interpret them.. Something to do with ref_counted errors & ServiceWorkerVersion??
,
Sep 12 2017
(the tombstones in #19 are from a CRASH in org.chromium.chrome.browser.push_messaging.PushMessagingTest#testPushPermissionDenied I should have mentioned)
,
Sep 12 2017
Looks like the immediate cause of the crash is: [FATAL:ref_counted.h(95)] Check failed: CalledOnValidSequence()
,
Sep 12 2017
Looks like it's a ServiceWorkerVersion, or possibly a PrefRegistry, being destroyed on the wrong thread: Stack Trace: RELADDR FUNCTION FILE:LINE 00021f90 tgkill+12 /system/lib/libc.so 00012fe1 pthread_kill+48 /system/lib/libc.so 000131f5 raise+10 /system/lib/libc.so 00011f2b <unknown> /system/lib/libc.so 00021844 abort+4 /system/lib/libc.so v------> base::debug::(anonymous namespace)::DebugBreak() /b/c/b/android/src/base/debug/debugger_posix.cc:228 002f6d31 base::debug::BreakDebugger()+20 /b/c/b/android/src/base/debug/debugger_posix.cc:258 0030c123 logging::LogMessage::~LogMessage()+930 /b/c/b/android/src/base/logging.cc:791 00319ba5 base::subtle::RefCountedBase::Release() const+92 /b/c/b/android/src/base/memory/ref_counted.h:95 0086f031 base::RefCounted<PrefRegistry>::Release() const+4 /b/c/b/android/src/base/memory/ref_counted.h:308 v------> scoped_refptr<net::HttpAuthController>::Release(net::HttpAuthController*) /b/c/b/android/src/base/memory/ref_counted.h:640 0086b8ef scoped_refptr<PrefRegistry>::~scoped_refptr()+10 /b/c/b/android/src/base/memory/ref_counted.h:535 v------> std::__ndk1::__tuple_leaf<2u, scoped_refptr<content::ServiceWorkerVersion>, false>::~__tuple_leaf() /b/c/b/android/src/third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libcxx/include/tuple:180 011d22e7 std::__ndk1::__tuple_impl<std::__ndk1::__tuple_indices<0u, 1u, 2u>, scoped_refptr<content::ServiceWorkerRegistration>, base::RepeatingCallback<void (content::ServiceWorkerStatusCode)>, scoped_refptr<content::ServiceWorkerVersion> >::~__tuple_impl()+10 /b/c/b/android/src/third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libcxx/include/tuple:383 v------> std::__ndk1::tuple<scoped_refptr<content::ServiceWorkerRegistration>, base::RepeatingCallback<void (content::ServiceWorkerStatusCode)>, scoped_refptr<content::ServiceWorkerVersion> >::~tuple() /b/c/b/android/src/third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libcxx/include/tuple:486 011d22cf base::internal::BindState<void (content::ServiceWorkerRegistration::*)(base::RepeatingCallback<void (content::ServiceWorkerStatusCode)> const&, scoped_refptr<content::ServiceWorkerVersion>, content::ServiceWorkerStatusCode), scoped_refptr<content::ServiceWorkerRegistration>, base::RepeatingCallback<void (content::ServiceWorkerStatusCode)>, scoped_refptr<content::ServiceWorkerVersion> >::~BindState()+8 /b/c/b/android/src/base/bind_internal.h:469 I'm now debugging by adding logs everywhere and running the test repeatedly until it fails with the following command: $ nice ninja -C out/AndroidRelease -j2000 -l20 chrome_public_test_apk && out/AndroidRelease/bin/run_chrome_public_test_apk --test-filter "*PushMessagingTest#testPushPermissionDenied" --repeat 50 --timeout-scale 10 --break-on-failure Seems to fail around 6% of the time.
,
Sep 13 2017
+falken This is a pretty confusing stack trace. But I'm fairly sure PrefRegistry and HttpAuthController in the trace are red herrings due to symbol deduplication (i.e. those method calls are identical to other method calls, meaning their symbols correspond to the same code, meaning that when the symbols are matched the deduping has basically picked one of them at random). I *think* you can trace this to something about the methods ServiceWorkerStorage::WriteRegistrationInDB, ServiceWorkerRegistration::AbortPendingClear, ServiceWorkerRegistration::OnRestoreFinished. All of these appear on the trace (based on looking around for the particular arguments in the BindState objects being destroyed - they're pretty unique combinations). So it looks like the callbacks for a bunch of service worker things are running, and *something* is being deref'd on the wrong thread, something with symbols identical to the PrefRegistry deref.....
,
Sep 13 2017
Interesting, I see a similar crash stack for the external/wpt/service-workers/service-worker/unregister-then-register-new-script.https.html flaky failures: http://build.chromium.org/p/chromium.webkit/builders/WebKit%20Win7%20(dbg)/builds/11004 1d0af0a0 0166411a base!base::debug::BreakDebugger+0x17 1d0af624 015a7f6d base!logging::LogMessage::~LogMessage+0x3ca 1d0af79c 105514d2 base!base::subtle::RefCountedBase::Release+0xfd 1d0af7b8 11636ede content!base::RefCounted<content::ServiceWorkerRegistration>::Release+0x12 1d0af7c0 116359da content!scoped_refptr<content::ServiceWorkerRegistration>::Release+0xe 1d0af7d0 1166029f content!scoped_refptr<content::ServiceWorkerRegistration>::~scoped_refptr<content::ServiceWorkerRegistration>+0x1a 1d0af7dc 124e64b2 content!std::_Tuple_val<scoped_refptr<content::ServiceWorkerRegistration> >::~_Tuple_val<scoped_refptr<content::ServiceWorkerRegistration> >+0xf 1d0af7e8 124e6032 content!std::tuple<scoped_refptr<content::ServiceWorkerRegistration>,base::RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)>,scoped_refptr<content::ServiceWorkerVersion> >::~tuple<scoped_refptr<content::ServiceWork erRegistration>,base::RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)>,scoped_refptr<content::ServiceWorkerVersion> >+0x12 1d0af7f4 124e70ef content!base::internal::BindState<void (__thiscall content::ServiceWorkerRegistration::*)(base::RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)> const &,scoped_refptr<content::ServiceWorkerVersion>,enum content::S erviceWorkerStatusCode),scoped_refptr<content::ServiceWorkerRegistration>,base::RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)>,scoped_refptr<content::ServiceWorkerVersion> >::~BindState<void (__thiscall content::ServiceWorkerRegi stration::*)(base::RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)> const &,scoped_refptr<content::ServiceWorkerVersion>,enum content::ServiceWorkerStatusCode),scoped_refptr<content::ServiceWorkerRegistration>,base::RepeatingCallba ck<void __cdecl(enum content::ServiceWorkerStatusCode)>,scoped_refptr<content::ServiceWorkerVersion> >+0x12 1d0af800 124e8932 content!base::internal::BindState<void (__thiscall content::ServiceWorkerRegistration::*)(base::RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)> const &,scoped_refptr<content::ServiceWorkerVersion>,enum content::S erviceWorkerStatusCode),scoped_refptr<content::ServiceWorkerRegistration>,base::RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)>,scoped_refptr<content::ServiceWorkerVersion> >::`scalar deleting destructor'+0xf 1d0af818 015ae87f content!base::internal::BindState<void (__thiscall content::ServiceWorkerRegistration::*)(base::RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)> const &,scoped_refptr<content::ServiceWorkerVersion>,enum content::S erviceWorkerStatusCode),scoped_refptr<content::ServiceWorkerRegistration>,base::RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)>,scoped_refptr<content::ServiceWorkerVersion> >::Destroy+0x22 1d0af824 015a7ddf base!base::internal::BindStateBaseRefCountTraits::Destruct+0xf 1d0af834 015ae96b base!base::RefCountedThreadSafe<base::internal::BindStateBase,base::internal::BindStateBaseRefCountTraits>::Release+0x1f 1d0af83c 015ae64a base!scoped_refptr<base::internal::BindStateBase>::Release+0xb 1d0af84c 015ae66f base!scoped_refptr<base::internal::BindStateBase>::~scoped_refptr<base::internal::BindStateBase>+0x1a 1d0af858 015a493f base!base::internal::CallbackBase::~CallbackBase+0xf 1d0af864 1054dc00 base!base::internal::CallbackBaseCopyable::~CallbackBaseCopyable+0xf 1d0af870 1166023f content!base::RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)>::~RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)>+0x10 1d0af87c 1251b865 content!std::_Tuple_val<base::RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)> >::~_Tuple_val<base::RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)> >+0xf 1d0af888 1251bb3d content!std::tuple<base::RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)>,content::ServiceWorkerDatabase::RegistrationData>::~tuple<base::RepeatingCallback<void __cdecl(enum content::ServiceWorkerStatusCode)>,content::ServiceWorkerDatabase::RegistrationData>+0x15
,
Sep 13 2017
falken: I'd be pretty suspicious of the set of calls through ServiceWorkerRegistration::AbortPendingClear in that case, it seems to appear in both of these stacks...
,
Sep 13 2017
Ah, ignore #26, that was just me breaking the javascript when adding logging.
,
Sep 13 2017
Today I tried to get my Android tablet and Windows machine setup again since I couldn't repro the layout test failure on Linux. If you can repro, could you try adding CHECKs in the ServiceWorkerRegistration and ServiceWorkerVersion destructors to see if they are not getting destroyed on the IO thread?
,
Sep 13 2017
falken: I already have logs in the destructors for both ServiceWorkerRegistration and ServiceWorkerVersion and they are not getting hit before the crash.
Interestingly, we just managed to symbolize a stack locally and interestingly got stuff about a WebServiceWorkerRegistrationImpl instead of the PrefRegistry/HttpAuth stuff (still not sure if these are the real symbols though):
Stack Trace:
RELADDR FUNCTION
FILE:LINE
000a8a23 ~LogMessage
/usr/local/google/home/awdf/repos/clankium/src/base/logging.cc:560
0051230f base::subtle::RefCountedBase::Release() const
/usr/local/google/home/awdf/repos/clankium/src/base/memory/ref_counted.h:95
00553321
base::RefCounted<content::WebServiceWorkerRegistrationImpl>::Release()
const
/usr/local/google/home/awdf/repos/clankium/src/base/memory/ref_counted.h:308
v------>
scoped_refptr<content::WebServiceWorkerRegistrationImpl>::Release(content::WebServiceWorkerRegistrationImpl*)
/usr/local/google/home/awdf/repos/clankium/src/base/memory/ref_counted.h:640
00552379 ~scoped_refptr
/usr/local/google/home/awdf/repos/clankium/src/base/memory/ref_counted.h:535
v------> ~__tuple_leaf
/usr/local/google/home/awdf/repos/clankium/src/third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libcxx/include/tuple:180
00a3b6f3 ~__tuple_impl
/usr/local/google/home/awdf/repos/clankium/src/third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libcxx/include/tuple:383
v------> ~tuple
/usr/local/google/home/awdf/repos/clankium/src/third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libcxx/include/tuple:486
00a3b6dd ~BindState
/usr/local/google/home/awdf/repos/clankium/src/base/bind_internal.h:469
00a3b6c7 base::internal::BindState<void
(content::ServiceWorkerRegistration::*)(base::RepeatingCallback<void
(content::ServiceWorkerStatusCode)> const&,
scoped_refptr<content::ServiceWorkerVersion>,
content::ServiceWorkerStatusCode),
scoped_refptr<content::ServiceWorkerRegistration>,
base::RepeatingCallback<void
(content::ServiceWorkerStatusCode)>,
scoped_refptr<content::ServiceWorkerVersion>
>::Destroy(base::internal::BindStateBase const*)
/usr/local/google/home/awdf/repos/clankium/src/base/bind_internal.h:472
v------>
scoped_refptr<base::internal::BindStateBase>::Release(base::internal::BindStateBase*)
/usr/local/google/home/awdf/repos/clankium/src/base/memory/ref_counted.h:640
0008dd8b ~scoped_refptr
/usr/local/google/home/awdf/repos/clankium/src/base/memory/ref_counted.h:535
v------> ~__tuple_leaf
/usr/local/google/home/awdf/repos/clankium/src/third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libcxx/include/tuple:180
00a45593 ~__tuple_impl
/usr/local/google/home/awdf/repos/clankium/src/third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libcxx/include/tuple:383
v------> ~tuple
/usr/local/google/home/awdf/repos/clankium/src/third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libcxx/include/tuple:486
00a45575 ~BindState
/usr/local/google/home/awdf/repos/clankium/src/base/bind_internal.h:469
00a4555f base::internal::BindState<void
(content::ServiceWorkerStorage::*)(base::RepeatingCallback<void
(content::ServiceWorkerStatusCode)> const&,
content::ServiceWorkerDatabase::RegistrationData
const&, GURL const&,
content::ServiceWorkerDatabase::RegistrationData
const&, std::__ndk1::vector<long long,
std::__ndk1::allocator<long long> > const&,
content::ServiceWorkerDatabase::Status),
base::WeakPtr<content::ServiceWorkerStorage>,
base::RepeatingCallback<void
(content::ServiceWorkerStatusCode)>,
content::ServiceWorkerDatabase::RegistrationData>::Destroy(base::internal::BindStateBase
const*)
/usr/local/google/home/awdf/repos/clankium/src/base/bind_internal.h:472
v------>
scoped_refptr<base::internal::BindStateBase>::Release(base::internal::BindStateBase*)
/usr/local/google/home/awdf/repos/clankium/src/base/memory/ref_counted.h:640
0008dd8b ~scoped_refptr
/usr/local/google/home/awdf/repos/clankium/src/base/memory/ref_counted.h:535
v------> ~__tuple_leaf
/usr/local/google/home/awdf/repos/clankium/src/third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libcxx/include/tuple:180
00a456a3 ~__tuple_impl
/usr/local/google/home/awdf/repos/clankium/src/third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libcxx/include/tuple:383
v------> ~tuple
/usr/local/google/home/awdf/repos/clankium/src/third_party/android_tools/ndk/sources/cxx-stl/llvm-libc++/libcxx/include/tuple:486
00a4568d ~BindState
/usr/local/google/home/awdf/repos/clankium/src/base/bind_internal.h:469
00a45677
base::internal::BindState<void
(*)(content::ServiceWorkerDatabase*,
scoped_refptr<base::SequencedTaskRunner>,
content::ServiceWorkerDatabase::RegistrationData
const&,
std::__ndk1::vector<content::ServiceWorkerDatabase::ResourceRecord,
std::__ndk1::allocator<content::ServiceWorkerDatabase::ResourceRecord>
> const&,
base::RepeatingCallback<void
(GURL const&,
content::ServiceWorkerDatabase::RegistrationData
const&,
std::__ndk1::vector<long long,
std::__ndk1::allocator<long
long> > const&,
content::ServiceWorkerDatabase::Status)>
const&),
content::ServiceWorkerDatabase*,
scoped_refptr<base::SingleThreadTaskRunner>,
content::ServiceWorkerDatabase::RegistrationData,
std::__ndk1::vector<content::ServiceWorkerDatabase::ResourceRecord,
std::__ndk1::allocator<content::ServiceWorkerDatabase::ResourceRecord>
>,
base::RepeatingCallback<void
(GURL const&,
content::ServiceWorkerDatabase::RegistrationData
const&,
std::__ndk1::vector<long long,
std::__ndk1::allocator<long
long> > const&,
content::ServiceWorkerDatabase::Status)>
>::Destroy(base::internal::BindStateBase
const*)
/usr/local/google/home/awdf/repos/clankium/src/base/bind_internal.h:472
v------>
scoped_refptr<base::internal::BindStateBase>::Release(base::internal::BindStateBase*)
/usr/local/google/home/awdf/repos/clankium/src/base/memory/ref_counted.h:640
0008dd8b ~scoped_refptr
/usr/local/google/home/awdf/repos/clankium/src/base/memory/ref_counted.h:535
0008d4df
base::OnceCallback<void
()>::Run() &&
/usr/local/google/home/awdf/repos/clankium/src/base/callback.h:65
000970d7
base::debug::TaskAnnotator::RunTask(char
const*,
base::PendingTask*)
/usr/local/google/home/awdf/repos/clankium/src/base/debug/task_annotator.cc:61
000df2c5
base::internal::TaskTracker::PerformRunTask(std::__ndk1::unique_ptr<base::internal::Task,
std::__ndk1::default_delete<base::internal::Task>
>,
base::internal::Sequence*)
/usr/local/google/home/awdf/repos/clankium/src/base/task_scheduler/task_tracker.cc:335
000df82f
base::internal::TaskTrackerPosix::PerformRunTask(std::__ndk1::unique_ptr<base::internal::Task,
std::__ndk1::default_delete<base::internal::Task>
>,
base::internal::Sequence*)
/usr/local/google/home/awdf/repos/clankium/src/base/task_scheduler/task_tracker_posix.cc:22
000deef3
base::internal::TaskTracker::RunNextTask(base::internal::Sequence*)
/usr/local/google/home/awdf/repos/clankium/src/base/task_scheduler/task_tracker.cc:251
000db62d
base::internal::SchedulerWorker::Thread::ThreadMain()
/usr/local/google/home/awdf/repos/clankium/src/base/task_scheduler/scheduler_worker.cc:73
000e3c33
base::(anonymous
namespace)::ThreadFunc(void*)
/usr/local/google/home/awdf/repos/clankium/src/base/threading/platform_thread_posix.cc:75
0000d173
<unknown>
/system/lib/libc.so
0000d30b
<unknown>
/system/lib/libc.so
➜
,
Sep 13 2017
WebServiceWorkerRegistrationImpl only lives in the renderer process (whereas ServiceWorkerRegistration/Version are in the browser process. So if this is a browser process crash it's probably not the right symbol.
,
Sep 14 2017
I don't have time to keep looking at this so re-marking as Available; hopefully there's a bit more to go on now for whoever picks this up. To summarize, we reckon there's something referenced by some callback being destroyed on the wrong thread. We don't know what or which callback, but it seems related to ServiceWorkerRegistration and ServiceWorkerVersion.
,
Sep 15 2017
Also, for the record, we added lots of logging to try to figure out at what point the crash occurred during PushMessagingTest#testPushPermissionDenied, and found that: - ServiceWorkerStorage::GetUserData runs and posts a task to the database task runner to call ServiceWorkerStorage::GetUserDataInDB on the database task runner thread - The javascript 'subscribe' call in push_messaging_test_android.html (l33) returns a promise successfully. - But ServiceWorkerStorage::GetUserDataInDB never actually executes in the test runs that crash. - And execution never reaches line 34 in push_messaging_test_android.html (ie the promise never resolves with a subscription) before the crash.
,
Sep 15 2017
,
Sep 15 2017
Thanks for all the notes! For the record I also could not find an issue via code inspection. My Linux box is not able to repro the layout tests crashes despite hundreds of iterations and random ordering... still working on setting up the other devices.
,
Jan 19 2018
falken@, since this is Pri1, can we please have a status update on this? What would you like to do with this bug next?
,
Jan 22 2018
P1 is indeed right as this is a disabled test. Unfortunately I don't think I'm going to have time to investigate in the near future.
,
Mar 21 2018
Hi Matt, are you still too busy to look at this? If so please mark it Available and I'll try to find a new owner or take another look myself. Thanks Anita
,
Mar 22 2018
Sorry, yes too busy.
,
Mar 22 2018
,
Aug 2
,
Nov 28
I'm trying to look into this now. I get: org.junit.ComparisonFailure: expected:<[clearCachedVerificationsForTesting] title> but was:<[reset] title> when running the test locally. I don't see how to discover if a crash was the reason for this. I've revived awdf's patch at https://chromium-review.googlesource.com/c/chromium/src/+/1353051 and am trying on the trybots.
,
Nov 28
The trybot fails also with "expected:<[clearCachedVerificationsForTesting] title> but was:<[reset] title>" but I'm not seeing evidence of a crash, unless I just don't know where to look. https://ci.chromium.org/buildbot/tryserver.chromium.android/linux_android_dbg_ng/1470 |
|||||||||||||||
►
Sign in to add a comment |
|||||||||||||||
Comment 1 by anthonyvd@chromium.org
, Apr 3 2017