cronet_unittests_android broken on android_cronet_tester |
|||||||
Issue descriptionI'm getting android_cronet_tester errors, so I ran it on an obviously unrelated CL -- https://crrev.com/c/1189170 Failure results: https://ci.chromium.org/buildbot/tryserver.chromium.android/android_cronet_tester/4141 The failure is always the same: [FATAL:test_suite.cc(118)] Check failed: scheduler_set_before_test_ == TaskScheduler::GetInstance() (0x0 vs. 0x76b3add8) Failing tests: C 95.904s Main [ FAILED ] 32 tests, listed below: C 95.904s Main [ FAILED ] EngineTest.CronetEngineDefaultUserAgent (CRASHED) C 95.904s Main [ FAILED ] EngineTest.InitDifferentEngines (CRASHED) C 95.904s Main [ FAILED ] EngineTest.InvalidPkpParams (CRASHED) C 95.904s Main [ FAILED ] EngineTest.StartCronetEngine (CRASHED) C 95.904s Main [ FAILED ] EngineTest.StartNetLogToFile (CRASHED) C 95.904s Main [ FAILED ] EngineTest.StartResults (CRASHED) C 95.904s Main [ FAILED ] EngineTest.ValidPkpParams (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.CancelRequest (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.FailedRequestHostNotFound (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.InitChecks (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.MultiRedirect (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.PerfTest (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.SimpleGet (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.SimpleRequest (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.TestCancel (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.UploadChangesDefaultMethod (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.UploadChunked (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.UploadChunkedLastReadZeroLengthBody (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.UploadEmptyBodySync (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.UploadFailsWithoutInitializingStream (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.UploadMultiplePiecesAsync (CRASHED) C 95.904s Main [ FAILED ] UrlRequestTest.UploadMultiplePiecesSync (CRASHED) C 95.905s Main [ FAILED ] UrlRequestTest.UploadReadFailAsync (CRASHED) C 95.905s Main [ FAILED ] UrlRequestTest.UploadReadFailSync (CRASHED) C 95.905s Main [ FAILED ] UrlRequestTest.UploadRedirectAsync (CRASHED) C 95.905s Main [ FAILED ] UrlRequestTest.UploadRedirectSync (CRASHED) C 95.905s Main [ FAILED ] UrlRequestTest.UploadRewindFailAsync (CRASHED) C 95.905s Main [ FAILED ] UrlRequestTest.UploadRewindFailSync (CRASHED) C 95.905s Main [ FAILED ] UrlRequestTest.UploadSync (CRASHED) C 95.905s Main [ FAILED ] UrlRequestTest.UploadWithBadLength (CRASHED) C 95.905s Main [ FAILED ] UrlRequestTest.UploadWithBadLengthBufferAligned (CRASHED) C 95.905s Main [ FAILED ] UrlRequestTest.UploadWithSetMethod (CRASHED) A somewhat legible stacktrace: C 95.901s Main 00ed901b logging::LogMessage::~LogMessage() ../../base/logging.cc:599:29 C 95.901s Main 00f4d813 base::(anonymous namespace)::CheckForLeakedGlobals::OnTestEnd(testing::TestInfo const&) ../../base/test/test_suite.cc:118:5 C 95.901s Main 003d7345 testing::internal::TestEventRepeater::OnTestEnd(testing::TestInfo const&) ../../third_party/googletest/src/googletest/src/gtest.cc:3378:1 C 95.901s Main 003d6119 testing::TestInfo::Run() ../../third_party/googletest/src/googletest/src/gtest.cc:2693:13 C 95.901s Main 003d62e5 testing::TestCase::Run() ../../third_party/googletest/src/googletest/src/gtest.cc:2800:28 C 95.901s Main 003d9d31 testing::internal::UnitTestImpl::RunAllTests() ../../third_party/googletest/src/googletest/src/gtest.cc:5124:43 C 95.901s Main 003d9b93 testing::UnitTest::Run() ../../third_party/googletest/src/googletest/src/gtest.cc:4733:10 C 95.901s Main 00f4ceff base::TestSuite::Run() ../../base/test/test_suite.cc:295:16 C 95.901s Main 0037df89 int base::internal::Invoker<base::internal::BindState<int (base::TestSuite::*)(), base::internal::UnretainedWrapper<base::TestSuite> >, int ()>::RunImpl<int (base::TestSuite::*)(), std::__ndk1::tuple<base::internal::UnretainedWrapper<base::TestSuite> >, 0u>(int (base::TestSuite::*&&)(), std::__ndk1::tuple<base::internal::UnretainedWrapper<base::TestSuite> >&&, std::__ndk1::integer_sequence<unsigned int, 0u>) ../../base/bind_internal.h:689:12 C 95.901s Main 003bed49 base::OnceCallback<bool ()>::Run() && ../../base/callback.h:99:12 C 95.901s Main v------> base::(anonymous namespace)::LaunchUnitTestsInternal(base::OnceCallback<int ()>, unsigned int, int, bool, base::OnceCallback<void ()>) ../../base/test/launcher/unit_test_launcher.cc:200:36 C 95.901s Main 00f573fb base::LaunchUnitTests(int, char**, base::OnceCallback<int ()>) ../../base/test/launcher/unit_test_launcher.cc:575:0 C 95.901s Main 0037df3b main ../../components/cronet/run_all_unittests.cc:16:10 C 95.901s Main v------> testing::android::JNI_NativeTest_RunTests(_JNIEnv*, base::android::JavaParamRef<_jobject*> const&, base::android::JavaParamRef<_jstring*> const&, base::android::JavaParamRef<_jstring*> const&, base::android::JavaParamRef<_jstring*> const&, base::android::JavaParamRef<_jobject*> const&, base::android::JavaParamRef<_jstring*> const&) ../../testing/android/native_test/native_test_launcher.cc:131:3 C 95.901s Main 00ebd243 Java_org_chromium_native_1test_NativeTest_nativeRunTests
,
Aug 27
Tentative fix: https://crrev.com/c/1189712
,
Aug 27
The straightforward version here breaks on 32-bit Windows 7. I tested the 32-bit build on my Windows 10 machine and it passes there. The failure is in //base/win/com_init_check_hook.cc and suggests that something else installed a hook on CoCreateInstance (COM). This might be a similar problem to Issue 737090 . [0827/011125.297:ERROR:stack_trace_win.cc(134)] SymInitialize failed: 87 [0827/011125.297:FATAL:com_init_check_hook.cc(164)] Check failed: false. CoCreateInstance appears to be previously patched. <e9 15 b2 79 8b eb f9> We were going to write <e9 a5 9c 6c f9 eb f9> Error initializing symbols (87). Dumping unresolved backtrace: 6F319BA0 6F31964D 6F29A620 6F3B34C7 6F329DEB 6F329C5A 6F2B2644 6F2510BB 6F262097 6F267DC8 0139FA83 0137F5A2 01380C3E 013956B0 01395C82 01396134 0139C2E5 0139BF69 013D8444 013DA89B 013D9BA5 013D9A77 01371085 0151839C 7603343D 77429832 77429805 Received fatal exception EXCEPTION_BREAKPOINT Backtrace: base::debug::BreakDebugger [0x6F31962C+12] logging::LogMessage::~LogMessage [0x6F29A912+850] base::win::ComInitCheckHook::ComInitCheckHook [0x6F3B34C7+1447] base::internal::TaskSchedulerImpl::TaskSchedulerImpl [0x6F329DEB+379] base::internal::TaskSchedulerImpl::TaskSchedulerImpl [0x6F329C5A+74] base::TaskScheduler::CreateAndStartWithDefaultParams [0x6F2B2644+36] cronet::EnsureInitialized [0x6F2510BB+187] cronet::Cronet_EngineImpl::StartWithParams [0x6F262097+37] Cronet_Engine_StartWithParams [0x6F267DC8+93] cronet::test::CreateTestEngine [0x0139FA83+211] base::internal::Invoker<base::internal::BindState<void (__cdecl*)(void const *),void const *>,void __cdecl(void)>::RunOnce [0x0137F5A2+12178] testing::PrintToString<__int64> [0x01380C3E+382] testing::Test::Run [0x013956B0+192] testing::TestInfo::Run [0x01395C82+210] testing::TestCase::Run [0x01396134+244] testing::internal::UnitTestImpl::RunAllTests [0x0139C2E5+629] testing::UnitTest::Run [0x0139BF69+153] base::TestSuite::Run [0x013D8444+100] base::OnceCallback<int __cdecl(void)>::Run [0x013DA89B+43] base::LaunchUnitTests [0x013D9BA5+485] base::LaunchUnitTests [0x013D9A77+183] main [0x01371085+133] __scrt_common_main_seh [0x0151839C+250] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283) BaseThreadInitThunk [0x7603343D+18] RtlInitializeExceptionChain [0x77429832+99] RtlInitializeExceptionChain [0x77429805+54]
,
Aug 27
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/caf38a109f73e6b74ae616a3c5df2af715856284 commit caf38a109f73e6b74ae616a3c5df2af715856284 Author: Victor Costan <pwnall@chromium.org> Date: Mon Aug 27 09:49:45 2018 cronet: Add ScopedTaskEnvironment to native tests. This is needed to fix cronet_unittests_android on the android_cronet_tester bot after https://crrev.com/c/1187783. TBR=mef Bug: 877868 , 875486 Cq-Include-Trybots: luci.chromium.try:ios-simulator-cronet;master.tryserver.chromium.android:android_cronet_tester Change-Id: If59e993e3e6b19cb54f04c2259014c34d6ad4592 Reviewed-on: https://chromium-review.googlesource.com/1189712 Reviewed-by: Victor Costan <pwnall@chromium.org> Commit-Queue: Victor Costan <pwnall@chromium.org> Cr-Commit-Position: refs/heads/master@{#586221} [modify] https://crrev.com/caf38a109f73e6b74ae616a3c5df2af715856284/components/cronet/native/test/engine_test.cc [modify] https://crrev.com/caf38a109f73e6b74ae616a3c5df2af715856284/components/cronet/native/test/url_request_test.cc
,
Aug 27
It looks like the crash on Widows is in the initialization of COM because some hook has already been patched. It looks like this may be a real issue, as tests should be able to initialize a ScopedTaskEnvironment. I think a new bug should be filled with OS=Windows, to investigate why the crash happens on Windows.
,
Aug 27
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8c752f9eff8445ee3743f20a812e4b0e578ffa3f commit 8c752f9eff8445ee3743f20a812e4b0e578ffa3f Author: Markus Heintz <markusheintz@chromium.org> Date: Mon Aug 27 15:13:01 2018 Revert "cronet: Add ScopedTaskEnvironment to native tests." This reverts commit caf38a109f73e6b74ae616a3c5df2af715856284. Reason for revert: Finit identified this CL as reason for breaking cronet_tests: - cronet_tests failing on chromium.mac/Mac10.13 Tests (dbg) - cronet_tests failing on chromium.linux/Linux Tests (dbg)(1) - cronet_tests failing on chromium.linux/Linux Tests (dbg)(1)(32) e.g. https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Tests%20%28dbg%29%281%29/74010 Original change's description: > cronet: Add ScopedTaskEnvironment to native tests. > > This is needed to fix cronet_unittests_android on the > android_cronet_tester bot after https://crrev.com/c/1187783. > > TBR=mef > > Bug: 877868 , 875486 > Cq-Include-Trybots: luci.chromium.try:ios-simulator-cronet;master.tryserver.chromium.android:android_cronet_tester > Change-Id: If59e993e3e6b19cb54f04c2259014c34d6ad4592 > Reviewed-on: https://chromium-review.googlesource.com/1189712 > Reviewed-by: Victor Costan <pwnall@chromium.org> > Commit-Queue: Victor Costan <pwnall@chromium.org> > Cr-Commit-Position: refs/heads/master@{#586221} TBR=wez@chromium.org,mef@chromium.org,pwnall@chromium.org Change-Id: Id1975c8bec0d0c863bbc36c23516e0c1cdc47ace No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 877868 , 875486 Cq-Include-Trybots: luci.chromium.try:ios-simulator-cronet;master.tryserver.chromium.android:android_cronet_tester Reviewed-on: https://chromium-review.googlesource.com/1190622 Reviewed-by: Markus Heintz <markusheintz@chromium.org> Commit-Queue: Markus Heintz <markusheintz@chromium.org> Cr-Commit-Position: refs/heads/master@{#586265} [modify] https://crrev.com/8c752f9eff8445ee3743f20a812e4b0e578ffa3f/components/cronet/native/test/engine_test.cc [modify] https://crrev.com/8c752f9eff8445ee3743f20a812e4b0e578ffa3f/components/cronet/native/test/url_request_test.cc
,
Aug 27
,
Aug 27
Issue 878001 has been merged into this issue.
,
Aug 27
I don't have any immediate plans past trying to land https://crrev.com/c/1189143 to help with future investigations. Marking available so someone with more knowledge / bandwidth can pick it up. Bumped to P1, because I don't think it's OK that it's impossible to land CLs where post-upload hooks add android_cronet_tester to the Cq-Include-Trybots: footer.
,
Aug 27
I'll take this.
,
Aug 27
it seems that CheckForLeakedGlobals is actually finding some real leak in cronet tests. wez@ has disabled it for desktop platforms, and disabling it for Android fixes the issue locally. I'd like to disable the check while I investigate proper fix: https://chromium-review.googlesource.com/c/chromium/src/+/1191247.
,
Aug 27
Re #11: I disabled the check only for cronet_tests, and only for component builds, because the cronet_tests executable and libcronet.so end up sharing components such as //base, //net etc in component builds - since the library assumes one-off initialization of the TaskScheduler, and is never unloaded by the cronet_tests binary, we would need to add a test-only reset API if we wanted to switch to having the suite use a ScopedTaskEnvironment. I'm not familiar with the Android tests that are failing, but if they're also a component-build config then they likely just need the check disabling as well.
,
Aug 27
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/212dfda2c3c7330796e668866fa9f0e5050ba171 commit 212dfda2c3c7330796e668866fa9f0e5050ba171 Author: Victor Costan <pwnall@chromium.org> Date: Mon Aug 27 19:43:54 2018 base: Add more diagnostics to COM hook injection. On 32-bit Windows builds where DCHECK is enabled, ComInitCheckHook hot-patches CoCreateInstance() to DCHECK that it is called in appropriate circumstances. The hot-patching method can only be used once (details in base/win/com_init_check_hook.cc), and Chrome's usage may conflict with other drivers that may want to hot-patch the same method (example in https://crbug.com/737090 ). This CL adds answers to the following questions that may arise when investigating hot-patch failures. 1) Is hot-patching failing due to a bug where the HookManager singleton is instantiated twice? This can happen if base is linked in twice. This question is answered by printing the bytes that would be written to the hot-patch area, so they can be compared with the current values. 2) Is hot-patching failing due to a previous failure to revert the hot-patch? This question is answered by DCHECKing that reverting the hot-patch never fails. (The code previously just assumed that.) Bug: 877868 Change-Id: I94f81b3816417447d31fc700c6b81d8325222d1a Reviewed-on: https://chromium-review.googlesource.com/1189143 Commit-Queue: Victor Costan <pwnall@chromium.org> Reviewed-by: Robert Liao <robliao@chromium.org> Cr-Commit-Position: refs/heads/master@{#586375} [modify] https://crrev.com/212dfda2c3c7330796e668866fa9f0e5050ba171/base/win/com_init_check_hook.cc [modify] https://crrev.com/212dfda2c3c7330796e668866fa9f0e5050ba171/base/win/com_init_check_hook_unittest.cc
,
Aug 27
I gtg, so making it available.
,
Aug 27
Auto-assigning; I'll (temporarily) add a pre-condition to disable the check for cronet_unittests under Android.
,
Aug 27
The problem here appears to be that Cronet is statically-linked into the cronet_unittests_android binary, causing the test framework and library to share global resources. This means that the cronet_native_unittests differ from cronet_common_unittests and cronet_native_unittests, in whether we can expect them not to create (and leak) TaskScheduler. Conceptually the cronet_tests/cronet_unittests seems equivalent to the browser_tests/unit_tests split - one runs integration tests, the other runs tests which we expect to be hermetic. IMO the simplest fix for this issue would be to have separate _unittests vs _tests binaries under Android, in the same manner as we do for desktop platforms, and provide a Cronet-native-tests run_all_unittests.cc that configures TestSuite to skip the check for leaked globals.
,
Aug 27
I don't think cronet_unittests_android care about TaskScheduler at all. Is the check for leaked globals valuable? If yes, then we should add some kind of 'Shutdown' method to Cronet that would free the global TaskScheduler and enable the check. If no, then I think we can just disable that check for all cronet [unit]tests. It looks like currently it only detects leaked TaskScheduler, is there an intention to beef it up? How is it better than leak sanitizer?
,
Aug 28
Re #17: Yes, the check was motivated by a desire to avoid having tests that are flaky because some earlier test happens to break them; these aren't leaks in the lost-memory sense, but in the sense of pollution of global state that in some way interferes with correct execution of tests which run later in the same process. Currently we check for TaskScheduler, but other things may get added as we encounter specific cases, to prevent them regressing once fixed (e.g. NetworkChangeNotifier is one that I hope to add, albeit at a different layer). I've uploaded two CLs: - https://chromium-review.googlesource.com/c/chromium/src/+/1192112 splits cronet_unittests_android to have a separate cronet_tests_android, for the integration tests. - https://chromium-review.googlesource.com/c/chromium/src/+/1192211 moves run_all_unittests.cc to be an explicit include of each test suite. Combined these allow us to retain leaked-globals checks for the unit-test suite while disabling them on the integration suite.
,
Aug 28
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/aacbf6768923294a26ea7568b93c15d3377533b3 commit aacbf6768923294a26ea7568b93c15d3377533b3 Author: Misha Efimov <mef@chromium.org> Date: Tue Aug 28 21:42:27 2018 [Cronet] Use common run_all_unittests.cc in cronet_test on iOS. Remove custom cronet_test_runner.mm and make it consistent with other platforms. Bug: 877868 Cq-Include-Trybots: luci.chromium.try:ios-simulator-cronet Change-Id: Ibee3f904cd658330a8887596c6c2014dcbd75c46 Reviewed-on: https://chromium-review.googlesource.com/1194492 Commit-Queue: Misha Efimov <mef@chromium.org> Reviewed-by: Wez <wez@chromium.org> Cr-Commit-Position: refs/heads/master@{#586860} [modify] https://crrev.com/aacbf6768923294a26ea7568b93c15d3377533b3/components/cronet/ios/test/BUILD.gn [delete] https://crrev.com/4184918e72d3095810e9bfdd73666f6580fe10d6/components/cronet/ios/test/cronet_test_runner.mm
,
Aug 28
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8b25664a0b3ee1289fa18add0396732fd8abe95f commit 8b25664a0b3ee1289fa18add0396732fd8abe95f Author: Wez <wez@chromium.org> Date: Tue Aug 28 23:51:10 2018 Split cronet-tests out of Cronet unit-tests target. This separates the Cronet native API tests from the unit-tests, allowing the TestSuite instances under which they run to be configured differently. run_all_unittests.cc is also moved from being part of the cronet_common_unittests source_set() to being an explicit source entry for each test() executable, to allow its behaviour to be configured. Bug: 877868 Cq-Include-Trybots: luci.chromium.try:ios-simulator-cronet;master.tryserver.chromium.android:android_cronet_tester Change-Id: I0b09a6fbefaa1da6f5f07cabf95d9daa76330740 Reviewed-on: https://chromium-review.googlesource.com/1192112 Commit-Queue: Wez <wez@chromium.org> Reviewed-by: Misha Efimov <mef@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#586938} [modify] https://crrev.com/8b25664a0b3ee1289fa18add0396732fd8abe95f/components/cronet/BUILD.gn [modify] https://crrev.com/8b25664a0b3ee1289fa18add0396732fd8abe95f/components/cronet/android/BUILD.gn [modify] https://crrev.com/8b25664a0b3ee1289fa18add0396732fd8abe95f/components/cronet/ios/BUILD.gn [modify] https://crrev.com/8b25664a0b3ee1289fa18add0396732fd8abe95f/components/cronet/run_all_unittests.cc [modify] https://crrev.com/8b25664a0b3ee1289fa18add0396732fd8abe95f/testing/buildbot/chromium.android.fyi.json [modify] https://crrev.com/8b25664a0b3ee1289fa18add0396732fd8abe95f/testing/buildbot/chromium.android.json [modify] https://crrev.com/8b25664a0b3ee1289fa18add0396732fd8abe95f/testing/buildbot/gn_isolate_map.pyl [modify] https://crrev.com/8b25664a0b3ee1289fa18add0396732fd8abe95f/testing/buildbot/test_suites.pyl [modify] https://crrev.com/8b25664a0b3ee1289fa18add0396732fd8abe95f/testing/buildbot/waterfalls.pyl
,
Aug 28
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by pwnall@chromium.org
, Aug 27