Consistent DCHECK crash in net::BrokenAlternativeServices::IsAlternativeServiceBroken |
||||||
Issue descriptionException Type: EXC_BREAKPOINT (SIGTRAP) Exception Codes: 0x0000000000000102, 0x000000002498e4dc Triggered by Thread: 13 Thread 13 name: Chrome Network IO Thread Thread 13 Crashed: 0 Cronet 0x05d1d972 base::debug::BreakDebugger() + 18 1 Cronet 0x05d56bc6 logging::LogMessage::~LogMessage() + 2616 2 Cronet 0x05d54bfa logging::LogMessage::~LogMessage() + 16 3 Cronet 0x060e102c net::BrokenAlternativeServices::IsAlternativeServiceBroken(net::AlternativeService const&) const + 208 4 Cronet 0x06151654 net::HttpServerPropertiesImpl::IsAlternativeServiceBroken(net::AlternativeService const&) const + 20 5 Cronet 0x0615193c net::HttpServerPropertiesImpl::GetAlternativeServiceInfoAsValue() const + 670 6 Cronet 0x061c077a net::GetNetInfo(net::URLRequestContext*, int) + 9510 7 Cronet 0x061c3e2c net::WriteToFileNetLogObserver::StopObserving(net::URLRequestContext*) + 206 8 Cronet 0x05cabeaa cronet::CronetEnvironment::StopNetLogOnNetworkThread(base::WaitableEvent*) + 342 9 Cronet 0x05cb2d64 void base::internal::FunctorTraits<void (cronet::CronetEnvironment::*)(base::WaitableEvent*), void>::Invoke<cronet::CronetEnvironment*, base::WaitableEvent* const&>(void (cronet::CronetEnvironment::*)(base::WaitableEvent*), cronet::CronetEnvironment*&&, base::WaitableEvent* const&&&) + 92 10 Cronet 0x05cb2cee void base::internal::InvokeHelper<false, void>::MakeItSo<void (cronet::CronetEnvironment::* const&)(base::WaitableEvent*), cronet::CronetEnvironment*, base::WaitableEvent* const&>(void (cronet::CronetEnvironment::* const&&&)(base::WaitableEvent*), cronet::CronetEnvironment*&&, base::WaitableEvent* const&&&) + 40 11 Cronet 0x05cb2cc2 void base::internal::Invoker<base::internal::BindState<void (cronet::CronetEnvironment::*)(base::WaitableEvent*), base::internal::UnretainedWrapper<cronet::CronetEnvironment>, base::WaitableEvent*>, void ()>::RunImpl<void (cronet::CronetEnvironment::* const&)(base::WaitableEvent*), std::__1::tuple<base::internal::UnretainedWrapper<cronet::CronetEnvironment>, base::WaitableEvent*> const&, 0ul, 1ul>(void (cronet::CronetEnvironment::* const&&&)(base::WaitableEvent*), std::__1::tuple<base::internal::UnretainedWrapper<cronet::CronetEnvironment>, base::WaitableEvent*> const&&&, base::IndexSequence<0ul, 1ul>) + 76 12 Cronet 0x05cb2c2c base::internal::Invoker<base::internal::BindState<void (cronet::CronetEnvironment::*)(base::WaitableEvent*), base::internal::UnretainedWrapper<cronet::CronetEnvironment>, base::WaitableEvent*>, void ()>::Run(base::internal::BindStateBase*) + 24 13 Cronet 0x05cce1ee base::Callback<void (), (base::internal::CopyMode)0, (base::internal::RepeatMode)0>::Run() + 138 14 Cronet 0x05d1f598 base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) + 520 15 Cronet 0x05d6cad4 base::MessageLoop::RunTask(base::PendingTask*) + 572 16 Cronet 0x05d6cde2 base::MessageLoop::DeferOrRunPendingTask(base::PendingTask) + 48 17 Cronet 0x05d6d362 base::MessageLoop::DoWork() + 378 18 Cronet 0x05ec04aa base::MessagePumpCFRunLoopBase::RunWork() + 158 19 Cronet 0x05ec0406 ___ZN4base24MessagePumpCFRunLoopBase13RunWorkSourceEPv_block_invoke + 18 20 Cronet 0x05eb47c2 base::mac::CallWithEHFrame(void () block_pointer) + 22 21 Cronet 0x05ebfe20 base::MessagePumpCFRunLoopBase::RunWorkSource(void*) + 86 22 CoreFoundation 0x251a7a66 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 14 23 CoreFoundation 0x251a7656 __CFRunLoopDoSources0 + 454 24 CoreFoundation 0x251a59be __CFRunLoopRun + 806 25 CoreFoundation 0x250f4288 CFRunLoopRunSpecific + 516 26 CoreFoundation 0x250f407c CFRunLoopRunInMode + 108 27 Foundation 0x2594157c -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 268 28 Cronet 0x05ec0e8c base::MessagePumpNSRunLoop::DoRun(base::MessagePump::Delegate*) + 180 29 Cronet 0x05ebf836 base::MessagePumpCFRunLoopBase::Run(base::MessagePump::Delegate*) + 54 30 Cronet 0x05d6c6a0 base::MessageLoop::Run() + 168 31 Cronet 0x05d6c6bc non-virtual thunk to base::MessageLoop::Run() + 24 32 Cronet 0x05dc2800 base::RunLoop::Run() + 194 33 Cronet 0x05e2d710 base::Thread::Run(base::RunLoop*) + 266 34 Cronet 0x05e2e01e base::Thread::ThreadMain() + 1642 35 Cronet 0x05e1983c base::(anonymous namespace)::ThreadFunc(void*) + 328 36 libsystem_pthread.dylib 0x24f18c7e _pthread_body + 138 37 libsystem_pthread.dylib 0x24f18bf2 _pthread_start + 110 38 libsystem_pthread.dylib 0x24f16a08 thread_start + 8
,
Jul 7 2017
Hi Yixin, Since you were the last who modified the BrokenAlternativeServices code could you please take a look at the crash and let us why it can happen and whether we are doing something wrong in Cronet? It consistently happens when we try to stop the netlog in one of our embedder applications. The release version works fine. Please feel free to reassign the bug back to me. Thanks, Andrei
,
Jul 7 2017
Hard to say without knowing what this application is doing. It looks like an alternative service associated with a blank host (a SchemeHostPort where host_ is empty string) was added to HttpServerPropertiesImpl. The offending DCHECK existed before my change; my change simply moved it from HttpServerPropertiesImpl to BrokenAlternativeServices, so I don't think my change is causing this. Is there any way to confirm that? zhongyi@, can you add anything?
,
Jul 7 2017
Here's what our Cronet configuration code looks like:
sQuicHosts = @[@"foo.googlevideo.com", @"foo.c.youtube.com"]
for (NSString *host in sQuicHosts) {
[Cronet addQuicHint:host port:443 altPort:443];
}
[Cronet setQuicEnabled:YES];
[Cronet setHttp2Enabled:YES];
[Cronet setUserAgent:YTUserAgent() partial:NO];
[Cronet setHttpCacheType:CRNHttpCacheTypeDisabled];
[Cronet start];
,
Jul 7 2017
Oh, correction, sQuicHosts also contains "https://youtubei.googleapis.com/" I wonder if the "https://" prefix is causing the problem. I'll try removing this entry and see if it does anything.
,
Jul 7 2017
Yes, that was the bug. When I replaced "https://youtubei.googleapis.com/" with "youtubei.googleapis.com" we no longer hit the assertion. Would it be possible to add the bad host to the debug message? That would have made this way easier to debug. Also, it seems really weird to me that the crash occurs upon stopping the log rather than at some other point, like during configuration or when starting Cronet. We wouldn't have even noticed that the hosts were misconfigured unless we happened to use the logging functionality.
,
Jul 7 2017
Hrm, I've also found that we pass empty host when we add quic hints: https://cs.chromium.org/chromium/src/components/cronet/ios/cronet_environment.mm?rcl=ca205ddf1a960d002da75bbcdd8243757a77e817&l=340 and appear to ignore altPort completely, so there seem to be several bugs on Cronet side.
,
Jul 7 2017
Lily has a CL to refactor Cronet configuration, and that should fix this issue.
,
Jul 7 2017
Tracing bottom up, the DCHECK fires as the alternative_service has an empty host, which we thought should be replace by server.host() which possibly is also an empty host. https://cs.chromium.org/chromium/src/net/http/http_server_properties_impl.cc?type=cs&l=503 in HttpServerPropertiesImpl::GetAlternativeServiceInfoAsValue() { for (const auto& alternative_service_map_item : alternative_service_map_) { std::unique_ptr<base::ListValue> alternative_service_list( new base::ListValue); const url::SchemeHostPort& server = alternative_service_map_item.first; ... if (alternative_service.host.empty()) { alternative_service.host = server.host(); } if (IsAlternativeServiceBroken(alternative_service)) { alternative_service_string.append(" (broken)"); } ... } This could source from the SetAlternativeServices. Combing with quic hints setup in cronet: CronetEnvironment::InitializeOnNetworkThread() { // ... for (const auto& quic_hint : quic_hints_) { net::AlternativeService alternative_service(net::kProtoQUIC, "", quic_hint.port()); url::SchemeHostPort quic_hint_server("https", quic_hint.host(), quic_hint.port()); http_server_properties->SetQuicAlternativeService( quic_hint_server, alternative_service, base::Time::Max(), net::QuicVersionVector()); } // ... } We have empty host specified when set up alternative service, which was then saved to alternative_service_map_.
,
Jul 7 2017
OH, that empty host is in alternative service, shouldn't cause issues here.
Well, the quic_hint_server is added via
void CronetEnvironment::AddQuicHint(const std::string& host,
int port,
int alternate_port) {
DCHECK(port == alternate_port);
quic_hints_.push_back(net::HostPortPair(host, port));
}
There was no check/processing on the host field, which might take an incorrect host and cause issues.
,
Jul 13 2017
lilyhoughton@: I think the current crash at net::BrokenAlternativeServices::IsAlternativeServiceBroken is just a trigger. The root cause is that when setting QUIC alternative service, the server field is empty and gets set to |alternative_service_map_|. This is a parsing failure from cronet when callsite mistakenly append "https://" prefix in the host field, this is not handled properly. You could verify this by adding DCHECKs when an entry is added to |quic_hints_| and when SetQuicAlternativeServices. https://cs.chromium.org/chromium/src/components/cronet/ios/cronet_environment.mm?type=cs&q=CronetEnvironment::InitializeOnNetworkThrea&sq=package:chromium&l=228 https://cs.chromium.org/chromium/src/components/cronet/ios/cronet_environment.mm?type=cs&q=CronetEnvironment::InitializeOnNetworkThrea&sq=package:chromium&l=358
,
Jul 13 2017
To repro the crash, have you try enable NetLog? It seems like GetAlternativeServiceInfoAsValue will be invoked in net logging. You might be able to trigger the crash if enabled.
,
Aug 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/5a1a911b0e131701940bacb3fcbef6951e5a6e64 commit 5a1a911b0e131701940bacb3fcbef6951e5a6e64 Author: Lily Houghton <lilyhoughton@chromium.org> Date: Thu Aug 10 17:21:55 2017 [Cronet] Add check to addQuicHint This should help fix Youtube's crash, and hopefully prevent similar problems in the future. Bug: 739979 Change-Id: Ifd0099439b1e33e85da4f73a522b2ba5c5bc063b Reviewed-on: https://chromium-review.googlesource.com/576287 Reviewed-by: Misha Efimov <mef@chromium.org> Commit-Queue: Lily Houghton <lilyhoughton@chromium.org> Cr-Commit-Position: refs/heads/master@{#493440} [modify] https://crrev.com/5a1a911b0e131701940bacb3fcbef6951e5a6e64/components/cronet/ios/Cronet.h [modify] https://crrev.com/5a1a911b0e131701940bacb3fcbef6951e5a6e64/components/cronet/ios/Cronet.mm [modify] https://crrev.com/5a1a911b0e131701940bacb3fcbef6951e5a6e64/components/cronet/ios/cronet_environment.mm [modify] https://crrev.com/5a1a911b0e131701940bacb3fcbef6951e5a6e64/components/cronet/ios/test/BUILD.gn [add] https://crrev.com/5a1a911b0e131701940bacb3fcbef6951e5a6e64/components/cronet/ios/test/cronet_quic_test.mm
,
Sep 14 2017
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by mmenke@chromium.org
, Jul 7 2017