Issue metadata
Sign in to add a comment
|
Chrome_Android: Crash Report - [ThreadWatcher IO hang] base::internal::IncomingTaskQueue::AddToIncomingQueue |
||||||||||||||||||||
Issue descriptionreporter:pbommana@google.com Magic Signature: [ThreadWatcher IO hang] base::internal::IncomingTaskQueue::AddToIncomingQueue Crash link: https://crash.corp.google.com//browse?q=product.name%3D'Chrome_Android'%20%20AND%20custom_data.ChromeCrashProto.ptype%3D'browser'%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D'%5BThreadWatcher%20IO%20hang%5D%20base%3A%3Ainternal%3A%3AIncomingTaskQueue%3A%3AAddToIncomingQueue'%20AND%20ReportID%3D'e3d680fb500cc583'&sql_dialect=googlesql&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#3 ------------------------------------------------------------------------------- Sample Report ------------------------------------------------------------------------------- Product name: Chrome_Android Magic Signature : [ThreadWatcher IO hang] base::internal::IncomingTaskQueue::AddToIncomingQueue Product Version: 66.0.3332.0 Process type: browser Report ID: e3d680fb500cc583 Report Url: https://crash.corp.google.com/e3d680fb500cc583 Report Time: 2018-01-26T04:34:26-08:00 Upload Time: 2018-01-26T04:44:10.356-08:00 Uptime: 210873 ms CumulativeProductUptime: 0 ms OS Name: Android OS Version: 0.0.0 Linux 3.18.20-g1e87e45-00001-gc656ccd #1 SMP PREEMPT Fri Mar 10 11:04:17 KST 2017 armv7l CPU Architecture: arm CPU Info: ARMv7 ARM part(0x4100d030) features: half,thumb,fastmult,vfpv2,edsp,neon,vfpv3,tls,vfpv4,idiva,idivt ------------------------------------------------------------------------------- Crashing thread: Thread index: 29. Stack Quality: 17%. Thread id: 10304. ------------------------------------------------------------------------------- 0xb6cc8128 (libc.so + 0x00041128) 0xb6ecb99f (libbinder.so + 0x0001e99f) 0xb6ecc055 (libbinder.so + 0x0001f055) 0xb6ecf871 (libbinder.so + 0x00022871) 0xb6ecbb33 (libbinder.so + 0x0001eb33) 0x1377f29e (dalvik-main space (deleted) + 0x00b7f29e) 0xb6ecc20b (libbinder.so + 0x0001f20b) 0xb48f21f2 (libart.so + 0x004001f2) 0xb48f2206 (libart.so + 0x00400206) 0xb48f2d3a (libart.so + 0x00400d3a) 0xb48f21f2 (libart.so + 0x004001f2) 0xb48f2206 (libart.so + 0x00400206) 0xb48f2d3a (libart.so + 0x00400d3a) 0xb6ec7181 (libbinder.so + 0x0001a181) 0xb6ec715f (libbinder.so + 0x0001a15f) 0xb6e46191 (libandroid_runtime.so + 0x0008e191) 0x7079e022 (system@framework@boot.art + 0x00140022) 0x70a31626 (system@framework@boot.art + 0x003d3626) 0x12c0103e (dalvik-main space (deleted) + 0x0000103e) 0x1377f2be (dalvik-main space (deleted) + 0x00b7f2be) 0x75200bd1 (boot.oat + 0x03ef2bd1) 0x74077263 (boot.oat + 0x02d69263) 0x7129b77e (system@framework@boot.art + 0x00c3d77e) 0x12c0103e (dalvik-main space (deleted) + 0x0000103e) 0x1377f2be (dalvik-main space (deleted) + 0x00b7f2be) 0x1377f29e (dalvik-main space (deleted) + 0x00b7f29e) 0x70ed583e (system@framework@boot.art + 0x0087783e) 0x70a4fc4e (system@framework@boot.art + 0x003f1c4e) 0x7079e022 (system@framework@boot.art + 0x00140022) 0x70a31626 (system@framework@boot.art + 0x003d3626) 0x12c0103e (dalvik-main space (deleted) + 0x0000103e) 0x1377f2be (dalvik-main space (deleted) + 0x00b7f2be) 0x1377f29e (dalvik-main space (deleted) + 0x00b7f29e) 0x75200b0d (boot.oat + 0x03ef2b0d) 0x7129b756 (system@framework@boot.art + 0x00c3d756) 0x12c0103e (dalvik-main space (deleted) + 0x0000103e) 0x1377f2be (dalvik-main space (deleted) + 0x00b7f2be) 0x1377f29e (dalvik-main space (deleted) + 0x00b7f29e) 0x70a4fc4e (system@framework@boot.art + 0x003f1c4e) 0x712a1ab6 (system@framework@boot.art + 0x00c43ab6) 0x7076e612 (system@framework@boot.art + 0x00110612) 0x1377f29e (dalvik-main space (deleted) + 0x00b7f29e) 0x12c0103e (dalvik-main space (deleted) + 0x0000103e) 0x1377f2be (dalvik-main space (deleted) + 0x00b7f2be) 0x12ff818e (dalvik-main space (deleted) + 0x003f818e) 0x750ea373 (boot.oat + 0x03ddc373) 0x711e5406 (system@framework@boot.art + 0x00b87406) 0xb6d099c6 (libc++.so + 0x000009c6) 0x1377f29e (dalvik-main space (deleted) + 0x00b7f29e) 0x12ff818e (dalvik-main space (deleted) + 0x003f818e) 0x12ff761e (dalvik-main space (deleted) + 0x003f761e) 0x750de1af (boot.oat + 0x03dd01af) 0x71127c76 (system@framework@boot.art + 0x00ac9c76) 0x9cdf36b5 (libchrome.so - incoming_task_queue.cc: 86) base::internal::IncomingTaskQueue::AddToIncomingQueue(base::Location const&, base::OnceCallback<void ()>, base::TimeDelta, base::Nestable) 0xb48db36b (libart.so + 0x003e936b) 0xb6cd29c7 (libc.so + 0x0004b9c7) 0xb6d099c6 (libc++.so + 0x000009c6) 0xaeb681b6 (dalvik-LinearAlloc (deleted) + 0x0000a1b6) 0xb45de609 (libart.so + 0x000ec609) 0x9f7b2fd1 (base.odex + 0x0052efd1) 0x9f7b2fd1 (base.odex + 0x0052efd1) 0xb6d0a30a (libc++.so + 0x0000130a) 0xaeb681b6 (dalvik-LinearAlloc (deleted) + 0x0000a1b6) 0xb480a7a7 (libart.so + 0x003187a7) 0x9f7b2fd1 (base.odex + 0x0052efd1) 0x9f7b2fd1 (base.odex + 0x0052efd1) 0x9f7b2fd1 (base.odex + 0x0052efd1) 0x9ce14421 (libchrome.so - gurl.cc: 136) GURL::~GURL() 0x9ce14399 (libchrome.so - proxy_config.cc: 200) net::ProxyConfig::~ProxyConfig() 0x9e6d2481 (libchrome.so - data_reduction_proxy_configurator.cc: 45) data_reduction_proxy::DataReductionProxyConfigurator::Enable(data_reduction_proxy::NetworkPropertiesManager const&, std::__ndk1::vector<data_reduction_proxy::DataReductionProxyServer, std::__ndk1::allocator<data_reduction_proxy::DataReductionProxyServer> > const&) 0x9cf511bd (libchrome.so - data_reduction_proxy_config.cc: 639) data_reduction_proxy::DataReductionProxyConfig::OnNetworkChanged(net::NetworkChangeNotifier::ConnectionType) 0x9cf285ab (libchrome.so - bind_internal.h: 350) base::internal::Invoker<base::internal::BindState<void (*)(void (content::GpuDataManagerObserver::*)(base::TerminationStatus), base::TerminationStatus, content::GpuDataManagerObserver*), void (content::GpuDataManagerObserver::*)(base::TerminationStatus), base::TerminationStatus>, void (content::GpuDataManagerObserver*)>::Run(base::internal::BindStateBase*, content::GpuDataManagerObserver*) 0x9cf27d47 (libchrome.so - callback.h: 94) base::ObserverListThreadSafe<breakpad::CrashDumpManager::Observer>::NotifyWrapper(breakpad::CrashDumpManager::Observer*, base::ObserverListThreadSafe<breakpad::CrashDumpManager::Observer>::NotificationData const&) 0x9ce5ebd5 (libchrome.so - bind_internal.h: 294) void base::internal::InvokeHelper<false, void>::MakeItSo<void (ProxyConfigServiceImpl::* const&)(ProxyPrefs::ConfigState, net::ProxyConfig const&), ProxyConfigServiceImpl*, ProxyPrefs::ConfigState const&, net::ProxyConfig const&>(void (ProxyConfigServiceImpl::* const&&&)(ProxyPrefs::ConfigState, net::ProxyConfig const&), ProxyConfigServiceImpl*&&, ProxyPrefs::ConfigState const&&&, net::ProxyConfig const&&&) 0x9cdf3f2b (libchrome.so - callback.h: 65) base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) 0x9cdf3c49 (libchrome.so - message_loop.cc: 399) base::MessageLoop::RunTask(base::PendingTask*) 0x9cdf16d3 (libchrome.so - message_loop.cc: 411) base::MessageLoop::DoWork() 0x9cdf1585 (libchrome.so - message_pump_libevent.cc: 220) base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) 0x9cdf13dd (libchrome.so - run_loop.cc: 130) base::RunLoop::Run() 0x9ce50429 (libchrome.so - browser_thread_impl.cc: 248) content::BrowserThreadImpl::IOThreadRun(base::RunLoop*) 0x9ce50081 (libchrome.so - browser_thread_impl.cc: 283) content::BrowserThreadImpl::Run(base::RunLoop*) 0x9cdee439 (libchrome.so - thread.cc: 338) base::Thread::ThreadMain() 0x9cdee26d (libchrome.so - platform_thread_posix.cc: 75) base::(anonymous namespace)::ThreadFunc(void*) 0xb6cc6bcb (libc.so + 0x0003fbcb) 0xb6cc6bab (libc.so + 0x0003fbab) 0xb6cc6bab (libc.so + 0x0003fbab) 0xb6ca10e5 (libc.so + 0x0001a0e5) 0x9cdee231 (libchrome.so - condition_variable_posix.cc: 76) base::ConditionVariable::Wait()
,
Jan 26 2018
This is an IO thread hang, and there is no way to get anything from a single report, since it could just land on some random thing. This is a bit too over-zealous with filing bugs. Re-open when we get more reports. cc gab, this is the first time I'm seeing a crash due IO thread timeout. What's the timeout here?
,
Jan 26 2018
cc gab, this is the first time I'm seeing a crash due IO thread timeout. What's the timeout here?
,
Jan 28 2018
FYI, all of the hang reports surfacing all of a sudden are real and due to resolving issue 804345 (I tried to be very loud about this -- sent emails to many mailing lists internally including TPMs and test leads -- but looks like I still didn't hit everyone with the news). I guess it's possible the system call hung somehow when scheduling the MessagePump maybe? If it's just one report with this stack it could just be noise (can possibly report a hang if the thread is just super dupper busy and slow), but any reasonable amount of stacks like this should be considered real.
,
Jan 28 2018
Not RBS though because these are not new bugs per se, just newly discovered in long standing system.
,
Jan 28 2018
Gab I tried my best to address most of the newly unearthed [ThreadWatcher IO hang] and [ThreadWatcher UI hang] related crashes example: https://bugs.chromium.org/p/chromium/issues/detail?id=805717#c1 1.The reason to log a bug even with 1 crash is based on email thread that these particular hangs are real. 2.Since I didn't had enough stack to figure out any owner for this crash hence cc'ed Stability sheriff. 3.Reason for adding the release block label with a comment that we aren't sure if crash rate would stay relatively low or spikes.
,
Jan 28 2018
Given that this still appears to be a lone report and that this report has an unreliable stack, I'd say we can WontFix this for now. Thanks for your time all! And thanks Prudhvi for filing all of these, hadn't realized this had already resulted in a mass triage (so smooth I didn't even hear about it :))! Please let me know whether these resulted in useful issues (we know the UI thread ones can be incorrect when waiting on user input to system dialogs and have turned it down again on UI thread while resolving issue 806174).
,
Jan 29 2018
So... what's the timeout? ie how many seconds the IO thread have to hang for this to fire?
,
Jan 29 2018
It has to fail to respond to 9 ping-pongs in a row with a 2 seconds timeout each (so 18 seconds minimum). The multiple ping-pongs ensure that the rest of the system is responsive enough for the WatchdogThread to be able to ping-pong. |
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by pbomm...@chromium.org
, Jan 26 2018Labels: -Type-Bug -Pri-2 ReleaseBlock-Stable TE-CrashTriage Target-66 M-66 FoundIn-66 Stability-Sheriff-Android Pri-1 Type-Bug-Regression