chromeos-4.14: Soft lockups in HCI code |
||||
Issue descriptionObserved on eve running chromeos-4.14 prototype. [90595.093069] wlan0: deauthenticating from 44:48:c1:a6:c9:b2 by local choice (Reason: 3=DEAUTH_LEAVING) [90596.527936] init: timberslide main process (560) killed by TERM signal [90596.546418] init: powerd main process (1140) killed by TERM signal [90596.554014] init: crasal [90596.564185] init: cros-machiness(1657)killed y TEM sinal [90596.583556] init: mtpd main process (1661) terminated with status 143 [90596.591933] init: log-rotate main process (1668) killed by TERM signal [90596.607123] init: cras main process (1724) terminated with status 143 [90596.626451] init: tpm_managerd main process (1857) killed by TERM signal [90596.637073] init: tlsdated main process (2733) killed by TERM signal [90596.646569] init: anomaly-collector main process (2754) killed by TERM signal [90596.655159] init: temp_logger main process (2794) killed by TERM signal [90596.695587] ip_local_port_range: prefer different parity for start/end values. [90601.772225] init: bluetoothd main process (1779) killed by KILL signal [90601.779911] init: firewalld main process (1206) killed by KILL signal [90610.287162] watchdog: BUG: soft lockup - CPU#2 stuck for 11s! [bluetoothd:1790] [90610.288368] watchdog: BUG: soft lockup - CPU#3 stuck for 11s! [kworker/u9:3:10617] [90610.288369] Modules linked in: nls_iso8859_1 nls_cp437 vfat fat uinput snd_soc_dmic snd_soc_kbl_rt5663_rt5514_max98927 snd_soc_hdac_hdmi joydev snd_soc_skl snd_soc_skl_ipc cmac snd_soc_sst_ipc rfcomm snd_soc_sst_dsp snd_soc_sst_match snd_hda_ext_core snd_hda_core btusb btrtl btbcm btintel uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core hid_multitouch snd_soc_max98927 snd_soc_rt5514 snd_soc_rt5663 snd_soc_rt5514_spi snd_soc_rl6231 xt_nat bridge stp llc ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat lzo lzo_compress zram bluetooth ecdh_generic fuse xt_mark ip6table_filter iio_trig_sysfs cros_ec_light_prox cros_ec_sensors cros_ec_sensors_core industrialio_triggered_buffer kfifo_buf industrialio iwlmvm mac80211 r8152 mii iwlwifi cfg80211 [90610.288445] CPU: 3 PID: 10617 Comm: kworker/u9:3 Not tainted 4.14.0 #6 [90610.288446] Hardware name: Google Eve/Eve, BIOS Google_Eve.9584.95.0 09/27/2017 [90610.288474] Workqueue: hci0 hci_power_off [bluetooth] [90610.288476] task: ffff8803cdd9b280 task.stack: ffff8803d80e0000 [90610.288482] RIP: 0010:queued_spin_lock_slowpath+0x74/0x229 [90610.288483] RSP: 0018:ffff8803d80e7b70 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff10 [90610.288486] RAX: 0000000000000101 RBX: ffffffffc036229c RCX: ffffffff8f6ed134 [90610.288487] RDX: 0000000000000100 RSI: 0000000000000007 RDI: ffffffffc036229c [90610.288489] RBP: ffff8803d80e7ba0 R08: dffffc0000000000 R09: 0000000000000008 [90610.288491] R10: ffffed0079c765ea R11: ffff8803ce3b2f4f R12: ffffffffc036229c [90610.288492] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000000 [90610.288494] FS: 0000000000000000(0000) GS:ffff8803eef80000(0000) knlGS:0000000000000000 [90610.288496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90610.288498] CR2: 00005766d182e008 CR3: 000000035ce15004 CR4: 00000000003606e0 [90610.288499] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [90610.288500] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [90610.288501] Call Trace: [90610.288505] queued_read_lock_slowpath+0x53/0x7f [90610.288508] do_raw_read_lock+0x4a/0x4d [90610.288512] _raw_read_lock+0x15/0x17 [90610.288538] hci_send_to_channel+0x2d/0xe4 [bluetooth] [90610.288564] hci_send_monitor_ctrl_event+0x1dd/0x21b [bluetooth] [90610.288590] mgmt_send_event+0x15f/0x179 [bluetooth] [90610.288616] new_settings+0x51/0x6e [bluetooth] [90610.288641] __mgmt_power_off+0x103/0x12e [bluetooth] [90610.288666] hci_dev_do_close+0x3fd/0x521 [bluetooth] [90610.288669] ? _raw_spin_unlock_irq+0xe/0x21 [90610.288672] ? finish_task_switch+0x1ba/0x22a [90610.288696] hci_power_off+0x15/0x17 [bluetooth] [90610.288699] process_one_work+0x2ca/0x4da [90610.288702] worker_thread+0x31c/0x483 [90610.288705] ? create_worker+0x2f9/0x2f9 [90610.288708] kthread+0x221/0x231 [90610.288711] ? kthread_flush_work+0x120/0x120 [90610.288713] ret_from_fork+0x22/0x30 [90610.288714] Code: 89 ca 89 f0 0f 44 d7 f0 0f b1 13 39 f0 74 04 89 c6 eb e2 ff ca 0f 84 b9 01 00 00 48 89 df e8 2a 57 13 00 8b 03 84 c0 74 04 f3 90 <eb> ee 48 89 df e8 97 56 13 00 66 c7 03 01 00 e9 95 01 00 00 49 [90610.288752] Kernel panic - not syncing: softlockup: hung tasks [90610.288755] CPU: 3 PID: 10617 Comm: kworker/u9:3 Tainted: G L 4.14.0 #6 [90610.288756] Hardware name: Google Eve/Eve, BIOS Google_Eve.9584.95.0 09/27/2017 [90610.288780] Workqueue: hci0 hci_power_off [bluetooth] [90610.288781] Call Trace: [90610.288783] <IRQ> [90610.288786] dump_stack+0x4d/0x63 [90610.288790] panic+0x103/0x249 [90610.288793] ? do_raw_spin_unlock+0xc7/0xd1 [90610.288796] watchdog_timer_fn+0x1fa/0x21d [90610.288800] __hrtimer_run_queues+0x18b/0x2a9 [90610.288802] ? watchdog+0x2c/0x2c [90610.288805] hrtimer_interrupt+0xd7/0x1ed [90610.288808] smp_apic_timer_interrupt+0xfe/0x198 [90610.288811] apic_timer_interrupt+0x90/0xa0 [90610.288812] </IRQ> [90610.288815] RIP: 0010:queued_spin_lock_slowpath+0x74/0x229 [90610.288816] RSP: 0018:ffff8803d80e7b70 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff10 [90610.288818] RAX: 0000000000000101 RBX: ffffffffc036229c RCX: ffffffff8f6ed134 [90610.288820] RDX: 0000000000000100 RSI: 0000000000000007 RDI: ffffffffc036229c [90610.288821] RBP: ffff8803d80e7ba0 R08: dffffc0000000000 R09: 0000000000000008 [90610.288823] R10: ffffed0079c765ea R11: ffff8803ce3b2f4f R12: ffffffffc036229c [90610.288825] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000000 [90610.288828] ? queued_spin_lock_slowpath+0x6c/0x229 [90610.288832] queued_read_lock_slowpath+0x53/0x7f [90610.288834] do_raw_read_lock+0x4a/0x4d [90610.288837] _raw_read_lock+0x15/0x17 [90610.288863] hci_send_to_channel+0x2d/0xe4 [bluetooth] [90610.288889] hci_send_monitor_ctrl_event+0x1dd/0x21b [bluetooth] [90610.288915] mgmt_send_event+0x15f/0x179 [bluetooth] [90610.288941] new_settings+0x51/0x6e [bluetooth] [90610.288966] __mgmt_power_off+0x103/0x12e [bluetooth] [90610.288991] hci_dev_do_close+0x3fd/0x521 [bluetooth] [90610.288994] ? _raw_spin_unlock_irq+0xe/0x21 [90610.288996] ? finish_task_switch+0x1ba/0x22a [90610.289021] hci_power_off+0x15/0x17 [bluetooth] [90610.289023] process_one_work+0x2ca/0x4da [90610.289026] worker_thread+0x31c/0x483 [90610.289029] ? create_worker+0x2f9/0x2f9 [90610.289031] kthread+0x221/0x231 [90610.289034] ? kthread_flush_work+0x120/0x120 [90610.289037] ret_from_fork+0x22/0x30 [90610.837836] Modules linked in: nls_iso8859_1 nls_cp437 vfat fat uinput snd_soc_dmic snd_soc_kbl_rt5663_rt5514_max98927 snd_soc_hdac_hdmi joydev snd_soc_skl snd_soc_skl_ipc cmac snd_soc_sst_ipc rfcomm snd_soc_sst_dsp snd_soc_sst_match snd_hda_ext_core snd_hda_core btusb btrtl btbcm btintel uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core hid_multitouch snd_soc_max98927 snd_soc_rt5514 snd_soc_rt5663 snd_soc_rt5514_spi snd_soc_rl6231 xt_nat bridge stp llc ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat lzo lzo_compress zram bluetooth ecdh_generic fuse xt_mark ip6table_filter iio_trig_sysfs cros_ec_light_prox cros_ec_sensors cros_ec_sensors_core industrialio_triggered_buffer kfifo_buf industrialio iwlmvm mac80211 r8152 mii iwlwifi cfg80211 [90610.915233] CPU: 2 PID: 1790 Comm: bluetoothd Tainted: G L 4.14.0 #6 [90610.923600] Hardware name: Google Eve/Eve, BIOS Google_Eve.9584.95.0 09/27/2017 [90610.931767] task: ffff8803c4bf8040 task.stack: ffff8803bf2a0000 [90610.938390] RIP: 0010:queued_write_lock_slowpath+0x7d/0xa1 [90610.944524] RSP: 0018:ffff8803bf2a7d18 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff10 [90610.952981] RAX: 0000000000000101 RBX: ffffffffc0362298 RCX: ffffffff8f6ef824 [90610.960955] RDX: 1ffffffff806c453 RSI: 0000000000000003 RDI: ffffffffc0362298 [90610.968939] RBP: ffff8803bf2a7d30 R08: dffffc0000000000 R09: ffffffff8ffbc45e [90610.976918] R10: ffffed0077e54f51 R11: ffff8803bf2a7a87 R12: ffffffffc036229c [90610.984892] R13: 00000000000000ff R14: ffff8803b72959a4 R15: ffff8803b7295678 [90610.992865] FS: 00007f5b5e95d740(0000) GS:ffff8803eef00000(0000) knlGS:0000000000000000 [90611.001911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [90611.008331] CR2: 00005766d182d018 CR3: 00000003c215c002 CR4: 00000000003606e0 [90611.016303] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [90611.024274] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [90611.032257] Call Trace: [90611.034986] do_raw_write_lock+0x9c/0xce [90611.039374] _raw_write_lock+0x15/0x17 [90611.043589] bt_sock_unlink+0x25/0xa2 [bluetooth] [90611.048878] hci_sock_release+0xe0/0x1b9 [bluetooth] [90611.054433] sock_release+0x49/0xd3 [90611.058333] sock_close+0x12/0x16 [90611.062040] __fput+0x1b0/0x2e5 [90611.065543] ____fput+0xe/0x10 [90611.068964] task_work_run+0x97/0xc0 [90611.072964] prepare_exit_to_usermode+0x13a/0x155 [90611.078225] syscall_return_slowpath+0x12f/0x13a [90611.083379] entry_SYSCALL_64_fastpath+0xa6/0xa8 [90611.088540] RIP: 0033:0x7f5b5e26a510 [90611.092536] RSP: 002b:00007ffef6b05608 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 [90611.101002] RAX: 0000000000000000 RBX: 00005698b60436c0 RCX: 00007f5b5e26a510 [90611.108974] RDX: 00005698b6043900 RSI: 00007f5b5e525b90 RDI: 000000000000000b [90611.116946] RBP: 00007ffef6b05620 R08: 00005698b6043950 R09: 0000000000000009 [90611.124918] R10: 0000000000000001 R11: 0000000000000246 R12: 00005698b603d400 [90611.132892] R13: 00005698b51d8c89 R14: 00005698b51c4540 R15: 0000000000000001 [90611.140865] Code: 90 48 89 df e8 51 2e 13 00 8a 03 84 c0 75 f0 f0 44 0f b0 2b 84 c0 75 e7 41 bd ff 00 00 00 eb 0b f0 44 0f b1 2b ff c8 74 13 f3 90 <48> 89 df e8 3a 30 13 00 8b 03 83 f8 01 75 ef eb e4 4c 89 e7 e8
,
Nov 18 2017
Possibly fixed with upstream commit a9ee77af751f ("Bluetooth: avoid recursive locking in hci_send_to_channel()").
,
Nov 18 2017
,
Nov 23 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/d6805485c59bd78968d79189939386e60f6dc4ad commit d6805485c59bd78968d79189939386e60f6dc4ad Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Date: Thu Nov 23 04:12:16 2017 UPSTREAM: Bluetooth: avoid recursive locking in hci_send_to_channel() Mart reported a deadlock in -RT in the call path: hci_send_monitor_ctrl_event() -> hci_send_to_channel() because both functions acquire the same read lock hci_sk_list.lock. This is also a mainline issue because the qrwlock implementation is writer fair (the traditional rwlock implementation is reader biased). To avoid the deadlock there is now __hci_send_to_channel() which expects the readlock to be held. Fixes: 38ceaa00d02d ("Bluetooth: Add support for sending MGMT commands and events to monitor") BUG= chromium:786148 TEST=B oot multiple times and check if deadlock occurs Change-Id: Ib7b611e86a0776058ccb533eca5097ac4b86ed8e Reported-by: Mart van de Wege <mvdwege@gmail.com> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by: Marcel Holtmann <marcel@holtmann.org> Signed-off-by: Guenter Roeck <groeck@chromium.org> (cherry picked from commit a9ee77af751f) Reviewed-on: https://chromium-review.googlesource.com/777642 Reviewed-by: Dylan Reid <dgreid@chromium.org> [modify] https://crrev.com/d6805485c59bd78968d79189939386e60f6dc4ad/net/bluetooth/hci_sock.c
,
Nov 25 2017
|
||||
►
Sign in to add a comment |
||||
Comment 1 by groeck@chromium.org
, Nov 17 2017