Circular locking in snd_soc_max98090 |
||||||
Issue description
Observed when booting chromeos-4.4 or chromeos-4.14 on cyan, with LOCKDEBUG enabled.
[ 142.586734] max98090 i2c-193C9890:00: DMIC Mux: put_dmic_mux enable DMIC
[ 142.596094] ======================================================
[ 142.603010] [ INFO: possible circular locking dependency detected ]
[ 142.610031] 4.4.131-14082-g28a14d95341a #2 Not tainted
[ 142.615784] -------------------------------------------------------
[ 142.622799] cras/2591 is trying to acquire lock:
[ 142.627966] (&card->dapm_mutex){+.+.+.}, at: [<ffffffffb69ff4f6>] snd_soc_dapm_add_routes+0xf3/0x8b8
[ 142.638351]
but task is already holding lock:
[ 142.644884] (&card->controls_rwsem){++++.+}, at: [<ffffffffb69a82ce>] snd_ctl_elem_write+0x91/0x2f1
[ 142.655159]
which lock already depends on the new lock.
[ 142.664313]
the existing dependency chain (in reverse order) is:
[ 142.672688]
-> #1 (&card->controls_rwsem){++++.+}:
[ 142.678287] [<ffffffffb5785dfc>] lock_acquire+0x2ec/0x3a0
[ 142.685036] [<ffffffffb6e857ad>] down_write+0x39/0x8c
[ 142.691392] [<ffffffffb69a26a8>] snd_ctl_add+0x144/0x6d1
[ 142.698034] [<ffffffffb6a0c551>] dapm_create_or_share_kcontrol+0x923/0x126e
[ 142.706528] [<ffffffffb6a00a74>] snd_soc_dapm_new_widgets+0x4d6/0xfc5
[ 142.714436] [<ffffffffc068dc5e>] sst_dsp_init_v2_dpcm+0xfd/0x7f5 [snd_soc_sst_mfld_platform]
[ 142.724597] [<ffffffffc0688cd4>] sst_soc_probe+0x99/0xa2 [snd_soc_sst_mfld_platform]
[ 142.733975] [<ffffffffb69f2408>] snd_soc_platform_drv_probe+0x5d/0x64
[ 142.741889] [<ffffffffb69f85b0>] soc_probe_component+0x714/0xb31
[ 142.749311] [<ffffffffb69ef00e>] snd_soc_register_card+0x1188/0x2c11
[ 142.757120] [<ffffffffb6a1eea3>] devm_snd_soc_register_card+0x44/0x94
[ 142.765027] [<ffffffffc08800a2>] snd_cht_mc_probe+0xa2/0x119 [snd_soc_sst_cht_bsw_max98090_ti]
[ 142.775367] [<ffffffffb63acb25>] platform_drv_probe+0xb5/0x121
[ 142.782596] [<ffffffffb63a7c7c>] driver_probe_device+0x551/0x5f6
[ 142.790011] [<ffffffffb63a8220>] __driver_attach+0xce/0x105
[ 142.796943] [<ffffffffb63a39db>] bus_for_each_dev+0x129/0x185
[ 142.804072] [<ffffffffb63a814f>] driver_attach+0x42/0x45
[ 142.810713] [<ffffffffb63a4c2e>] bus_add_driver+0x2a9/0x444
[ 142.817648] [<ffffffffb63aa023>] driver_register+0x21d/0x29f
[ 142.824678] [<ffffffffb63aca69>] __platform_driver_register+0xc3/0xca
[ 142.832589] [<ffffffffc0888017>] 0xffffffffc0888017
[ 142.838750] [<ffffffffb56004b0>] do_one_initcall+0x1be/0x355
[ 142.845789] [<ffffffffb57fa6d8>] do_init_module+0x1b6/0x512
[ 142.852729] [<ffffffffb57f97f9>] load_module+0x5bcf/0x6820
[ 142.859569] [<ffffffffb57f0396>] SyS_finit_module+0x30a/0x36f
[ 142.866702] [<ffffffffb6e8ce13>] entry_SYSCALL_64_fastpath+0x27/0xa0
[ 142.874519]
-> #0 (&card->dapm_mutex){+.+.+.}:
[ 142.879724] [<ffffffffb57880b1>] __lock_acquire+0x2201/0x27f9
[ 142.886856] [<ffffffffb5785dfc>] lock_acquire+0x2ec/0x3a0
[ 142.893591] [<ffffffffb6e82e5b>] __mutex_lock_common+0x1b5/0x228c
[ 142.901104] [<ffffffffb6e82ca0>] mutex_lock_nested+0x3e/0x44
[ 142.908137] [<ffffffffb69ff4f6>] snd_soc_dapm_add_routes+0xf3/0x8b8
[ 142.915850] [<ffffffffc06caa5e>] put_dmic_mux+0x118/0x181 [snd_soc_max98090]
[ 142.924453] [<ffffffffb69a8437>] snd_ctl_elem_write+0x1fa/0x2f1
[ 142.931780] [<ffffffffb69a590f>] snd_ctl_ioctl+0x6b0/0xb00
[ 142.938611] [<ffffffffb5c3e186>] SyS_ioctl+0x219/0x431
[ 142.945063] [<ffffffffb6e8ce13>] entry_SYSCALL_64_fastpath+0x27/0xa0
[ 142.947647] sdhci: Timeout waiting for Buffer Read Ready interrupt during tuning procedure, falling back to fixed sampling clock
[ 142.965810]
other info that might help us debug this:
[ 142.974770] Possible unsafe locking scenario:
[ 142.981394] CPU0 CPU1
[ 142.986464] ---- ----
[ 142.991533] lock(&card->controls_rwsem);
[ 142.996139] lock(&card->dapm_mutex);
[ 143.003162] lock(&card->controls_rwsem);
[ 143.010582] lock(&card->dapm_mutex);
[ 143.014789]
*** DEADLOCK ***
Complete boot log attached.
The problem was introduced with 'CHROMIUM: ASoC: max98090: Enable DMIC at playback when selected'.
,
May 16 2018
I'll take a look.
,
May 16 2018
Now I got it. 1. Path one: probing sound card: snd_soc_dapm_new_widgets acquire dapm_mutex snd_ctl_add acquire controls_rwsem 2. Path two: setting control value to adjust DAPM route: snd_ctl_elem_write acquire read lock for control_rwsem snd_soc_dapm_add_routes acquire dapm_mutex Normally, the deadlock case will not happen because CRAS will wait for the sound card probing be finished. After that, it will see the card uevent, and set the value for DMIC Mux. But we should fix this. I think we can move snd_soc_dapm_add_routes into a workqueue. It is not needed right away when user select DMIC. It can be finished later. So control_rwsem can be freed after schedule_work, and we can avoid the possible deadlock. Tzungbi, could you try the above approach on a cyan device ? Thanks!
,
May 29 2018
Hi groeck, two questions: 1. Could you tell us (more detail) how you boot v4.4 or v4.14 on cyan? Cyan's default kernel version is 3.18. 2. What do you mean "LOCKDEBUG"? Is it enough if we set kernel configurations as following? CONFIG_DEBUG_LOCKDEP=y CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_DEBUG_MUTEXES=y CONFIG_DEBUG_RT_MUTEXES=y CONFIG_LOCKDEP=y CONFIG_LOCK_STAT=y CONFIG_PROVE_LOCKING=y
,
May 29 2018
USE="kasan lockdebug pcserial tty_console_ttyS0" emerge-cyan chromeos-kernel-4_14 then install the kernel on the target system as usual. To run images in trybot I use a number of CLs to make necessary changes and enable the required drivers.
,
Jun 6 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/0fadc6169cac33bbd5129d2c6ccdeb060c1e82ab commit 0fadc6169cac33bbd5129d2c6ccdeb060c1e82ab Author: Tzung-Bi Shih <tzungbi@chromium.org> Date: Wed Jun 06 08:16:18 2018 FIXUP: CHROMIUM: ASoC: max98090: Enable DMIC at playback when selected LOCKDEP will warn if put_dmic_mux() calls snd_soc_dapm_add_routes(). It warns a deadlock possibility if someone uses snd_ctl_ioctl() to reach put_dmic_mux() when sound cards are still probing: a lock inversion could exist. The fix defers snd_soc_dapm_add_routes() to workqueue so that locks will not be acquired reversely within two threads. BUG=chromium:842327 TEST=Plug loopback dongle and switch two microphones alternatively on "cyan" with forcely install kernel 4.4 Signed-off-by: Tzung-Bi Shih <tzungbi@chromium.org> Change-Id: I70f1c33b3bbab945ce3ebea9606e101649865d8f Reviewed-on: https://chromium-review.googlesource.com/1082252 Commit-Ready: Tzung-Bi Shih <tzungbi@chromium.org> Tested-by: Tzung-Bi Shih <tzungbi@chromium.org> Reviewed-by: Dylan Reid <dgreid@chromium.org> [modify] https://crrev.com/0fadc6169cac33bbd5129d2c6ccdeb060c1e82ab/sound/soc/codecs/max98090.h [modify] https://crrev.com/0fadc6169cac33bbd5129d2c6ccdeb060c1e82ab/sound/soc/codecs/max98090.c
,
Jun 8 2018
,
Jun 8 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/07ffc107a64cd9fa9914b82aac34f4e5c305cd88 commit 07ffc107a64cd9fa9914b82aac34f4e5c305cd88 Author: Tzung-Bi Shih <tzungbi@chromium.org> Date: Fri Jun 08 10:15:31 2018 FIXUP: CHROMIUM: ASoC: max98090: Enable DMIC at playback when selected LOCKDEP will warn if put_dmic_mux() calls snd_soc_dapm_add_routes(). It warns a deadlock possibility if someone uses snd_ctl_ioctl() to reach put_dmic_mux() when sound cards are still probing: a lock inversion could exist. The fix defers snd_soc_dapm_add_routes() to workqueue so that locks will not be acquired reversely within two threads. BUG=chromium:842327 TEST=None Signed-off-by: Tzung-Bi Shih <tzungbi@chromium.org> Change-Id: I96d6e0063eb036be2694bcac693bf3557129e237 Reviewed-on: https://chromium-review.googlesource.com/1084352 Commit-Ready: Tzung-Bi Shih <tzungbi@chromium.org> Tested-by: Tzung-Bi Shih <tzungbi@chromium.org> Reviewed-by: Dylan Reid <dgreid@chromium.org> [modify] https://crrev.com/07ffc107a64cd9fa9914b82aac34f4e5c305cd88/sound/soc/codecs/max98090.h [modify] https://crrev.com/07ffc107a64cd9fa9914b82aac34f4e5c305cd88/sound/soc/codecs/max98090.c
,
Jun 10 2018
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by conradlo@chromium.org
, May 12 2018