New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 756830 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

cras: SIGSEGV in mixer_control_set_dBFS

Project Member Reported by smbar...@chromium.org, Aug 18 2017

Issue description

Lots of crashes in mixer_control_set_dBFS: https://crash.corp.google.com/browse?stbtiq=mixer_control_set_dBFS

Attaching an example minidump from 9765.21.0 (Official Build) dev-channel guado test. This one happened when hotplugging some USB audio devices.

gdb indicates that in mixer_control_set_dBFS, control->elements was 0x20. So not NULL and also not a valid pointer :\

Backtrace:
#0  0x00005d13dd4affdc in mixer_control_set_dBFS (control=<optimized out>, to_set=<optimized out>) at server/cras_alsa_mixer.c:368
#1  0x00005d13dd4b01f3 in cras_alsa_mixer_set_capture_dBFS (cras_mixer=<optimized out>, dBFS=<optimized out>, mixer_input=0x5d13df0f9560) at server/cras_alsa_mixer.c:1001
#2  0x00005d13dd4aad4e in set_alsa_capture_gain (iodev=0x5d13df185740) at server/cras_alsa_io.c:756
#3  0x00005d13dd4ab124 in open_dev (iodev=0x5d13df185740) at server/cras_alsa_io.c:378
#4  0x00005d13dd4bd100 in cras_iodev_open (iodev=0x5d13df185740, cb_level=1024) at server/cras_iodev.c:827
#5  0x00005d13dd4be6f4 in init_device (rstream=<optimized out>, dev=<optimized out>) at server/cras_iodev_list.c:439
#6  pinned_stream_added (rstream=0x5d13df0ebbc0) at server/cras_iodev_list.c:611
#7  stream_added_cb (rstream=0x5d13df0ebbc0) at server/cras_iodev_list.c:629
#8  0x00005d13dd4c7ec2 in stream_list_add (list=0x5d13def8ace0, stream_config=<optimized out>, stream=0x7fff8ad43e48) at server/stream_list.c:88
#9  0x00005d13dd4c2422 in handle_client_stream_connect (client=<optimized out>, msg=<optimized out>, aud_fd=<optimized out>) at server/cras_rclient.c:69
#10 cras_rclient_message_from_client (client=0x5d13df1ecf70, msg=0x7fff8ad44090, fd=42) at server/cras_rclient.c:394
#11 0x00005d13dd4c418b in handle_message_from_client (client=<optimized out>) at server/cras_server.c:128
#12 cras_server_run (profile_disable_mask=<optimized out>) at server/cras_server.c:545
#13 0x00005d13dd4a78cd in main (argc=<optimized out>, argv=<optimized out>) at server/cras.c:118
 
cras.20170818.114738.1822.dmp
47.2 KB Download
dgreid pointed out that this is likely a use-after-free.
Cc: hychao@chromium.org dgreid@chromium.org cychiang@chromium.org
cras_iodev_list.c:close_dev() will not close an iodev with a pinned stream. This means when the card is being torn down (i.e. we unplugged USB) we'll end up leaving the iodev intact but with dangling pointers to other things that were maintained by the card. See cras_alsa_io.c:alsa_iodev_destroy() and cras_alsa_card.c:cras_alsa_card_destroy(). The mixer pointer, for example, has definitely been reused:

p *((struct alsa_io *)devs[1]->iodevs->next)->mixer                                                                                                                                                  
$257 = {
  mixer = 0x5d13defef840,
  main_volume_controls = 0x100000001, # not a valid pointer
  output_controls = 0x5d13df1ecc20,
  playback_switch = 0x5d13df1a4c00,
  main_capture_controls = 0x0,
  input_controls = 0x21,              # not a valid pointer
  capture_switch = 0x65727574706143,  # C string "Capture"
  max_volume_dB = 138896501050328,    # pointer
  min_volume_dB = 32                  # unlikely
}

Should we just close the iodev anyway? If I unplug my USB speaker/mic, it doesn't matter if the stream is pinned - the thing's got to be closed. +audio people
Owner: cychiang@chromium.org
Status: Assigned (was: Untriaged)
Hi Steve, thank you for tracking down this bug.

The function close_dev was used by two cases:

1. Close the device anyway.
2. Possibly close the device, still keep it for pinned stream.

cras_iodev_list is complicated since these functions

cras_iodev_list_disable_dev
disable_device
close_dev
possibly_close_enabled_devs

do not let user show clear intention whether it is case 1 or case 2.

The consistent way to reproduce this issue is to use

cras_test_client --playback_f /dev/zero --pin_device X:Y
where X:Y is USB device index.

Unplug USB

ctrl+c to stop playback, and see CRAS crashed.

I will check if there is any quick fix for this particular case.
But I really doubt there can be a quick fix.
The cras_iodev_list needs to be refactored somehow so pinned stream can be managed cleanly.

Hsinyu is working on hotword fallback device recently which will further complicate cras_iodev_list.

We need to sort this out.

Hm, I'm having trouble with the repro in #3 with my USB speaker. I added a few extra logs and it looks like I'm never actually getting a stream pinned properly on the device, so the bad codepath is never triggered. So no luck verifying #4.
Hi Steve, you can play some tone like

cras_test_client --playback_f /usr/local/autotest/client/cros/audio/fix_440_16.raw --pin_device X:Y
and check tone is from USB speaker.

Then, select active node to internal speaker from UI, and check whether sine tone is still coming from USB speaker. That way you can make sure stream is pinned to USB device.

Anyway I can verify the fix since it seems pretty easy to trigger the crash for me.

As a side note, we did not expect a pinned stream can be used on a device that can be unplugged. As I understand the pinned stream is used in hotword only as for now. Maybe there is something else causing the crash in #0.
Maybe the crash has something to do with loopback device or casting?
Project Member

Comment 8 by bugdroid1@chromium.org, Aug 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/adhd/+/105baf9f2e68dd411f6c8b84b663f81ae45b40c7

commit 105baf9f2e68dd411f6c8b84b663f81ae45b40c7
Author: Dylan Reid <dgreid@chromium.org>
Date: Thu Aug 31 09:37:09 2017

CRAS: iodev_list - Allow device to be force closed

When a device is removed we have to close it, we can't leave it open if
there is a pinned stream.

BUG= chromium:756830 
TEST=cras_test_client --playback_f /dev/zero --pin_device X:Y, where X:Y
is USB device. Unplug USB device and see CRAS does not crash.

Change-Id: I7ac5172162b9dcc66b6318117685e144aee40473
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/636770
Commit-Ready: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/105baf9f2e68dd411f6c8b84b663f81ae45b40c7/cras/src/server/cras_iodev_list.c
[modify] https://crrev.com/105baf9f2e68dd411f6c8b84b663f81ae45b40c7/cras/src/tests/iodev_list_unittest.cc
[modify] https://crrev.com/105baf9f2e68dd411f6c8b84b663f81ae45b40c7/cras/src/server/cras_bt_device.c
[modify] https://crrev.com/105baf9f2e68dd411f6c8b84b663f81ae45b40c7/cras/src/server/cras_iodev_list.h
[modify] https://crrev.com/105baf9f2e68dd411f6c8b84b663f81ae45b40c7/cras/src/server/cras_device_monitor.c
[modify] https://crrev.com/105baf9f2e68dd411f6c8b84b663f81ae45b40c7/cras/src/server/cras_iodev.c
[modify] https://crrev.com/105baf9f2e68dd411f6c8b84b663f81ae45b40c7/cras/src/tests/device_monitor_unittest.cc

Comment 9 by dgreid@chromium.org, Aug 31 2017

Fix landed in master, should we cherry pick to 61 after a couple days on canary?
Cc: katierh@chromium.org
#9, yes, please :)
Labels: -Pri-3 M-61 Merge-Request-61 Pri-1
Matching priority on b/65196100 and requesting merge to 61.
Project Member

Comment 13 by sheriffbot@chromium.org, Sep 8 2017

Labels: -Merge-Request-61 Merge-Review-61 Hotlist-Merge-Review
This bug requires manual review: Request affecting a post-stable build
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), ketakid@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Labels: -Merge-Review-61 Merge-Approved-61
Approving merge to M61.
Project Member

Comment 15 by bugdroid1@chromium.org, Sep 8 2017

Labels: merge-merged-release-R61-9765.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/adhd/+/b2af0b6ba525a8a76dbef69cf79a092f589bc5f6

commit b2af0b6ba525a8a76dbef69cf79a092f589bc5f6
Author: Dylan Reid <dgreid@chromium.org>
Date: Fri Sep 08 19:31:03 2017

CRAS: iodev_list - Allow device to be force closed

When a device is removed we have to close it, we can't leave it open if
there is a pinned stream.

BUG= chromium:756830 
TEST=cras_test_client --playback_f /dev/zero --pin_device X:Y, where X:Y
is USB device. Unplug USB device and see CRAS does not crash.

Change-Id: I7ac5172162b9dcc66b6318117685e144aee40473
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/636770
Commit-Ready: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
(cherry picked from commit 105baf9f2e68dd411f6c8b84b663f81ae45b40c7)
Reviewed-on: https://chromium-review.googlesource.com/657302
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Commit-Queue: Stephen Barber <smbarber@chromium.org>
Tested-by: Stephen Barber <smbarber@chromium.org>

[modify] https://crrev.com/b2af0b6ba525a8a76dbef69cf79a092f589bc5f6/cras/src/server/cras_iodev_list.c
[modify] https://crrev.com/b2af0b6ba525a8a76dbef69cf79a092f589bc5f6/cras/src/tests/iodev_list_unittest.cc
[modify] https://crrev.com/b2af0b6ba525a8a76dbef69cf79a092f589bc5f6/cras/src/server/cras_bt_device.c
[modify] https://crrev.com/b2af0b6ba525a8a76dbef69cf79a092f589bc5f6/cras/src/server/cras_iodev_list.h
[modify] https://crrev.com/b2af0b6ba525a8a76dbef69cf79a092f589bc5f6/cras/src/server/cras_device_monitor.c
[modify] https://crrev.com/b2af0b6ba525a8a76dbef69cf79a092f589bc5f6/cras/src/server/cras_iodev.c
[modify] https://crrev.com/b2af0b6ba525a8a76dbef69cf79a092f589bc5f6/cras/src/tests/device_monitor_unittest.cc

Labels: -Merge-Approved-61 Merge-Merged
Status: Fixed (was: Assigned)

Comment 18 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Comment 19 by dchan@chromium.org, Jan 23 2018

Status: Fixed (was: Archived)

Sign in to add a comment