I checked the minidump:

Core was generated by `/usr/bin/cras'.
#0  append_stream (num_iodevs=1, iodevs=0x7fff5f7a0bc8, stream=0x7fc51acc0c30, thread=0x7fc51abcca80) at server/audio_thread.c:244
244     server/audio_thread.c: No such file or directory.



(gdb) bt
#0  append_stream (num_iodevs=1, iodevs=0x7fff5f7a0bc8, stream=0x7fc51acc0c30, thread=0x7fc51abcca80) at server/audio_thread.c:244
#1  thread_add_stream (num_iodevs=1, iodevs=0x7fff5f7a0bc8, stream=0x7fc51acc0c30, thread=0x7fc51abcca80) at server/audio_thread.c:509
#2  handle_playback_thread_message (thread=0x7fc51abcca80) at server/audio_thread.c:844
#3  audio_io_thread (arg=0x7fc51abcca80) at server/audio_thread.c:1618
#4  0x00007fc519143307 in ?? ()
#5  0x0000000000000000 in ?? ()


(gdb) info local
open_dev = <optimized out>
dev = 0x7fc51ac90880
init_cb_ts = {tv_sec = 1114, tv_nsec = 857124824}
i = <optimized out>
out = 0x0
rc = <optimized out>

It shows that crash happened in append_stream, but we don't know the exact place.
info local shows that dev at 0x7fc51ac90880. This must be iodevs[0] because num_iodevs = 1.
Also, we know out = 0x0 and init_cb_ts has been set.

I also investigated the history of this crash.
Using dremel I found that there was no such crash on R52, but happened a lot on R53.

dremel> SELECT COUNT(*), Product.Version FROM crash.prod.latest WHERE product.name='ChromeOS' AND exec_name='cras' AND stable_signature CONTAINS 'audio_io_thread' OMIT RECORD IF SUM(thread.StackTrace.StackFrame.FunctionName='audio_thread_add_stream') = 0 GROUP BY Product.Version, ORDER BY Product.Version;
+----------+-----------------+
| COUNT(*) | Product.Version |
+----------+-----------------+
|        6 | 6310.61.0       |
|        4 | 7834.70.0       |
|        6 | 8172.60.0       |
|       15 | 8530.35.0       |
|       38 | 8530.43.0       |
|       36 | 8530.49.0       |
|       20 | 8530.62.0       |
|        4 | 8688.0.0        |
+----------+-----------------+

This suggests that some CL introduced in R53 caused the crash.

From the blamelist between R52 and R53

https://crosland.corp.google.com/log/8350.68.0..8530.35.0

These two are related to audio_thread.
https://chromium-review.googlesource.com/#/c/363713/
https://chromium-review.googlesource.com/#/c/361172/

And this one seems to be most related as it changed append_stream.

CRAS: audio_thread - Align stream fetch time
https://chromium-review.googlesource.com/#/c/363713/

Still need to investigate what might be wrong in the change but at least we have a clue now.

Thank Jimmy, great job investigating.

Looks like it's probably this line:
	
		if (dev->streams)
			init_cb_ts = *dev_stream_next_cb_ts(dev->streams);

dev_stream_next_cb_ts returns null for hotword streams and all the crashes are samus/chell.  I'll post a quick patch.

https://chromium-review.googlesource.com/374978

Yes you are right! It only happens on samus and chell:

dremel> SELECT COUNT(*), hwclass FROM crash.prod.latest WHERE product.name='ChromeOS' AND exec_name='cras' AND stable_signature CONTAINS 'audio_io_thread' AND Product.Version CONTAINS "8530" OMIT RECORD IF SUM(thread.StackTrace.StackFrame.FunctionName='audio_thread_add_stream') = 0 GROUP BY hwclass
;
+----------+-----------------------+
| COUNT(*) | hwclass               |
+----------+-----------------------+
|        9 | CHELL C25-F3K-K6A-A5H |
|        4 | SAMUS E25-G7M-K9M     |
|       99 | SAMUS E25-H7R-W5L     |
+----------+-----------------------+

Thanks for the fix!
I also put up a doc https://goto.google.com/cras-crash-debug describing how to investigate the history of the bug. It should be helpful in the future.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/adhd/+/2222e7f3b756bdc3fb7825727c85302b75a892df

commit 2222e7f3b756bdc3fb7825727c85302b75a892df
Author: Dylan Reid <dgreid@chromium.org>
Date: Wed Aug 24 16:20:58 2016

CRAS: audio_thread - Check for NULL next_cb_ts

Hotword streams return NULL from dev_stream_next_cb_ts, account for that when
setting the initial callback timestamp.  This fixes a crash seen somewhat
regularly on R53.  I'm not sure why there is ever two Hotword streams attached,
but it shouldn't crash anyways.

BUG= chromium:640290 
TEST=None

Change-Id: I7edc0411d05e7e35c48904460297d26daa5be323
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/374978
Reviewed-by: Hsinyu Chao <hychao@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/2222e7f3b756bdc3fb7825727c85302b75a892df/cras/src/server/audio_thread.c

[Automated comment] Less than 2 weeks to go before stable on M53, manual review required.

Approving merge to M53 cros. dgreid@ Please merge this in asap.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/adhd/+/fe461bde8bb9095106ea8ac67b0701a366d81801

commit fe461bde8bb9095106ea8ac67b0701a366d81801
Author: Dylan Reid <dgreid@chromium.org>
Date: Wed Aug 24 16:20:58 2016

CRAS: audio_thread - Check for NULL next_cb_ts

Hotword streams return NULL from dev_stream_next_cb_ts, account for that when
setting the initial callback timestamp.  This fixes a crash seen somewhat
regularly on R53.  I'm not sure why there is ever two Hotword streams attached,
but it shouldn't crash anyways.

BUG= chromium:640290 
TEST=None

Change-Id: I7edc0411d05e7e35c48904460297d26daa5be323
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/374978
Reviewed-by: Hsinyu Chao <hychao@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/379117

[modify] https://crrev.com/fe461bde8bb9095106ea8ac67b0701a366d81801/cras/src/server/audio_thread.c

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/adhd/+/fe461bde8bb9095106ea8ac67b0701a366d81801

commit fe461bde8bb9095106ea8ac67b0701a366d81801
Author: Dylan Reid <dgreid@chromium.org>
Date: Wed Aug 24 16:20:58 2016

CRAS: audio_thread - Check for NULL next_cb_ts

Hotword streams return NULL from dev_stream_next_cb_ts, account for that when
setting the initial callback timestamp.  This fixes a crash seen somewhat
regularly on R53.  I'm not sure why there is ever two Hotword streams attached,
but it shouldn't crash anyways.

BUG= chromium:640290 
TEST=None

Change-Id: I7edc0411d05e7e35c48904460297d26daa5be323
Signed-off-by: Dylan Reid <dgreid@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/374978
Reviewed-by: Hsinyu Chao <hychao@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/379117

[modify] https://crrev.com/fe461bde8bb9095106ea8ac67b0701a366d81801/cras/src/server/audio_thread.c

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

crash signature is no longer seen after 8530.62.0 (Sept 3).