New issue
Advanced search Search tips

Issue 685077 link

Starred by 3 users

Issue metadata

Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Feature



Sign in to add a comment

Audio conformance tests

Project Member Reported by chinyue@chromium.org, Jan 25 2017

Issue description

CTS tests have found several issues on various boards that the audio device took too long to open and most of them are the snd_pcm_hw_params() spent too much time.  We want to come out with some performance tests/requirements beyond functional tests so we can catch these issues during bring-up.

The plan is to extend alsa_api_test to cover more alsa ops.

 
Status: Assigned (was: Untriaged)
Hardware/driver misbehavior instance:

The input device on chell has very unstable hw_level, it could be empty for a hundred ms and then suddenly jumping to 16384.

Related issues:
b/36204501
b/37263560
https://bugs.chromium.org/p/chromium/issues/detail?id=711072

Hardware/driver misbehavior instance:

Audio capture failed due to no captured data from hardware.
This happens under stress test by opening capture streams with various formats.

Related issues:
b/31386388
b/35575874
b/35575985

Hardware/driver misbehavior instance:

There's an overhead in audio hw/driver that adds 50ms to round-trip latency even the smallest buffer size is used.

Related issues:
b/27105066
b/35578323

Hardware/driver misbehavior instance:

Device open takes too much time on headphone.
Opening speaker is okay, but opening headphone takes additional 500ms in snd_pcm_hw_params() calls.

Related issues:
b/32170350
b/35583816

Labels: -Pri-2 Pri-1
Status: Started (was: Assigned)
Summary: Audio conformance tests (was: ALSA hardware operation performance test)
Owner: ----
Cc: yuhsuan@chromium.org paulhsia@chromium.org
+Paul and YuHsuan

The doc Chinyue wrote:
https://docs.google.com/document/d/1AJi3Qa7AoVE1d1xEJ64O0tX6Fva_XB1pYVQcFL7JuxQ/edit?ts=5ab8acb5

Please take a look and see if you are interested in this work.

Thanks!

Owner: yuhsuan@chromium.org
Project Member

Comment 12 by bugdroid1@chromium.org, Sep 10

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/809df308e02abd1dc569b500a2527e6b4c0882eb

commit 809df308e02abd1dc569b500a2527e6b4c0882eb
Author: Yu-hsuan Hsu <yuhsuan@google.com>
Date: Mon Sep 10 15:13:59 2018

audiotest: Install alsa_conformance_test

There is a new binary alsa_conformance_test to be installed.

BUG=chromium:685077
TEST=Tested on several board including samus, peppy.
CQ-DEPEND=CL:1113716

Change-Id: If8fd06d1a84fa59ca833c7eff74fe9c0a791e90b
Reviewed-on: https://chromium-review.googlesource.com/1113715
Commit-Ready: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Tested-by: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/809df308e02abd1dc569b500a2527e6b4c0882eb/chromeos-base/audiotest/audiotest-9999.ebuild

Project Member

Comment 13 by bugdroid1@chromium.org, Sep 10

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/audiotest/+/e45bff58740fbd95405053be74f8bde3263122db

commit e45bff58740fbd95405053be74f8bde3263122db
Author: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Date: Mon Sep 10 15:13:58 2018

alsa_conformance_test: Print basic information of playback device

Print basic information of playback device before setting params.
Such as range of channel, format, rate, period size and buffer
size.

BUG=chromium:685077
TEST=Tested on several board including samus, peppy.
     $ alsa_conformance_test -P hw:0,0

CQ-DEPEND=CL:1113715

Change-Id: Ifc3cc5c7356cab1bcfa7ff83bfdf66b85a2e94fc
Reviewed-on: https://chromium-review.googlesource.com/1113716
Commit-Ready: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Tested-by: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[add] https://crrev.com/e45bff58740fbd95405053be74f8bde3263122db/src/alsa_conformance_thread.c
[modify] https://crrev.com/e45bff58740fbd95405053be74f8bde3263122db/src/module.mk
[add] https://crrev.com/e45bff58740fbd95405053be74f8bde3263122db/include/alsa_conformance_helper.h
[add] https://crrev.com/e45bff58740fbd95405053be74f8bde3263122db/include/alsa_conformance_args.h
[modify] https://crrev.com/e45bff58740fbd95405053be74f8bde3263122db/Makefile
[add] https://crrev.com/e45bff58740fbd95405053be74f8bde3263122db/src/alsa_conformance_args.c
[add] https://crrev.com/e45bff58740fbd95405053be74f8bde3263122db/src/alsa_conformance_helper.c
[add] https://crrev.com/e45bff58740fbd95405053be74f8bde3263122db/src/alsa_conformance_test.c
[add] https://crrev.com/e45bff58740fbd95405053be74f8bde3263122db/include/alsa_conformance_thread.h

Project Member

Comment 14 by bugdroid1@chromium.org, Sep 10

Project Member

Comment 15 by bugdroid1@chromium.org, Sep 15

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/audiotest/+/d3425a28229eadfe4b8dfb0b34f2ea19b83262fd

commit d3425a28229eadfe4b8dfb0b34f2ea19b83262fd
Author: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Date: Sat Sep 15 00:05:50 2018

alsa_conformance_test: Add timer to calculate runtime.

Add timer to calculate runtime of ALSA APIs.

BUG=chromium:685077
TEST=Tested on several board including samus, peppy.

Change-Id: I8f5e5fce395f0c7cf48ec7eb9cd457503df8272c
Reviewed-on: https://chromium-review.googlesource.com/1115040
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/d3425a28229eadfe4b8dfb0b34f2ea19b83262fd/src/alsa_conformance_thread.c
[modify] https://crrev.com/d3425a28229eadfe4b8dfb0b34f2ea19b83262fd/src/module.mk
[modify] https://crrev.com/d3425a28229eadfe4b8dfb0b34f2ea19b83262fd/include/alsa_conformance_helper.h
[add] https://crrev.com/d3425a28229eadfe4b8dfb0b34f2ea19b83262fd/src/alsa_conformance_timer.c
[add] https://crrev.com/d3425a28229eadfe4b8dfb0b34f2ea19b83262fd/include/alsa_conformance_timer.h
[modify] https://crrev.com/d3425a28229eadfe4b8dfb0b34f2ea19b83262fd/src/alsa_conformance_helper.c
[modify] https://crrev.com/d3425a28229eadfe4b8dfb0b34f2ea19b83262fd/src/alsa_conformance_test.c
[modify] https://crrev.com/d3425a28229eadfe4b8dfb0b34f2ea19b83262fd/include/alsa_conformance_thread.h

Project Member

Comment 16 by bugdroid1@chromium.org, Oct 6

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/audiotest/+/89d51e09f5b89d46d2f0b7df26318da48f524e38

commit 89d51e09f5b89d46d2f0b7df26318da48f524e38
Author: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Date: Sat Oct 06 02:37:52 2018

alsa_conformance_test: Can run playback in thread.

Add duration and block_size arguments. It will play duration * rate
frames through direct access(mmap). Once it detects hw_level(frames_left)
is less than block_size, it will write new frames into buffer with
block_size.

BUG=chromium:685077
TEST=Tested on several board including samus, peppy.
     $ alsa_conformance_test -P hw:1,0 -d 3

Change-Id: Icb5119e9f5eb1938dd96f79ffe737d368009bcc1
Reviewed-on: https://chromium-review.googlesource.com/1115239
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/89d51e09f5b89d46d2f0b7df26318da48f524e38/src/alsa_conformance_thread.c
[modify] https://crrev.com/89d51e09f5b89d46d2f0b7df26318da48f524e38/include/alsa_conformance_helper.h
[modify] https://crrev.com/89d51e09f5b89d46d2f0b7df26318da48f524e38/include/alsa_conformance_args.h
[modify] https://crrev.com/89d51e09f5b89d46d2f0b7df26318da48f524e38/src/alsa_conformance_args.c
[modify] https://crrev.com/89d51e09f5b89d46d2f0b7df26318da48f524e38/src/alsa_conformance_helper.c
[modify] https://crrev.com/89d51e09f5b89d46d2f0b7df26318da48f524e38/src/alsa_conformance_test.c
[modify] https://crrev.com/89d51e09f5b89d46d2f0b7df26318da48f524e38/include/alsa_conformance_thread.h

Project Member

Comment 17 by bugdroid1@chromium.org, Oct 6

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/audiotest/+/8aba363dfdf6b73139f39080a3e60073b7550898

commit 8aba363dfdf6b73139f39080a3e60073b7550898
Author: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Date: Sat Oct 06 02:37:53 2018

alsa_conformance_test: Add recorder to check stability of playback.

We record the point when each hw_level change. And then we can
check stability of playback from these methods.

1. The granularity
We check whether device can consume samples in stable step. We compute
max and min step of device. We hope largest step is small enough
otherwise it may cause underrun.

2. The rate
We hope the rate of samples consumed in device ~= sampling rate we
set. We use linear regression to compute rate from data we recorded.

BUG=chromium:685077
TEST=Tested on several board including samus, peppy.

Change-Id: I4d87533af8ad452ccc7aa7e88b1baf47d07121bf
Reviewed-on: https://chromium-review.googlesource.com/1116217
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[add] https://crrev.com/8aba363dfdf6b73139f39080a3e60073b7550898/include/alsa_conformance_recorder.h
[add] https://crrev.com/8aba363dfdf6b73139f39080a3e60073b7550898/src/alsa_conformance_recorder.c
[modify] https://crrev.com/8aba363dfdf6b73139f39080a3e60073b7550898/src/module.mk
[modify] https://crrev.com/8aba363dfdf6b73139f39080a3e60073b7550898/src/alsa_conformance_thread.c

Project Member

Comment 18 by bugdroid1@chromium.org, Oct 6

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/audiotest/+/adee75cd6b73f8ae0cbe95f81f9ba545e8c36c5d

commit adee75cd6b73f8ae0cbe95f81f9ba545e8c36c5d
Author: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Date: Sat Oct 06 02:37:53 2018

alsa_conformance_test: Add debug mode and debug messages.

Sometimes we want to get more detailed log. We can use --debug or -D
argument to enable debug mode now. It will print information when
everytime hw_level changes.

BUG=chromium:685077
TEST=Tested on several board including samus, peppy.
     $ alsa_conformance_test -P hw:1,0 --debug

Change-Id: I2b45f5a9300f0a7dd8a6a8c926456ad791255bfd
Reviewed-on: https://chromium-review.googlesource.com/1124260
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/adee75cd6b73f8ae0cbe95f81f9ba545e8c36c5d/src/module.mk
[modify] https://crrev.com/adee75cd6b73f8ae0cbe95f81f9ba545e8c36c5d/src/alsa_conformance_test.c
[modify] https://crrev.com/adee75cd6b73f8ae0cbe95f81f9ba545e8c36c5d/src/alsa_conformance_thread.c
[add] https://crrev.com/adee75cd6b73f8ae0cbe95f81f9ba545e8c36c5d/include/alsa_conformance_debug.h
[add] https://crrev.com/adee75cd6b73f8ae0cbe95f81f9ba545e8c36c5d/src/alsa_conformance_debug.c

Project Member

Comment 19 by bugdroid1@chromium.org, Oct 12

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/audiotest/+/0f8938d7970242d3d39b55aedbba0d2ec7546db0

commit 0f8938d7970242d3d39b55aedbba0d2ec7546db0
Author: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Date: Fri Oct 12 08:17:12 2018

alsa_conformance_test: Support capture stream.

We can check stability of capture by using the same method as playback.
Record points to compute rate and granularity of steps.

BUG=chromium:685077
TEST=Tested on several boards including samus, peppy.
     $ alsa_conformance_test -C hw:0,0

Change-Id: I3cd663a2f0751cd41d2d041c949ed6c0a69731cd
Reviewed-on: https://chromium-review.googlesource.com/1137795
Commit-Ready: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Tested-by: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/0f8938d7970242d3d39b55aedbba0d2ec7546db0/src/alsa_conformance_thread.c
[modify] https://crrev.com/0f8938d7970242d3d39b55aedbba0d2ec7546db0/include/alsa_conformance_helper.h
[modify] https://crrev.com/0f8938d7970242d3d39b55aedbba0d2ec7546db0/include/alsa_conformance_args.h
[modify] https://crrev.com/0f8938d7970242d3d39b55aedbba0d2ec7546db0/src/alsa_conformance_args.c
[modify] https://crrev.com/0f8938d7970242d3d39b55aedbba0d2ec7546db0/src/alsa_conformance_helper.c
[modify] https://crrev.com/0f8938d7970242d3d39b55aedbba0d2ec7546db0/src/alsa_conformance_test.c

Project Member

Comment 20 by bugdroid1@chromium.org, Oct 16

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/audiotest/+/5a7dbd41e1fc9500d3edc393099990cae1df43e1

commit 5a7dbd41e1fc9500d3edc393099990cae1df43e1
Author: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Date: Tue Oct 16 17:30:41 2018

alsa_conformance_test: Support multi-thread.

Now we can playback and capture at the same time.

BUG=chromium:685077
TEST=Tested on several boards including samus, peppy.

Change-Id: Iee72447fe702819b1e3c8f58eea3322c8883058b
Reviewed-on: https://chromium-review.googlesource.com/1137797
Commit-Ready: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Tested-by: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/5a7dbd41e1fc9500d3edc393099990cae1df43e1/src/alsa_conformance_test.c
[modify] https://crrev.com/5a7dbd41e1fc9500d3edc393099990cae1df43e1/src/alsa_conformance_thread.c

Project Member

Comment 21 by bugdroid1@chromium.org, Nov 15

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/audiotest/+/ea833946dc03660975299147cce423e2506188f6

commit ea833946dc03660975299147cce423e2506188f6
Author: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Date: Thu Nov 15 10:16:36 2018

alsa_conformance_test: Support multi-device.

Now we support multiple devices. With devices file, user can set
different rate, format, ... , for each device.

BUG=chromium:685077
TEST=Tested on several boards including samus, peppy.

Change-Id: I737eca74694196940f7f28f872ea93e12a5ef491
Reviewed-on: https://chromium-review.googlesource.com/1143170
Commit-Ready: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Tested-by: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/ea833946dc03660975299147cce423e2506188f6/src/alsa_conformance_test.c
[modify] https://crrev.com/ea833946dc03660975299147cce423e2506188f6/src/alsa_conformance_thread.c
[modify] https://crrev.com/ea833946dc03660975299147cce423e2506188f6/src/alsa_conformance_args.c
[modify] https://crrev.com/ea833946dc03660975299147cce423e2506188f6/include/alsa_conformance_thread.h
[modify] https://crrev.com/ea833946dc03660975299147cce423e2506188f6/include/alsa_conformance_args.h

Project Member

Comment 22 by bugdroid1@chromium.org, Nov 16

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/audiotest/+/11c486b68c789e30f9e7c55e53abf79951f126db

commit 11c486b68c789e30f9e7c55e53abf79951f126db
Author: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Date: Fri Nov 16 23:10:48 2018

alsa_conformance_test: Add strict and dev_info_only mode.

Strict mode: If it can not set params to the value user set, it
will fail.

Dev_info_only mode: Show device information only without run and set
params.

BUG=chromium:685077
TEST=Tested on several boards including samus, peppy.

Change-Id: Ibb5d897006bb0c39922c94091acc05d9e9561393
Reviewed-on: https://chromium-review.googlesource.com/1149668
Commit-Ready: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Tested-by: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/11c486b68c789e30f9e7c55e53abf79951f126db/src/alsa_conformance_test.c
[modify] https://crrev.com/11c486b68c789e30f9e7c55e53abf79951f126db/src/alsa_conformance_thread.c
[modify] https://crrev.com/11c486b68c789e30f9e7c55e53abf79951f126db/src/alsa_conformance_args.c
[modify] https://crrev.com/11c486b68c789e30f9e7c55e53abf79951f126db/include/alsa_conformance_args.h

Project Member

Comment 23 by bugdroid1@chromium.org, Dec 29

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/audiotest/+/f3256e11b3b9f7dbe50dd4f0ac375802cb590462

commit f3256e11b3b9f7dbe50dd4f0ac375802cb590462
Author: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Date: Sat Dec 29 00:14:02 2018

alsa_conformance_test: Add script to test basic function automatically

This script can check the basic function of one pcm device.

We have one basic test in this change:

Test Params - Check whether all params supported by pcm device can be
set correctly.

BUG=chromium:685077
TEST=alsa_conformace_test.py hw:0,0 PLAYBACK
CQ-DEPEND=CL:1339279

Change-Id: Ic592e6ce8ade37ef3724246f09d30fecbb7d7fd7
Reviewed-on: https://chromium-review.googlesource.com/1338782
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[add] https://crrev.com/f3256e11b3b9f7dbe50dd4f0ac375802cb590462/script/alsa_conformance_test.py
[modify] https://crrev.com/f3256e11b3b9f7dbe50dd4f0ac375802cb590462/Makefile

Project Member

Comment 24 by bugdroid1@chromium.org, Dec 29

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/a3035690cf665ead3f84b0c8b4bd46b6e8a9c974

commit a3035690cf665ead3f84b0c8b4bd46b6e8a9c974
Author: Yu-Hsuan Hsu <yuhsuan@google.com>
Date: Sat Dec 29 00:14:03 2018

audiotest: Install alsa_conformance_test.py script

There is a new script alsa_conformance_test.py to be installed.

BUG=chromium:685077
TEST=Tested on several board including peppy, eve.
CQ-DEPEND=CL:1338782

Change-Id: Ib7bc6325504ed31c56f0e73a7822a4f4deee6e40
Reviewed-on: https://chromium-review.googlesource.com/1339279
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/a3035690cf665ead3f84b0c8b4bd46b6e8a9c974/chromeos-base/audiotest/audiotest-9999.ebuild

Project Member

Comment 25 by bugdroid1@chromium.org, Jan 9

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/audiotest/+/8cb6a1a2a04acf304721ddc098b814cb54792e3f

commit 8cb6a1a2a04acf304721ddc098b814cb54792e3f
Author: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Date: Wed Jan 09 19:55:37 2019

alsa_conformance_test.py: Modify json format

Modify json format to make it easier to be parsed.

BUG=chromium:685077
TEST=alsa_conformace_test.py hw:0,0 PLAYBACK

Change-Id: If8972cf2aace9cb4c753f254a03fd9369ef1f87b
Reviewed-on: https://chromium-review.googlesource.com/1397596
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Yu-Hsuan Hsu <yuhsuan@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

[modify] https://crrev.com/8cb6a1a2a04acf304721ddc098b814cb54792e3f/script/alsa_conformance_test.py

Sign in to add a comment