New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 877854 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Dec 8
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Task



Sign in to add a comment

autotest: wifi: add a reboot stress test

Project Member Reported by kirtika@chromium.org, Aug 27

Issue description

Motivation: 
- Acer Cyan issues:StP2 has a PERST rise time constraint in the spec,
which if not met, leads to wifi card not being detected at boot.
This has caused severe bugs/customer complaints on all StP2 devices, but most notably Cyan and Caroline. 
- Newer designs: JfP2 has different, but similar constraint thanks to CNVio integrated architecture. The chip must be powered on before/at same time as the SoC
- Concerns around StP2 in newer (KBL/AML) designs. See threads on <redacted> board. 
A reboot test running nightly as part of wifi_matfunc (requires an AP) or network_nightly (does not require an AP, just checks for the interface) would be helpful in validating new platforms and also discovering issues before we get complaints from the field. 


For now, the following works well for a manual invocation: 
$ test_that ${DUT_HOSTNAME} stress_ClientTestReboot --board=nocturne --args="testname=network_WlanDriver loops=100"

stress_ClientTestReboot is a server-side test that will run any client-side test given number of times.
network_WlanDriver is perfect here since it checks that a wifi device is discovered and a matching kernel module is found. 

bluetooth and touch subsystems have their own control files
in stress_ClientTestReboot. Perhaps we should do the same for wifi. 



 

 
Cc: harpreet@chromium.org akhouderchah@chromium.org briannorris@chromium.org snanda@chromium.org
Added: https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1189722

@Harpreet: Need to check whether the suites and iterations are correct. 
While wifi_stress would be the natural suite for this, I avoided that as pool:stress-wifi is only 3 devices right now. 
added network_nightly to coverage on pool:suites which has all the models, and restricted iterations to 20 to allow "TIME= medium".

network_WlanDriver runtime is ~1min which mean this will add ~20mins to wifi_matfunc. Can we only add it to network_nightly?
@Harpreet: most of that one minute is the overhead to start and stop a test.
network_WlanDriver takes only 2 seconds for its actual content, at least that's what my status.log says from a run just now on my caroline (12.18.27 - 12.18.25): 

 cat results-1-network_WlanDriver/status.log 
INFO	----	----	kernel=3.18.0-18191-g527ff26c280f	localtime=Aug 27 12:18:21	timestamp=1535397501	
START	----	----	timestamp=1535397505	localtime=Aug 27 12:18:25	
	START	network_WlanDriver	network_WlanDriver	timestamp=1535397505	localtime=Aug 27 12:18:25	
		GOOD	network_WlanDriver	network_WlanDriver	timestamp=1535397507	localtime=Aug 27 12:18:27	completed successfully
	END GOOD	network_WlanDriver	network_WlanDriver	timestamp=1535397507	localtime=Aug 27 12:18:27	
END GOOD	----	----	timestamp=1535397508	localtime=Aug 27 12:18:28	
I checked again and #c2 is right - the runtime is ~20 mins with most of the time being spent in reboot and verify. 
https://paste.googleplex.com/6021646745862144?raw
https://paste.googleplex.com/5817716229275648?raw
Amended CL to add test to network nightly only. 

Project Member

Comment 5 by bugdroid1@chromium.org, Aug 29

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/49a572ca43b7b21984dda9d80dd1b45b42cc7b6a

commit 49a572ca43b7b21984dda9d80dd1b45b42cc7b6a
Author: Kirtika Ruchandani <kirtika@google.com>
Date: Wed Aug 29 18:15:23 2018

autotest: wifi: Add a reboot stress test for wifi

 crbug.com/877854  discusses the motivation to have a wifi reboot stress
test. Adding a control file to stress_ClientTestReboot like bluetooth
and touch have done, with network_WlanDriver as the client-side test
to run after each reboot, seems a better option that writing a reboot
test from scratch. This test deliberately avoids the use of an AP so that
it can be run on pools other than pool:wificell.

BUG= chromium:877854 
TEST=Ran stress_ClientTestReboot.wifi on samus (WP2), eve (StP2),
blue (QCA6174A) and octopus (JfP2) at my desk. Reliability of ethernet
(i.e. finding eth0 when rebooting to get back the ssh connection) seems
a problem at my desk, masking wifi failures if there are any. Hopefully
the situation is better in the lab.

Change-Id: I7e8348683c3e427dbeb81a205f251b9a2092e0c8
Signed-off-by: Kirtika Ruchandani <kirtika@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1189722
Commit-Ready: Kirtika Ruchandani <kirtika@chromium.org>
Tested-by: Kirtika Ruchandani <kirtika@chromium.org>
Reviewed-by: Harpreet Grewal <harpreet@chromium.org>

[add] https://crrev.com/49a572ca43b7b21984dda9d80dd1b45b42cc7b6a/server/site_tests/stress_ClientTestReboot/control.wifi

Project Member

Comment 6 by bugdroid1@chromium.org, Oct 14

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/7d14a8a2c359cbebb4a04b09519e3512ad784fe5

commit 7d14a8a2c359cbebb4a04b09519e3512ad784fe5
Author: Kirtika Ruchandani <kirtika@google.com>
Date: Sun Oct 14 08:09:39 2018

autotest-server-tests: Add entry for stress_ClientRebootTest

We added reboot stress tests for wifi, bluetooth and touch.
None of these are running in the lab right now despite being
part of a regularly run suite, because of the missing ebuild entry.

BUG= chromium:877854 
TEST=None

Change-Id: I91a8b8aa83f331b35eefa1dc4c38cc9c9dc28636
Signed-off-by: Kirtika Ruchandani <kirtika@google.com>
Reviewed-on: https://chromium-review.googlesource.com/1278936
Commit-Ready: Kirtika Ruchandani <kirtika@chromium.org>
Tested-by: Kirtika Ruchandani <kirtika@chromium.org>
Reviewed-by: Harpreet Grewal <harpreet@chromium.org>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/7d14a8a2c359cbebb4a04b09519e3512ad784fe5/chromeos-base/autotest-server-tests/autotest-server-tests-9999.ebuild

Labels: wifi-test-failures
Status: Fixed (was: Untriaged)
This test has been running for a few months. It's somewhat flaky so far, where I've seen a few unlabeled ABORTs, and a few "Host did not return from reboot". The "did not return" seem to be USB ethernet flakiness mostly. Not sure if that's something we want to tackle in this context, so I'll just call this Fixed.

Sign in to add a comment