New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 906277 link

Starred by 2 users

Issue metadata

Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Get wifi interop lab up and going

Project Member Reported by harpreet@chromium.org, Nov 16

Issue description

WiFi interop lab needs to be resurrected. Below is a list of things that were done to get it going:



1. Analyze all existing 250+ APs in chaos chamber with a plan to remove older APs that matched the following criteria:
   - APs that are EOL (end of line); no longer supported by the manufacturer
   - APs that only support A/B/G and don't support N or AC

2. Plan the split of remaining ~190 APs across 3 RF chambers to get provide similar coverage across all chambers

3. Remove all APs from chaos lab

4. Install APs across 3 chambers per #2

5. Come up with configuration parameters for each AP in chaos chamber (chromeos3)

6. Reset and reconfigure all APs per new configurations

7. Update DHCP based on new AP install locations

8. Enable remote management on each AP to allow for making changes remotely as necessary during the bring up / testing phase

9. Update AP hostname -> RPM mapping to reflect new setup

10. Create new .conf file containing current set of APs with appropriate configurations

11. Create new control files to split the runs for all 2Ghz and 5Ghz channels

12. Make other necessary changes to the exiting code to get the new static setup working



Note - I may have missed some items that were necessary to get this going and most of the items listed above required multiple iterations and lots of testing.



Wifi interop lab will allow for testing ChromeOS DUTs against 50+ commercially sold APs across different channels/frequencies. All the APs are statically configured and this configuration is reflected in chaos_ap_list.conf file. 

Due to the time it takes to test a given DUT against all APs (~3hrs) and the fact that we can only run the test on 1 DUT at a time (due to current design limitations), the plan is to split the test runs across different DUTs on different days such that the test(s) will be run on each DUT once per week.

We also plan to continue adding newer APs to the interop lab as needed.
 
Cc: briannorris@chromium.org kirtika@chromium.org
Status: Started (was: Assigned)
Project Member

Comment 3 by bugdroid1@chromium.org, Nov 21

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/infra/suite_scheduler/+/7f7c7a3a71535520b579c5af6de5e919e7599fd0

commit 7f7c7a3a71535520b579c5af6de5e919e7599fd0
Author: harpreet <harpreet@google.com>
Date: Wed Nov 21 02:27:37 2018

[suite_scheduler] Update suites for interop lab

Separate suite is needed for each DUT in interop lab at the moment to be
able to schedule weekly tests on different days as we can not have
multiple DUTs running the chaos tests at once due to current design
limitations.

BUG=chromium:906277
TEST=None

Change-Id: Ib0f2fdd15a63784ab933171f2b1cae5f5dc678b4
Reviewed-on: https://chromium-review.googlesource.com/1342670
Commit-Ready: Harpreet Grewal <harpreet@chromium.org>
Tested-by: Harpreet Grewal <harpreet@chromium.org>
Reviewed-by: Harpreet Grewal <harpreet@chromium.org>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/7f7c7a3a71535520b579c5af6de5e919e7599fd0/configs/suite_scheduler.ini

Project Member

Comment 4 by bugdroid1@chromium.org, Nov 30

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/2337061e7383480c84c4b864044d7f68373cf44a

commit 2337061e7383480c84c4b864044d7f68373cf44a
Author: harpreet <harpreet@google.com>
Date: Fri Nov 30 00:09:02 2018

[wifi interop lab] Get interop lab up and going

Wifi interop lab will allow for testing ChromeOS DUTs against 50+
commercially sold APs across different channels/frequencies. All the APs
are statically configured and this configuration is reflected in
chaos_ap_list.conf.

Due to the time it takes to test a given DUT against all APs (~3hrs) and
the fact that we can only run the test on 1 DUT at a time (due to
current design limitations), the plan is to split the test runs across
different days such that the test(s) will be run on each DUT once per
week.

Old design docs:
https://docs.google.com/document/d/1tnp0JbL6Z9kxNsrBKeh2ZoTFxiH2WzFK49LsGZRGLd0/edit#
https://docs.google.com/document/d/1BxwqwTZ7c84nXyiJPpyzSX1hDvZTch4kWtPpIBluNa4/edit

AP configurations:
https://go/chaos-ap-configs

DUT setups:
http://go/wifi-device-life-cycle-chaos

Test run flow chart:
https://go/chaos-test-run-flow-chart

We also plan to continue adding newer APs to the interop lab as needed.

BUG=chromium:906277
TEST=Tested using kevin against all APs in chaos lab; unit tests (e.g.,
     `utils/unittest_suite.py -r server/cros/ap_configurators/`)

Change-Id: Ief141529fc0df264dac804e771aaba82fbcb5d65
Reviewed-on: https://chromium-review.googlesource.com/1341180
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Dinesh Kumar Sunkara <dsunkara@google.com>
Reviewed-by: Kirtika Ruchandani <kirtika@chromium.org>

[modify] https://crrev.com/2337061e7383480c84c4b864044d7f68373cf44a/server/cros/ap_configurators/ap_spec_unittest.py
[modify] https://crrev.com/2337061e7383480c84c4b864044d7f68373cf44a/server/cros/ap_config.py
[add] https://crrev.com/2337061e7383480c84c4b864044d7f68373cf44a/server/site_tests/network_WiFi_ChaosConnectDisconnect/control.static_2Ghz
[delete] https://crrev.com/4c99e31da15a2742abf14cb21ce1c0cbcbd6e3e2/server/cros/chaos_dynamic_ap_list.conf
[modify] https://crrev.com/2337061e7383480c84c4b864044d7f68373cf44a/server/cros/ap_configurators/ap_spec.py
[modify] https://crrev.com/2337061e7383480c84c4b864044d7f68373cf44a/server/cros/chaos_lib/static_runner.py
[add] https://crrev.com/2337061e7383480c84c4b864044d7f68373cf44a/server/site_tests/network_WiFi_ChaosConnectDisconnect/control.static_5Ghz
[modify] https://crrev.com/2337061e7383480c84c4b864044d7f68373cf44a/server/cros/network/chaos_clique_utils.py
[modify] https://crrev.com/2337061e7383480c84c4b864044d7f68373cf44a/test_suites/control.wifi_interop
[modify] https://crrev.com/2337061e7383480c84c4b864044d7f68373cf44a/server/cros/ap_configurators/ap_configurator_factory.py
[modify] https://crrev.com/2337061e7383480c84c4b864044d7f68373cf44a/client/common_lib/cros/network/iw_runner.py
[add] https://crrev.com/2337061e7383480c84c4b864044d7f68373cf44a/server/cros/chaos_ap_list.conf

Components: Infra>Client>ChromeOS
network_WiFi_ChaosConnectDisconnect test is now up and running in interop lab. For the most part it looks good except for random failures on few APs due to association time out. Best way to check the AP level results is via sponge as stainless only shows job level granularity (not test level). I have a separate thread going with the stainless author to see if they can add an option to view results at test granularity as well.


https://sponge.corp.google.com/invocations?searchFor=user%3Achromeos-test%20network_WiFi_ChaosConnectDisconnect


2Ghz

coral: 4 failures (zyxel_nbg6716__n_ch1_wpa2, netgear_wndr_3700_n_ch6_wpa2, belkin_f7d2301_n_ch6_wpa2, belkin_f7d5301_n_ch11_wpa2) -- 32 passed
https://sponge.corp.google.com/target?showTestCases=FAILURE&showTestCases=PASSED&showTestCases=&id=d5af1a15-1c05-4403-8489-64d8d5367ede&searchFor=user%3Achromeos-test+network_WiFi_ChaosConnectDisconnect&show=ALL&tab=Test+Cases&target=coral-release%2FR73-11343.0.0%2Fwifi_interop%2Fnetwork_WiFi_ChaosConnectDisconnect.static_2Ghz&view=FLAT&sortMethodsBy=METHOD&showRuns=SHOW_ALL_RUNS

samus: 4 failures (medialink_mwnwapr150__n_ch6_wpa2, buffalo_wzr-hp-g450h_n_ch6_wpa2, netgear_wndr_3700_n_ch6_wpa2, netgear_wnr834b__n_ch11_wpa2) -- 29 passed
https://sponge.corp.google.com/target?showTestCases=FAILURE&showTestCases=PASSED&showTestCases=&id=3f1dc4db-b820-449a-9d32-9cbbd1ff3c14&searchFor=user%3Achromeos-test+network_WiFi_ChaosConnectDisconnect&show=ALL&tab=Test+Cases&target=samus-release%2FR73-11355.0.0%2Fwifi_interop%2Fnetwork_WiFi_ChaosConnectDisconnect.static_2Ghz&view=FLAT&sortMethodsBy=METHOD&showRuns=SHOW_ALL_RUNS

kevin: 3 failures (zyxel_nbg6716__n_ch1_wpa2, netgear_wndr_3700_n_ch6_wpa2, belkin_f7d5301_n_ch11_wpa2) -- 33 passed
https://sponge.corp.google.com/target?showTestCases=FAILURE&showTestCases=PASSED&showTestCases=&id=53f8254c-76bb-44dc-91b4-54f5ca70cc68&searchFor=user%3Achromeos-test+network_WiFi_ChaosConnectDisconnect&show=ALL&tab=Test+Cases&target=kevin-release%2FR73-11355.0.0%2Fwifi_interop%2Fnetwork_WiFi_ChaosConnectDisconnect.static_2Ghz&view=FLAT&sortMethodsBy=METHOD&showRuns=SHOW_ALL_RUNS




5Ghz (1 failure on netgear_wndr_3700_n_ch157_wpa2) -- (23 passed)

coral: https://sponge.corp.google.com/target?showTestCases=FAILURE&showTestCases=PASSED&showTestCases=&id=54e0ddfa-620f-4eec-bb99-a591e59caa17&searchFor=user%3Achromeos-test+network_WiFi_ChaosConnectDisconnect&show=ALL&tab=Test+Cases&target=coral-release%2FR73-11343.0.0%2Fwifi_interop%2Fnetwork_WiFi_ChaosConnectDisconnect.static_5Ghz&view=FLAT&sortMethodsBy=METHOD&showRuns=SHOW_ALL_RUNS

samus: https://sponge.corp.google.com/target?showTestCases=FAILURE&showTestCases=PASSED&showTestCases=&id=3bffcbbb-81a6-4a38-b3df-6fd6e4ebc892&searchFor=user%3Achromeos-test+network_WiFi_ChaosConnectDisconnect&show=ALL&tab=Test+Cases&target=samus-release%2FR73-11355.0.0%2Fwifi_interop%2Fnetwork_WiFi_ChaosConnectDisconnect.static_5Ghz&view=FLAT&sortMethodsBy=METHOD&showRuns=SHOW_ALL_RUNS

kevin: https://sponge.corp.google.com/target?showTestCases=FAILURE&showTestCases=PASSED&showTestCases&id=542aa93e-fb51-4039-8706-9e632905eb1f&searchFor=user:chromeos-test+network_WiFi_ChaosConnectDisconnect&show=ALL&tab=Test+Cases&target=kevin-release/R73-11355.0.0/wifi_interop/network_WiFi_ChaosConnectDisconnect.static_5Ghz&view=FLAT&sortMethodsBy=METHOD&showRuns=SHOW_ALL_RUNS




Failure message needs to be improved/fixed...
https://stainless.corp.google.com/search?exclude_retried=true&first_date=2018-12-01&master_builder_name=&builder_name_number=&shard=&exclude_acts=true&builder_name=&master_builder_name_number=&owner=&retry=&exclude_cts=true&exclude_non_production=false&hostname=&board=&branch=&test=&suite=%5Ewifi%5C_interop%24&build=&status=FAIL&status=ERROR&status=ABORT&reason=&waterfall=&exclude_not_run=false&last_date=2018-12-07&exclude_non_release=true&exclude_au=true&model=&view=list&board_model=%5Ekevin%5C%2Fkevin%24&queued_date=%5E20181207%24




Cc: cros-conn-test-team@google.com
Project Member

Comment 8 by bugdroid1@chromium.org, Dec 17

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/infra/suite_scheduler/+/76ea445be0bafb21115495f5fea708cb590bffc1

commit 76ea445be0bafb21115495f5fea708cb590bffc1
Author: harpreet <harpreet@google.com>
Date: Mon Dec 17 21:38:42 2018

[suite_scheduler] Update wifi interop suite schedule

WiFi interop suites need to be scheduled to run at different hours
during the day.

BUG=chromium:906277
TEST=None
CQ-DEPEND=CL:1372282

Change-Id: Id47b409f2b88b821860f7a063f2c79f482cf5140

[modify] https://crrev.com/76ea445be0bafb21115495f5fea708cb590bffc1/configs/suite_scheduler.ini

Sign in to add a comment