Issue metadata
Sign in to add a comment
|
netperf uprev breaks network_wifi_perf (suite:wifi_perf) |
||||||||||||||||||||||
Issue descriptionThis change landed today: https://chromium-review.googlesource.com/c/chromiumos/overlays/portage-stable/+/877415 and suite:wifi_perf runs have been failing since the first client builds started using netperf-2.7. The APs running netserver are still using netperf-2.4. The issue is netperf-2.7 is not compatible with netperf-2.4 and attempts with one talking to the other will time out. Which is exactly what we are seeing in the test logs starting with job id 173917769: ... 02/01 09:21:45.883 DEBUG| ssh_host:0301| Running (ssh) 'ifconfig wlan0 up' from '_call_run_once|run_once|do_run|powersave_switch|run|run_very_slowly' 02/01 09:21:46.358 DEBUG| ssh_host:0301| Running (ssh) 'iw dev wlan0 set power_save on' from '_call_run_once|run_once|do_run|powersave_switch|run|run_very_slowly' 02/01 09:21:46.827 INFO | netperf_session:0065| Warming up the client WiFi rate controller. 02/01 09:21:46.853 DEBUG| ssh_host:0301| Running (ssh) 'ls netserver >/dev/null 2>&1' from 'warmup_stations|warmup_wifi_part|__init__|must_be_installed|run|run_very_slowly' 02/01 09:21:47.481 DEBUG| ssh_host:0301| Running (ssh) 'ls /bin/netserver /sbin/netserver /system/bin/netserver /system/xbin/netserver /usr/bin/netserver /usr/sbin/netserver /usr/local/bin/netserver /usr/local/sbin/netserver 2> /dev/null' from 'warmup_wifi_part|__init__|must_be_installed|get_install_path|run|run_very_slowly' 02/01 09:21:48.024 DEBUG| utils:0282| [stdout] /usr/sbin/netserver 02/01 09:21:48.062 DEBUG| ssh_host:0301| Running (ssh) 'ls netperf >/dev/null 2>&1' from 'warmup_stations|warmup_wifi_part|__init__|must_be_installed|run|run_very_slowly' 02/01 09:21:48.571 DEBUG| ssh_host:0301| Running (ssh) 'ls /bin/netperf /sbin/netperf /system/bin/netperf /system/xbin/netperf /usr/bin/netperf /usr/sbin/netperf /usr/local/bin/netperf /usr/local/sbin/netperf 2> /dev/null' from 'warmup_wifi_part|__init__|must_be_installed|get_install_path|run|run_very_slowly' 02/01 09:21:49.047 DEBUG| utils:0282| [stdout] /usr/local/bin/netperf 02/01 09:21:49.095 INFO | netperf_runner:0518| Starting netserver... 02/01 09:21:49.136 DEBUG| ssh_host:0301| Running (ssh) 'pkill netserver' from 'warmup_wifi_part|__enter__|_restart_netserv|_kill_netserv|run|run_very_slowly' 02/01 09:21:49.841 DEBUG| ssh_host:0301| Running (ssh) '/usr/sbin/netserver -p 12865 >/dev/null 2>&1' from 'warmup_stations|warmup_wifi_part|__enter__|_restart_netserv|run|run_very_slowly' 02/01 09:21:50.466 DEBUG| ssh_host:0301| Running (ssh) 'iptables -I INPUT -s 192.168.0.254/32 -p tcp -m tcp -j ACCEPT' from 'warmup_wifi_part|__enter__|_restart_netserv|firewall_open|run|run_very_slowly' 02/01 09:21:50.980 DEBUG| ssh_host:0301| Running (ssh) 'iptables -I INPUT -s 192.168.0.254/32 -p udp -m udp -j ACCEPT' from 'warmup_wifi_part|__enter__|_restart_netserv|firewall_open|run|run_very_slowly' 02/01 09:21:53.488 DEBUG| netperf_runner:0548| Running netperf client. 02/01 09:21:53.490 INFO | netperf_runner:0549| Running netperf for 2 seconds. 02/01 09:21:53.518 DEBUG| ssh_host:0301| Running (ssh) '/usr/local/bin/netperf -H 192.168.0.254 -p 12865 -t TCP_STREAM -l 2 -- -P 0,12866' from 'do_run|warmup_stations|warmup_wifi_part|run|run|run_very_slowly' 02/01 09:22:55.880 WARNI| utils:0925| run process timeout (62) fired on: /usr/bin/ssh -a -x -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22 chromeos15-row1-rack9-host2 "export LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack::warmup_wifi_part|run|run] -> ssh_run(/usr/local/bin/netperf -H 192.168.0.254 -p 12865 -t TCP_STREAM -l 2 -- -P 0,12866)\";fi; /usr/local/bin/netperf -H 192.168.0.254 -p 12865 -t TCP_STREAM -l 2 -- -P 0,12866" 02/01 09:22:56.913 DEBUG| ssh_host:0301| Running (ssh) 'iptables -D INPUT -s 192.168.0.254/32 -p tcp -m tcp -j ACCEPT' from 'warmup_stations|warmup_wifi_part|__exit__|firewall_cleanup|run|run_very_slowly' 02/01 09:22:57.374 DEBUG| ssh_host:0301| Running (ssh) 'iptables -D INPUT -s 192.168.0.254/32 -p udp -m udp -j ACCEPT' from 'warmup_stations|warmup_wifi_part|__exit__|firewall_cleanup|run|run_very_slowly' 02/01 09:22:57.800 DEBUG| ssh_host:0301| Running (ssh) 'pkill netserver' from 'warmup_stations|warmup_wifi_part|__exit__|_kill_netserv|run|run_very_slowly' 02/01 09:22:58.471 DEBUG| ssh_host:0301| Running (ssh) 'kill 5516' from '__exit__|assert_disconnect_count|__exit__|stop|run|run_very_slowly' 02/01 09:22:58.914 DEBUG| abstract_ssh:0413| get_file. source: /tmp/autoserv-sL8yr9/iw_event.log, dest: ./debug/iw_event_0.log, delete_dest: False,preserve_perm: True, preserve_symlinks:False 02/01 09:22:58.938 DEBUG| ssh_host:0301| Running (ssh) 'rsync --version' from 'stop|get_file|use_rsync|check_rsync|run|run_very_slowly' 02/01 09:22:59.386 DEBUG| abstract_ssh:0425| Using Rsync. 02/01 09:22:59.387 DEBUG| utils:0214| Running 'rsync -L --timeout=1800 --rsh='/usr/bin/ssh -a -x -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22' -az --no-o --no-g root@chromeos15-row1-rack9-host2:"/tmp/autoserv-sL8yr9/iw_event.log" "/usr/local/autotest/results/173917769-chromeos-test/network_WiFi_Perf.11g_aes/debug/iw_event_0.log"' 02/01 09:22:59.893 INFO | iw_event_logger:0088| iw event log saved to ./debug/iw_event_0.log 02/01 09:22:59.898 DEBUG| test:0410| Test failed due to Timeout encountered: /usr/bin/ssh -a -x -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22 chromeos15-row1-rack9-host2 "export LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack::warmup_wifi_part|run|run] -> ssh_run(/usr/local/bin/netperf -H 192.168.0.254 -p 12865 -t TCP_STREAM -l 2 -- -P 0,12866)\";fi; /usr/local/bin/netperf -H 192.168.0.254 -p 12865 -t TCP_STREAM -l 2 -- -P 0,12866" * Command: /usr/bin/ssh -a -x -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22 chromeos15-row1-rack9-host2 "export LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack::warmup_wifi_part|run|run] -> ssh_run(/usr/local/bin/netperf -H 192.168.0.254 -p 12865 -t TCP_STREAM -l 2 -- -P 0,12866)\";fi; /usr/local/bin/netperf -H 192.168.0.254 -p 12865 -t TCP_STREAM -l 2 -- -P 0,12866" Exit status: 143
,
Feb 2 2018
While Brian is out, Ed, Bernie, do you know of any shortcut to sync just the whirlwind build to the manifest used for our latest testlab router images? That would be "trybot-whirlwind-test-ap-tryjob/R65-10241.0.0-b8". We'd like to keep that image, and just uprev netperf. Upreving the image would mean testing both wifi+bt all over, and we dont have the cycles for that.
,
Feb 2 2018
,
Feb 2 2018
+ Ruchi, Dmitry, in case network_WiFi_BluetoothScanPerf or similar tests were running in the test lab and got borked by this.
,
Feb 3 2018
If we need a one off image with a particular change, we would usually make a stabilize branch and do a build off of it, does that sound like it would work here? I am not sure what happens with the image after generation for this sort of test setup though.
,
Feb 3 2018
"I am not sure what happens with the image after generation for this sort of test setup though." IIUC, the following: 1. There is a mechanism to specify somewhere, "hey dont get rid of this image archive, keep it visible forever on pantheon.corp / devserver". Brian, Harpreet and Ed know more about this. We make it available forever 2. This trybot identifier goes into network_WiFi_UpdateRouter autotest, which fetches this image and applies it to the routers in the test lab nightly. 3. We'd like to find a way to make one off changes to the image (so a stabilize branch sounds good!), until someone has the cycles to uprev or finds a better (less manual testing) process to uprev router images.
,
Feb 3 2018
ecgh references crbug.com/770255 in https://chromium-review.googlesource.com/757122 I'll repeat the very high level instructions and then try to fill those in the instructions at go/upgrade-test-ap : ---- Now that we have a new whirlwind testbed-ap build succeeding with wireless-4.2 ( bug 702007 ), need to: 1) Produce a new build (at gs://chromeos-image-archive/trybot-whirlwind-test-ap/) using the test_ap builder (https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/test_ap/, cbuildbot --remote whirlwind-test-ap). 2) Verify build by checking wifi_matfunc results for a few DUTs. 3) Update the whirlwind stable version in network_WiFi_UpdateRouter to the new build. 4) Check that lab whirlwinds move to the new version. 5) Check for any variation in wifi test results. Detailed instructions don't appear to be available in existing documentation on how to do (1). I'll dig in the test_ap build logs to see if that command line was included. Then include those in the go/cros-wifi-test-setup document since I can't find the "WiFi / BT AVL Qualification Test Equipments” that is supposed to include these instructions.
,
Feb 3 2018
Don Garrett suggested we could also reference an existing "stable" (R65) branch and manually add any additional changes as CLs to the "cros tryjob" that we send to the test_ap try-server. BTW, Don would like to split the test-ap try-server into three separate try servers (one for each board that this server builds) and just specify the three try-servers on the command line. This would get the team one step closer to eliminating some complexity in the tryjob code that handles multiple boards. He's going to file another issue to track this.
,
Feb 3 2018
I see, if these don't need to be signed, and they are truly tryjobs, than putting patches in with a tryjob against a release branch seems quite reasonable. The stabilize branch is mostly so that we can have a particular version number fork for this (e.g. 1234.1), and generate officially signed artifacts.
,
Feb 3 2018
Yes, these are tryjobs, they just have to be run on the test_ap builder. I think the new version of "cbuildbot --remote whirlwind-test-ap" is "cros tryjob whirlwind-test-ap". The new build should then appear here: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/test_ap/ Then network_WiFi_UpdateRouter can be updated to point to it, like this: https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/757122
,
Feb 3 2018
Ok - thanks Edward! To be able to "patch" the builds, we should be based on release branches. I'd like to propose going forward, we use: cros tryjob -b release-R65-10323.B -g 877415 --hwtest --committer grundler@google.com stumpy-test-ap-tryjob panther-test-ap-tryjob whirlwind-test-ap-tryjob This base the build on R65 release branch. Even though release branches will change, I expect it to be at a more reasonable rate than ToT (which is the default if we specify nothing). The "-g" specifies the list of CLs if our CL hasn't been cherry-picked into that release branch. The "--hwtest" was advice giving in one of the docs to force the artifacts to be permanently stored. The three builds I kicked off above are visible here: http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8955652615874123104 https://uberchromegw.corp.google.com/i/chromiumos.tryserver/waterfall?committer=grundler@google.com&builder=test_ap http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8955652615424143712 https://uberchromegw.corp.google.com/i/chromiumos.tryserver/waterfall?committer=grundler@google.com&builder=test_ap http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8955652614859204656 https://uberchromegw.corp.google.com/i/chromiumos.tryserver/waterfall?committer=grundler@google.com&builder=test_ap Since none of these is using the current version of netperf, I'll have to update all three and will start with the whirlwind devices since they are supposed to the closest to R65 release branch.
,
Feb 6 2018
SGTM. As discussed in person, it's probably wise to make use of our existing release stabilization anyway. Don't know why we were happy to accept ToT instability previously...it already bit us on issue 795448. One would hope such a bug wouldn't make it out to a "stable" Whirlwind release.
,
Feb 6 2018
,
Feb 6 2018
Thanks Brian! The three builds I previously kicked off failed: 1) I was missing a second patch that added netperf user/group 2) I didn't cherry-pick them to the R65 branch (might not be required but it seems like a good idea) Successfully built a whirlwind image with: $ cros tryjob -b release-R65-10323.B -g 900169 -g 902706 --hwtest --committer grundler@google.com whirlwind-test-ap-tryjob And test_ap builder output is here: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/test_ap/builds/15 I tried to use "cros flash" to push the images to a whirlwind router: cros flash chromeos15-row1-rack4-host5-router.cros xbuddy://remote/trybot-whirlwind-test-ap-tryjob/R65-10323.15.0-c39332 And that failed with: 17:59:55: WARNING: GS_ERROR: ServiceException: 401 Anonymous caller does not have storage.objects.get access to chromeos-image-archive/trybot-whirlwind-test-ap-tryjob/R65-10323.15.0-c39332. I've sent mail to ask for help from infra team since my ~/.boto file looks intact from last year. Once I get whirlwind sorted out, I'll try to rinse and repeat for the stumpy and panther images still in use. In theory those should work exactly the same.
,
Feb 6 2018
grundler@ - fyi I just re-imaged chromeos15-row1-rack4-host5-router with R65-10323.15.0-c39332
,
Feb 6 2018
Really? Are you sure? I sorted out the access issues with chromeos-infra team and upgraded the same router this morning: (cr) ((93c015d...)) grundler@firesword ~/trunk/src/scripts $ cros flash chromeos15-row1-rack4-host5-router.cros xbuddy://remote/trybot-whirlwind-test-ap-tryjob/R65-10323.15.0-c39332 09:25:56: NOTICE: Preparing to update the remote device chromeos15-row1-rack4-host5-router.cros 09:26:00: NOTICE: Using image trybot-whirlwind-test-ap-tryjob/R65-10323.15.0-c39332/chromiumos_test_image.bin [################################################################################] 100% 09:31:25: NOTICE: Finalizing image. 09:32:47: NOTICE: Update completed. 09:33:02: WARNING: Could not copy /var/log/quick-provision.log from device. 09:33:42: NOTICE: rebooting device... 09:36:52: ERROR: Reboot has not completed after 180 seconds; giving up. But it actually did update and just took longer to reboot or something. ... 64 bytes from 100.115.124.102 (100.115.124.102): icmp_seq=269 ttl=56 time=5.30 ms 64 bytes from 100.115.124.102 (100.115.124.102): icmp_seq=270 ttl=56 time=5.01 ms 64 bytes from 100.115.124.102 (100.115.124.102): icmp_seq=329 ttl=56 time=4.49 ms 64 bytes from 100.115.124.102 (100.115.124.102): icmp_seq=330 ttl=56 time=4.65 ms [ note the 59 second gap ] Whirlwind eMMC is very slow (<5MB/s write speed) and the device has 1GB of RAM...so it's likely the "sync" before unmounting the file systems (and rebooting) could have take a long time. And then add another 59 seconds just for reboot.
,
Feb 6 2018
BTW, Harpreet, I thought I have this system locked - please email or warn me before changing anything on the system. I will need your help to update the rest of the whirlwind devices in the lab once I finish testing this device. My next step is to kick off the test suite and verify wifiperf test works with this image. And then run the entire suite to verify there are no regressions. I'm still working out how to build stumpy/panther test_ap images and will repeat the process for those AND will need your help to upgrade those as well. thank you! :)
,
Feb 6 2018
Nevermind... looks like there was a typo in my command. I updated chromeos15-row1-rack9-host5 instead of chromeos15-row1-rack4-host5. :-/ Re #17. correct it 15-1-4-5 was locked but I was trying to check if would see the same issue as you did in #14. Sorry about that, will ping next time. We should be able to use network_WiFi_UpdateRouter to update all the routers.
,
Feb 7 2018
Ok - good! :) suite:wifi_perf seems to have run but still have failures due to sox missinge (despite having run "cros deploy ... sox") Before starting the test: B=kefka ; HOST=chromeos15-row1-rack4-host5.cros; test_that -b $B $HOST suite:wifi_perf and the latest output: /tmp/test_that_results_oBDjDo/results-01-network_WiFi_BluetoothStreamPerf.11a [ FAILED ] /tmp/test_that_results_oBDjDo/results-01-network_WiFi_BluetoothStreamPerf.11a ERROR: sox command is not installed. sudo emerge sox to install sox in chroot /tmp/test_that_results_oBDjDo/results-01-network_WiFi_BluetoothStreamPerf.11a/network_WiFi_BluetoothStreamPerf.11a [ FAILED ] /tmp/test_that_results_oBDjDo/results-01-network_WiFi_BluetoothStreamPerf.11a/network_WiFi_BluetoothStreamPerf.11a ERROR: sox command is not installed. sudo emerge sox to install sox in chroot /tmp/test_that_results_oBDjDo/results-02-network_WiFi_BluetoothStreamPerf.11b [ FAILED ] /tmp/test_that_results_oBDjDo/results-02-network_WiFi_BluetoothStreamPerf.11b ERROR: sox command is not installed. sudo emerge sox to install sox in chroot /tmp/test_that_results_oBDjDo/results-02-network_WiFi_BluetoothStreamPerf.11b/network_WiFi_BluetoothStreamPerf.11b [ FAILED ] /tmp/test_that_results_oBDjDo/results-02-network_WiFi_BluetoothStreamPerf.11b/network_WiFi_BluetoothStreamPerf.11b ERROR: sox command is not installed. sudo emerge sox to install sox in chroot /tmp/test_that_results_oBDjDo/results-03-network_WiFi_Perf.ht20_aes [ FAILED ] /tmp/test_that_results_oBDjDo/results-03-network_WiFi_Perf.ht20_aes ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-03-network_WiFi_Perf.ht20_aes/network_WiFi_Perf.ht20_aes [ FAILED ] /tmp/test_that_results_oBDjDo/results-03-network_WiFi_Perf.ht20_aes/network_WiFi_Perf.ht20_aes ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-03-network_WiFi_Perf.ht20_aes/network_WiFi_Perf.ht20_aes ch001_modeHT20_psk_PSon_signal -36 /tmp/test_that_results_oBDjDo/results-03-network_WiFi_Perf.ht20_aes/network_WiFi_Perf.ht20_aes ch001_modeHT20_psk_PSon_tcp_rx_throughput 73.35+-0.13 /tmp/test_that_results_oBDjDo/results-03-network_WiFi_Perf.ht20_aes/network_WiFi_Perf.ht20_aes ch001_modeHT20_psk_PSon_tcp_tx_throughput 82.56+-0.39 /tmp/test_that_results_oBDjDo/results-03-network_WiFi_Perf.ht20_aes/network_WiFi_Perf.ht20_aes ch001_modeHT20_psk_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-03-network_WiFi_Perf.ht20_aes/network_WiFi_Perf.ht20_aes ch001_modeHT20_psk_PSon_udp_tx_throughput 104.52+-1.20 /tmp/test_that_results_oBDjDo/results-04-network_WiFi_Perf.vht80 [ FAILED ] /tmp/test_that_results_oBDjDo/results-04-network_WiFi_Perf.vht80 ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-04-network_WiFi_Perf.vht80/network_WiFi_Perf.vht80 [ FAILED ] /tmp/test_that_results_oBDjDo/results-04-network_WiFi_Perf.vht80/network_WiFi_Perf.vht80 ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-04-network_WiFi_Perf.vht80/network_WiFi_Perf.vht80 ch044_mode11A_none_PSon_signal -43 /tmp/test_that_results_oBDjDo/results-04-network_WiFi_Perf.vht80/network_WiFi_Perf.vht80 ch044_mode11A_none_PSon_tcp_rx_throughput 171.51+-1.88 /tmp/test_that_results_oBDjDo/results-04-network_WiFi_Perf.vht80/network_WiFi_Perf.vht80 ch044_mode11A_none_PSon_tcp_tx_throughput 153.07+-0.20 /tmp/test_that_results_oBDjDo/results-04-network_WiFi_Perf.vht80/network_WiFi_Perf.vht80 ch044_mode11A_none_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-04-network_WiFi_Perf.vht80/network_WiFi_Perf.vht80 ch044_mode11A_none_PSon_udp_tx_throughput 186.94+-0.19 /tmp/test_that_results_oBDjDo/results-05-network_WiFi_Perf.11a_aes [ FAILED ] /tmp/test_that_results_oBDjDo/results-05-network_WiFi_Perf.11a_aes ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-05-network_WiFi_Perf.11a_aes/network_WiFi_Perf.11a_aes [ FAILED ] /tmp/test_that_results_oBDjDo/results-05-network_WiFi_Perf.11a_aes/network_WiFi_Perf.11a_aes ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-05-network_WiFi_Perf.11a_aes/network_WiFi_Perf.11a_aes ch044_mode11A_psk_PSon_signal -44 /tmp/test_that_results_oBDjDo/results-05-network_WiFi_Perf.11a_aes/network_WiFi_Perf.11a_aes ch044_mode11A_psk_PSon_tcp_rx_throughput 23.66+-0.01 /tmp/test_that_results_oBDjDo/results-05-network_WiFi_Perf.11a_aes/network_WiFi_Perf.11a_aes ch044_mode11A_psk_PSon_tcp_tx_throughput 22.56+-0.41 /tmp/test_that_results_oBDjDo/results-05-network_WiFi_Perf.11a_aes/network_WiFi_Perf.11a_aes ch044_mode11A_psk_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-05-network_WiFi_Perf.11a_aes/network_WiFi_Perf.11a_aes ch044_mode11A_psk_PSon_udp_tx_throughput 33.60+-0.01 /tmp/test_that_results_oBDjDo/results-06-network_WiFi_Perf.ht40_aes [ FAILED ] /tmp/test_that_results_oBDjDo/results-06-network_WiFi_Perf.ht40_aes ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-06-network_WiFi_Perf.ht40_aes/network_WiFi_Perf.ht40_aes [ FAILED ] /tmp/test_that_results_oBDjDo/results-06-network_WiFi_Perf.ht40_aes/network_WiFi_Perf.ht40_aes ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-06-network_WiFi_Perf.ht40_aes/network_WiFi_Perf.ht40_aes ch001_modeHT40p_psk_PSon_signal -37 /tmp/test_that_results_oBDjDo/results-06-network_WiFi_Perf.ht40_aes/network_WiFi_Perf.ht40_aes ch001_modeHT40p_psk_PSon_tcp_rx_throughput 74.20+-0.06 /tmp/test_that_results_oBDjDo/results-06-network_WiFi_Perf.ht40_aes/network_WiFi_Perf.ht40_aes ch001_modeHT40p_psk_PSon_tcp_tx_throughput 142.83+-1.36 /tmp/test_that_results_oBDjDo/results-06-network_WiFi_Perf.ht40_aes/network_WiFi_Perf.ht40_aes ch001_modeHT40p_psk_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-06-network_WiFi_Perf.ht40_aes/network_WiFi_Perf.ht40_aes ch001_modeHT40p_psk_PSon_udp_tx_throughput 191.88+-4.17 /tmp/test_that_results_oBDjDo/results-07-network_WiFi_Perf.11a_tkip [ FAILED ] /tmp/test_that_results_oBDjDo/results-07-network_WiFi_Perf.11a_tkip ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-07-network_WiFi_Perf.11a_tkip/network_WiFi_Perf.11a_tkip [ FAILED ] /tmp/test_that_results_oBDjDo/results-07-network_WiFi_Perf.11a_tkip/network_WiFi_Perf.11a_tkip ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-07-network_WiFi_Perf.11a_tkip/network_WiFi_Perf.11a_tkip ch044_mode11A_psk_PSon_signal -44 /tmp/test_that_results_oBDjDo/results-07-network_WiFi_Perf.11a_tkip/network_WiFi_Perf.11a_tkip ch044_mode11A_psk_PSon_tcp_rx_throughput 23.61+-0.07 /tmp/test_that_results_oBDjDo/results-07-network_WiFi_Perf.11a_tkip/network_WiFi_Perf.11a_tkip ch044_mode11A_psk_PSon_tcp_tx_throughput 22.49+-0.41 /tmp/test_that_results_oBDjDo/results-07-network_WiFi_Perf.11a_tkip/network_WiFi_Perf.11a_tkip ch044_mode11A_psk_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-07-network_WiFi_Perf.11a_tkip/network_WiFi_Perf.11a_tkip ch044_mode11A_psk_PSon_udp_tx_throughput 33.56+-0.02 /tmp/test_that_results_oBDjDo/results-08-network_WiFi_Perf.ht40 [ FAILED ] /tmp/test_that_results_oBDjDo/results-08-network_WiFi_Perf.ht40 ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-08-network_WiFi_Perf.ht40/network_WiFi_Perf.ht40 [ FAILED ] /tmp/test_that_results_oBDjDo/results-08-network_WiFi_Perf.ht40/network_WiFi_Perf.ht40 ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-08-network_WiFi_Perf.ht40/network_WiFi_Perf.ht40 ch001_modeHT40p_none_PSon_signal -36 /tmp/test_that_results_oBDjDo/results-08-network_WiFi_Perf.ht40/network_WiFi_Perf.ht40 ch001_modeHT40p_none_PSon_tcp_rx_throughput 74.43+-0.12 /tmp/test_that_results_oBDjDo/results-08-network_WiFi_Perf.ht40/network_WiFi_Perf.ht40 ch001_modeHT40p_none_PSon_tcp_tx_throughput 146.42+-2.78 /tmp/test_that_results_oBDjDo/results-08-network_WiFi_Perf.ht40/network_WiFi_Perf.ht40 ch001_modeHT40p_none_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-08-network_WiFi_Perf.ht40/network_WiFi_Perf.ht40 ch001_modeHT40p_none_PSon_udp_tx_throughput 192.29+-2.41 /tmp/test_that_results_oBDjDo/results-09-network_WiFi_Perf.ht20 [ FAILED ] /tmp/test_that_results_oBDjDo/results-09-network_WiFi_Perf.ht20 ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-09-network_WiFi_Perf.ht20/network_WiFi_Perf.ht20 [ FAILED ] /tmp/test_that_results_oBDjDo/results-09-network_WiFi_Perf.ht20/network_WiFi_Perf.ht20 ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-09-network_WiFi_Perf.ht20/network_WiFi_Perf.ht20 ch001_modeHT20_none_PSon_signal -36 /tmp/test_that_results_oBDjDo/results-09-network_WiFi_Perf.ht20/network_WiFi_Perf.ht20 ch001_modeHT20_none_PSon_tcp_rx_throughput 73.39+-0.08 /tmp/test_that_results_oBDjDo/results-09-network_WiFi_Perf.ht20/network_WiFi_Perf.ht20 ch001_modeHT20_none_PSon_tcp_tx_throughput 84.10+-1.22 /tmp/test_that_results_oBDjDo/results-09-network_WiFi_Perf.ht20/network_WiFi_Perf.ht20 ch001_modeHT20_none_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-09-network_WiFi_Perf.ht20/network_WiFi_Perf.ht20 ch001_modeHT20_none_PSon_udp_tx_throughput 105.49+-1.34 /tmp/test_that_results_oBDjDo/results-10-network_WiFi_Perf.11g_wep [ FAILED ] /tmp/test_that_results_oBDjDo/results-10-network_WiFi_Perf.11g_wep ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-10-network_WiFi_Perf.11g_wep/network_WiFi_Perf.11g_wep [ FAILED ] /tmp/test_that_results_oBDjDo/results-10-network_WiFi_Perf.11g_wep/network_WiFi_Perf.11g_wep ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-10-network_WiFi_Perf.11g_wep/network_WiFi_Perf.11g_wep ch006_mode11G_wep_PSon_signal -37 /tmp/test_that_results_oBDjDo/results-10-network_WiFi_Perf.11g_wep/network_WiFi_Perf.11g_wep ch006_mode11G_wep_PSon_tcp_rx_throughput 23.45+-0.04 /tmp/test_that_results_oBDjDo/results-10-network_WiFi_Perf.11g_wep/network_WiFi_Perf.11g_wep ch006_mode11G_wep_PSon_tcp_tx_throughput 22.82+-0.12 /tmp/test_that_results_oBDjDo/results-10-network_WiFi_Perf.11g_wep/network_WiFi_Perf.11g_wep ch006_mode11G_wep_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-10-network_WiFi_Perf.11g_wep/network_WiFi_Perf.11g_wep ch006_mode11G_wep_PSon_udp_tx_throughput 33.34+-0.01 /tmp/test_that_results_oBDjDo/results-11-network_WiFi_Perf.11b_wep [ FAILED ] /tmp/test_that_results_oBDjDo/results-11-network_WiFi_Perf.11b_wep ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-11-network_WiFi_Perf.11b_wep/network_WiFi_Perf.11b_wep [ FAILED ] /tmp/test_that_results_oBDjDo/results-11-network_WiFi_Perf.11b_wep/network_WiFi_Perf.11b_wep ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-11-network_WiFi_Perf.11b_wep/network_WiFi_Perf.11b_wep ch006_mode11B_wep_PSon_signal -36 /tmp/test_that_results_oBDjDo/results-11-network_WiFi_Perf.11b_wep/network_WiFi_Perf.11b_wep ch006_mode11B_wep_PSon_tcp_rx_throughput 5.58+-0.03 /tmp/test_that_results_oBDjDo/results-11-network_WiFi_Perf.11b_wep/network_WiFi_Perf.11b_wep ch006_mode11B_wep_PSon_tcp_tx_throughput 5.36+-0.05 /tmp/test_that_results_oBDjDo/results-11-network_WiFi_Perf.11b_wep/network_WiFi_Perf.11b_wep ch006_mode11B_wep_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-11-network_WiFi_Perf.11b_wep/network_WiFi_Perf.11b_wep ch006_mode11B_wep_PSon_udp_tx_throughput 7.13+-0.05 /tmp/test_that_results_oBDjDo/results-12-network_WiFi_Perf.11b [ FAILED ] /tmp/test_that_results_oBDjDo/results-12-network_WiFi_Perf.11b ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-12-network_WiFi_Perf.11b/network_WiFi_Perf.11b [ FAILED ] /tmp/test_that_results_oBDjDo/results-12-network_WiFi_Perf.11b/network_WiFi_Perf.11b ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-12-network_WiFi_Perf.11b/network_WiFi_Perf.11b ch006_mode11B_none_PSon_signal -35 /tmp/test_that_results_oBDjDo/results-12-network_WiFi_Perf.11b/network_WiFi_Perf.11b ch006_mode11B_none_PSon_tcp_rx_throughput 5.60+-0.03 /tmp/test_that_results_oBDjDo/results-12-network_WiFi_Perf.11b/network_WiFi_Perf.11b ch006_mode11B_none_PSon_tcp_tx_throughput 5.26+-0.13 /tmp/test_that_results_oBDjDo/results-12-network_WiFi_Perf.11b/network_WiFi_Perf.11b ch006_mode11B_none_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-12-network_WiFi_Perf.11b/network_WiFi_Perf.11b ch006_mode11B_none_PSon_udp_tx_throughput 7.14+-0.05 /tmp/test_that_results_oBDjDo/results-13-network_WiFi_Perf.11g_aes [ FAILED ] /tmp/test_that_results_oBDjDo/results-13-network_WiFi_Perf.11g_aes ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-13-network_WiFi_Perf.11g_aes/network_WiFi_Perf.11g_aes [ FAILED ] /tmp/test_that_results_oBDjDo/results-13-network_WiFi_Perf.11g_aes/network_WiFi_Perf.11g_aes ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-13-network_WiFi_Perf.11g_aes/network_WiFi_Perf.11g_aes ch006_mode11G_psk_PSon_signal -37 /tmp/test_that_results_oBDjDo/results-13-network_WiFi_Perf.11g_aes/network_WiFi_Perf.11g_aes ch006_mode11G_psk_PSon_tcp_rx_throughput 23.46+-0.02 /tmp/test_that_results_oBDjDo/results-13-network_WiFi_Perf.11g_aes/network_WiFi_Perf.11g_aes ch006_mode11G_psk_PSon_tcp_tx_throughput 22.79+-0.03 /tmp/test_that_results_oBDjDo/results-13-network_WiFi_Perf.11g_aes/network_WiFi_Perf.11g_aes ch006_mode11G_psk_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-13-network_WiFi_Perf.11g_aes/network_WiFi_Perf.11g_aes ch006_mode11G_psk_PSon_udp_tx_throughput 33.26+-0.02 /tmp/test_that_results_oBDjDo/results-14-network_WiFi_Perf.11a [ FAILED ] /tmp/test_that_results_oBDjDo/results-14-network_WiFi_Perf.11a ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-14-network_WiFi_Perf.11a/network_WiFi_Perf.11a [ FAILED ] /tmp/test_that_results_oBDjDo/results-14-network_WiFi_Perf.11a/network_WiFi_Perf.11a ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-14-network_WiFi_Perf.11a/network_WiFi_Perf.11a ch044_mode11A_none_PSon_signal -44 /tmp/test_that_results_oBDjDo/results-14-network_WiFi_Perf.11a/network_WiFi_Perf.11a ch044_mode11A_none_PSon_tcp_rx_throughput 23.75+-0.06 /tmp/test_that_results_oBDjDo/results-14-network_WiFi_Perf.11a/network_WiFi_Perf.11a ch044_mode11A_none_PSon_tcp_tx_throughput 22.79+-0.20 /tmp/test_that_results_oBDjDo/results-14-network_WiFi_Perf.11a/network_WiFi_Perf.11a ch044_mode11A_none_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-14-network_WiFi_Perf.11a/network_WiFi_Perf.11a ch044_mode11A_none_PSon_udp_tx_throughput 33.93+-0.03 /tmp/test_that_results_oBDjDo/results-15-network_WiFi_Perf.11g [ FAILED ] /tmp/test_that_results_oBDjDo/results-15-network_WiFi_Perf.11g ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-15-network_WiFi_Perf.11g/network_WiFi_Perf.11g [ FAILED ] /tmp/test_that_results_oBDjDo/results-15-network_WiFi_Perf.11g/network_WiFi_Perf.11g ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-15-network_WiFi_Perf.11g/network_WiFi_Perf.11g ch006_mode11G_none_PSon_signal -37 /tmp/test_that_results_oBDjDo/results-15-network_WiFi_Perf.11g/network_WiFi_Perf.11g ch006_mode11G_none_PSon_tcp_rx_throughput 23.69+-0.02 /tmp/test_that_results_oBDjDo/results-15-network_WiFi_Perf.11g/network_WiFi_Perf.11g ch006_mode11G_none_PSon_tcp_tx_throughput 22.98+-0.06 /tmp/test_that_results_oBDjDo/results-15-network_WiFi_Perf.11g/network_WiFi_Perf.11g ch006_mode11G_none_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-15-network_WiFi_Perf.11g/network_WiFi_Perf.11g ch006_mode11G_none_PSon_udp_tx_throughput 33.73+-0.01 /tmp/test_that_results_oBDjDo/results-16-network_WiFi_Perf.11g_tkip [ FAILED ] /tmp/test_that_results_oBDjDo/results-16-network_WiFi_Perf.11g_tkip ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-16-network_WiFi_Perf.11g_tkip/network_WiFi_Perf.11g_tkip [ FAILED ] /tmp/test_that_results_oBDjDo/results-16-network_WiFi_Perf.11g_tkip/network_WiFi_Perf.11g_tkip ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-16-network_WiFi_Perf.11g_tkip/network_WiFi_Perf.11g_tkip ch006_mode11G_psk_PSon_signal -37 /tmp/test_that_results_oBDjDo/results-16-network_WiFi_Perf.11g_tkip/network_WiFi_Perf.11g_tkip ch006_mode11G_psk_PSon_tcp_rx_throughput 23.34+-0.01 /tmp/test_that_results_oBDjDo/results-16-network_WiFi_Perf.11g_tkip/network_WiFi_Perf.11g_tkip ch006_mode11G_psk_PSon_tcp_tx_throughput 22.75+-0.19 /tmp/test_that_results_oBDjDo/results-16-network_WiFi_Perf.11g_tkip/network_WiFi_Perf.11g_tkip ch006_mode11G_psk_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-16-network_WiFi_Perf.11g_tkip/network_WiFi_Perf.11g_tkip ch006_mode11G_psk_PSon_udp_tx_throughput 33.30+-0.03 /tmp/test_that_results_oBDjDo/results-17-network_WiFi_Perf.11a_wep [ FAILED ] /tmp/test_that_results_oBDjDo/results-17-network_WiFi_Perf.11a_wep ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-17-network_WiFi_Perf.11a_wep/network_WiFi_Perf.11a_wep [ FAILED ] /tmp/test_that_results_oBDjDo/results-17-network_WiFi_Perf.11a_wep/network_WiFi_Perf.11a_wep ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status /tmp/test_that_results_oBDjDo/results-17-network_WiFi_Perf.11a_wep/network_WiFi_Perf.11a_wep ch044_mode11A_wep_PSon_signal -44 /tmp/test_that_results_oBDjDo/results-17-network_WiFi_Perf.11a_wep/network_WiFi_Perf.11a_wep ch044_mode11A_wep_PSon_tcp_rx_throughput 23.65+-0.02 /tmp/test_that_results_oBDjDo/results-17-network_WiFi_Perf.11a_wep/network_WiFi_Perf.11a_wep ch044_mode11A_wep_PSon_tcp_tx_throughput 22.29+-0.60 /tmp/test_that_results_oBDjDo/results-17-network_WiFi_Perf.11a_wep/network_WiFi_Perf.11a_wep ch044_mode11A_wep_PSon_udp_tx_errors 0.00+-0.00 /tmp/test_that_results_oBDjDo/results-17-network_WiFi_Perf.11a_wep/network_WiFi_Perf.11a_wep ch044_mode11A_wep_PSon_udp_tx_throughput 33.61+-0.01 ------------------------------------------------------------------------------------------------------------------------------ Total PASS: 0/34 (0%) 18:35:27 INFO | Finished running tests. Results can be found in /tmp/test_that_results_oBDjDo or /tmp/test_that_latest
,
Feb 7 2018
Running one of the netperf commands by hand appears to work fine (this would be over wired ethernet, not wireless). I'll look through the autotest logs to sort out what is failing and why sox is still missing.
,
Feb 7 2018
ERROR: Command </usr/bin/netperf -H 192.168.0.104 -p 12865 -t UDP_STREAM -l 10 -- -P 0,12866> failed, rc=255, Command returned non-zero exit status netperf is located at /usr/local/bin/netperf. Maybe the test needs to be fixed to use the newer location? Or just depend on $PATH to have it?
,
Feb 7 2018
/usr/bin vs. /usr/local/bin is by virtue of whether it's included in the base image or test image, respectively. For the testbed-ap profile, we depend on it directly, so it's in the base image. For DUTs, it's only part of -dev and -test images.
,
Feb 7 2018
Yes - I was testing "backwards" (kirtika, you are right!) netperf runs on the whirlwind netserver runs on the DUT (kefka) in my case. So netserver is located under /usr/local/sbin/. And I was able to reproduce the same issue kirtika was seeing but couldn't understand what netserver was trying to tell us...but something is broken with netserver if it's not installed in /usr/sbin/ AND debug output is enabled. Running "netserver -N" (disable debug output) not only shuts up the "file not found error", it also lets /usr/bin/netperf complete from the whirlwind. I'll check with the netperf author if he's heard this issue before.
,
Feb 7 2018
[pid 1742] open("/var/log/netperf//netserver.debug_1742", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 ENOENT (No such file or directory)
is enough to cause the netserver child process exit.
The following should probably be somehow mounted on /var/log/netperf :/
localhost ~ # find / -name netperf -type d
/mnt/stateful_partition/var_overlay/log/netperf
Any suggestions on how to do that?
,
Feb 7 2018
Mike, sorry to bother you - can you help with comment #25?
,
Feb 7 2018
this is largely wai. when you cros deploy, it puts everything in the destination root which is /usr/local. plus ebuilds cannot be relied upon to create any /var state ahead of time due to its volatility. you can use --root / when deploying, or change netperf to create the dirs it needs on the fly, or log via syslog, or use cli options to redirect somewhere else.
,
Feb 7 2018
This one's a strange one, because for some reason, gentoo decided to override upstream netperf, which tries to log to /tmp/. But because gentoo chose /var/log/<foo>/ now (as of 2.7), they're having permissions trouble, etc. We'd probably prefer to just go back to the upstream default, since this is a test utility, not a proper system daemon that we want persistent logging for. Any suggestions on doing that? We can just move to chromiumos-overlay and drop the gentoo patches. Or if we can find some way to undo this that gentoo would support...
,
Feb 7 2018
Oh, I see. People were worried about this? https://security-tracker.debian.org/tracker/CVE-2007-1444 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=413658 Seems like there are better ways to handle that? Are we never allowed to have tools write to /tmp, just because someone might run the tool as root and start symlink attacking themselves?
,
Feb 7 2018
Also, Mike, I don't think this problem is just about 'cros deploy'; it exists here on test images too, because netperf is pulled in (for regular ChromiumOS images) only on dev/test images, and therefore it goes into the stateful partition. I wonder if we should just get upstream to use mkstemp() or similar. They seemed to have poorly implemented the following, soon after that CVE: https://github.com/HewlettPackard/netperf/commit/69b623dbdbc6fb38d4a89f58a68f3a75db628fc8 But that looks wrong too...it just makes it slightly more obscure for someone to target the symlink attack, since they'd need to figure out your PID.
,
Feb 7 2018
I'll look into alternatives to /var/log/netperf with Rick Jones (netperf author). Getting this fixed upstream should be easy. FWIW, manually creating "ln -s /mnt/stateful_partition/var_overlay/log/netperf /var/log/netperf" allows network_wifiperf tests to complete successfully. Well, except for the missing sox package which is otherwise unrelated to the netperf change: /tmp/test_that_results_dtsLyX/results-01-network_WiFi_BluetoothStreamPerf.11a [ FAILED ] /tmp/test_that_results_dtsLyX/results-01-network_WiFi_BluetoothStreamPerf.11a ERROR: sox command is not installed. sudo emerge sox to install sox in chroot I've filed crbug.com/810062 for the "case of the missing sox". :)
,
Feb 7 2018
you could add a bashrc file that adds a -DDEBUG_LOG_FILE_DIR='"/tmp"' to CPPFLAGS
something like (untested):
$ cat chromiumos-overlay net-analyzer/netperf/netperf.bashrc
cros_post_src_configure_logdir() {
CPPFLAGS+=" -DDEBUG_LOG_FILE_DIR='\"/tmp\"'"
}
,
Feb 7 2018
but my point about ebuilds not relying on /var still stands. in upstream Gentoo we'll want to (at some point) delete all the code that precreates dirs under /var.
,
Feb 7 2018
@33: Sure. Then I think it would be a prime time to stop using /var/log/ entirely for this. AFAICT, it's only there as a quick and dirty hack to stop symlink attacks. (And it's a different, but more correct, version of upstream's hack.) @32: Wouldn't that be undoing the fix for the symlink attack, and instead relying on upstream's poorer attempt? Admittedly, we probably don't *really* care about symlink attacks on a debug/testing tool, but still... But otherwise, yes that does look like it could work.
,
Feb 7 2018
since netperf is only being used on dev/test images in a single user setup for testing purposes, the attack vectors of /tmp usage don't seem like a problem relative to getting your test frameworks passing again i'm just offering possible solutions/workarounds and setting expectations (from an ebuild/deploy pov). where you take it is up to you ;).
,
Feb 7 2018
I've uploaded a CL based on Mike's comment #32:
https://chromium-review.googlesource.com/907384
(I'll clean up the comment so it's easier to read and a bit more complete)
Tryjob for whirlwind-test-ap building now (#17) and I'll push a local kefka build to the lab test machine listed in the CL.
,
Feb 7 2018
you'll want to drop a symlink revbump into portage-stable to make sure the change is picked up everywhere
,
Feb 8 2018
Mike, I'm going to guess I should be using: ln -s netperf-2.7.0-r2.ebuild netperf-2.7.0-r3.ebuild and hope there isn't a conflict in the future with upstream gentoo. I've read: https://devmanual.gentoo.org/ebuild-writing/file-format/index.html#file-naming-rules and https://devmanual.gentoo.org/general-concepts/ebuild-revisions/index.html and ebuilds revisions don't appear have a "subversioning".
,
Feb 8 2018
i wouldn't worry about conflicts that much
,
Feb 9 2018
Here is another view of what's happening in the lab (derived from Stainless BigQuery table and result log scraping): https://ecgh.users.x20web.corp.google.com/www/cros_test/test_hosts.html?date=201801 I noticed that network_WiFi_UpdateRouter has been mostly failing to upgrade whirlwinds from 10007 to 10241 since Jan 12: https://stainless.corp.google.com/search?status=FAIL&status=ERROR&status=ABORT&test=%5Enetwork%5C_WiFi%5C_UpdateRouter%24&exclude_non_release=true&exclude_cts=true&view=list&first_date=2018-01-13&last_date=2018-02-09 I'm not sure what exactly is going wrong, but this reminded me of a discussion of network_WiFi_UpdateRouter a few months ago: https://groups.google.com/a/google.com/forum/#!msg/chromeos-infra-discuss/I2cYalvmV00/D_CGHRvSBAAJ Would it be a good idea to make the DUT provision/repair process include the router, to replace network_WiFi_UpdateRouter? I noticed recently in a provision log, it was checking the beaglebone_servo version. That seems similar to checking the router version.
,
Feb 10 2018
ecgh - awesome graphs - thank you! Since no good deed goes unpunished, can you take ownership of crbug.com/808288? I'll repost your comments there. If someone else could sort out why updater is failing, I should very soon have a whirlwind image that actually works. And I have made some progress on building a stumpy image based on "features/embedded" (ie no chrome). If I can get that to build/work it should be easy to repeat for panther/guado/whatever.
,
Feb 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/1ad3717e4f13e61be4d77ecd4ccfceab1e953dad commit 1ad3717e4f13e61be4d77ecd4ccfceab1e953dad Author: Grant Grundler <grundler@google.com> Date: Wed Feb 14 05:16:44 2018 netperf: workaround DEBUG_LOG_FILE_DIR brain damage Netserver child exits because /var/log/netperf doesn't exist. This only happens when adding netperf package to an already installed image. Netserver included in a build will result in /var/log/netperf existing and thus work fine. Brian Norris observed that Debian added a local patch to redefine DEBUG_LOG_FILE_DIR to user /var/log possibly due to: https://security-tracker.debian.org/tracker/CVE-2007-1444 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=413658 Then gentoo added another local patch to create /var/log/netperf and created netperf:netperf user/group as well. That works fine if netperf is included in the original Chromium OS "build_image". As noted above, netserver is broken when we install netperf package on an existing system (e.g. as dependency of autotest) due to missing /var/log/netperf directory. This workaround can be dropped once upstream netperf uses mkstemp(3) to create it's temp files. But for now, given we want WIFI tests to run and netperf is only used for dev and test images, just override gentoo patches and use original /tmp. BUG= chromium:808174 TEST=emerge-$BOARD --verbose netperf # verify "-DDEBUG_LOG..." is used. TEST=cros deploy chromeos15-row1-rack4-host5.cros net-analyzer/netperf test_that -b $BOARD chromeos15-row1-rack4-host5.cros suite:wifi_perf Change-Id: I0fb813aed887ac8d2154374f4f4d02be4b9ee1c9 Reviewed-on: https://chromium-review.googlesource.com/907220 Commit-Ready: Grant Grundler <grundler@chromium.org> Tested-by: Grant Grundler <grundler@chromium.org> Reviewed-by: Grant Grundler <grundler@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Brian Norris <briannorris@chromium.org> [add] https://crrev.com/1ad3717e4f13e61be4d77ecd4ccfceab1e953dad/chromeos/config/env/net-analyzer/netperf
,
Feb 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/portage-stable/+/5b79e8881fa1edebb8722391a0a3a2ba651b18e1 commit 5b79e8881fa1edebb8722391a0a3a2ba651b18e1 Author: Grant Grundler <grundler@google.com> Date: Wed Feb 14 12:07:03 2018 netperf: bump ebuild to pick up DEBUG_LOG fix CQ-DEPEND=CL:907220 BUG= chromium:808174 TEST=cros deploy chromeos15-row1-rack4-host5.cros net-analyzer/netperf test_that -b $BOARD chromeos15-row1-rack4-host5.cros suite:wifi_perf Change-Id: Ia6610525f282a47502e3cb040bccf4e7a553c66b Reviewed-on: https://chromium-review.googlesource.com/907384 Commit-Ready: Grant Grundler <grundler@chromium.org> Tested-by: Grant Grundler <grundler@chromium.org> Reviewed-by: Grant Grundler <grundler@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [add] https://crrev.com/5b79e8881fa1edebb8722391a0a3a2ba651b18e1/net-analyzer/netperf/netperf-2.7.0-r3.ebuild
,
Feb 14 2018
,
Feb 17 2018
I'm using this bug to also "bump" the UpdateRouter config and ran the following tests: test_that -b $BOARD $DUT_IP suite:wifi_perf test_that -b $BOARD $DUT_IP suite:wifi_matfunc All the BT tests in wifi_perf failed with "Chrome not started". Ignoring that. One matfunc test failed on kefka: /tmp/test_that_results_VCWtbX/results-30-network_WiFi_WakeOnWiFiThrottling [ FAILED ] all other tests passed. Is this ok?
,
Feb 17 2018
What does the log say for WakeOnWiFiThrottling? I saw that test fail when testing my whirlwind build for bug 770255 because it couldn't connect to the servo. I think it is only a problem when using test_that to run the suite from outside the lab.
,
Feb 17 2018
$ cd /tmp/test_that_results_VCWtbX/results-30-network_WiFi_WakeOnWiFiThrottling $ cat status.log INFO ---- ---- timestamp=1518806529 job_abort_reason=Check lid switch: lid_open is no localtime=Feb 16 12:42:09 Check lid switch: lid_open is no Hrm...seems like it will not pass on this device until someone opens the lid?
,
Feb 17 2018
Ah..looking at the autoserv.DEBUG log says exactly what ecgh mentioned (issue detecting servod or talking to servod. ... 02/16 12:42:05.695 INFO | repair:0349| Verifying this condition: servo SERIAL setting is correct 02/16 12:42:05.705 DEBUG| ssh_host:0301| Running (ssh) 'grep -q CHROMEOS /etc/lsb-release' from '_verify_host|verify|is_cros_host|run|run|run_very_slowly' 02/16 12:42:05.834 INFO | repair:0349| Verifying this condition: servod upstart job is running 02/16 12:42:05.845 DEBUG| ssh_host:0301| Running (ssh) 'grep -q CHROMEOS /etc/lsb-release' from '_verify_host|verify|is_cros_host|run|run|run_very_slowly' 02/16 12:42:05.983 DEBUG| ssh_host:0301| Running (ssh) 'status servod PORT=9999' from '_verify_list|_verify_host|verify|run|run|run_very_slowly' 02/16 12:42:06.120 DEBUG| utils:0282| [stdout] servod (9999) start/running, process 694 02/16 12:42:06.130 INFO | repair:0349| Verifying this condition: servod service is taking calls 02/16 12:42:06.131 DEBUG| abstract_ssh:0934| Full tunnel command: /usr/bin/ssh -a -x -n -N -q -L 35999:localhost:9999 -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=180 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22 chromeos15-row1-rack4-host5-servo.cros 02/16 12:42:06.170 DEBUG| abstract_ssh:0942| Started ssh tunnel, local = 35999 remote = 9999, pid = 229127 02/16 12:42:06.173 INFO |rpc_server_tracker:0196| Waiting 60 seconds for XMLRPC server to start. 02/16 12:42:06.175 WARNI| retry:0228| <class 'socket.error'>([Errno 111] Connection refused) 02/16 12:42:06.175 WARNI| retry:0183| Retrying in 0.728244 seconds... 02/16 12:42:06.931 INFO |rpc_server_tracker:0208| XMLRPC server started successfully. 02/16 12:42:07.069 INFO | servo:0529| Setting usb_mux_oe1 to on 02/16 12:42:09.235 DEBUG| servo:0225| Servo initialized, version is servo_v3 02/16 12:42:09.235 INFO | repair:0349| Verifying this condition: pwr_button control is normal 02/16 12:42:09.262 INFO | repair:0533| No failed triggers, skipping repair: Start servod with the proper config settings. 02/16 12:42:09.262 INFO | repair:0533| No failed triggers, skipping repair: Wait for update, then reboot servo host. 02/16 12:42:09.262 INFO | repair:0349| Verifying this condition: lid_open control is normal 02/16 12:42:09.290 ERROR| repair:0354| Failed: lid_open control is normal
,
Feb 17 2018
Even if the CL was marked CR+2, I don't intend on marking the CL CQ+1 until monday even though I'm satisfied this build is as good and likely better (since netperf will run) than existing builds. Anyone else is welcome to push this with CQ+1 though once someone marks CR+2.
,
Feb 21 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/78e419d05e8600c34ed624f662b3c30339920e7e commit 78e419d05e8600c34ed624f662b3c30339920e7e Author: Grant Grundler <grundler@chromium.org> Date: Wed Feb 21 04:32:08 2018 autotest: network_WiFi_UpdateRouter: update whirlwind Whirlwind APs in the test lab can start using R65-10323.33.0-c40061 build. netperf-2.7 is working again with this and leaving loads of "netserver.debug-XXXX" files in /tmp. BUG= chromium:808174 TEST=cros tryjob -b release-R65-10323.B \ -g 900169 -g 902706 -g 908769 -g 908770 --hwtest \ --committer grundler@google.com whirlwind-test-ap-tryjob cros flash $ROUTER_IP \ xbuddy://remote/trybot-whirlwind-test-ap-tryjob/R65-10323.33.0-c40061 test_that -b $BOARD $DUT_IP suite:wifi_perf test_that -b $BOARD $DUT_IP suite:wifi_matfunc Change-Id: I8fb484e2cc7622161c81aa700b73fa0a20a847c9 Reviewed-on: https://chromium-review.googlesource.com/922892 Commit-Ready: Grant Grundler <grundler@chromium.org> Tested-by: Grant Grundler <grundler@chromium.org> Tested-by: Brian Norris <briannorris@chromium.org> Reviewed-by: Grant Grundler <grundler@chromium.org> Reviewed-by: Brian Norris <briannorris@chromium.org> [modify] https://crrev.com/78e419d05e8600c34ed624f662b3c30339920e7e/server/site_tests/network_WiFi_UpdateRouter/network_WiFi_UpdateRouter.py
,
Mar 3 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/83f842af97ef166e5b9ef4e29182de2d2616f73b commit 83f842af97ef166e5b9ef4e29182de2d2616f73b Author: Grant Grundler <grundler@google.com> Date: Sat Mar 03 02:38:07 2018 netperf: workaround DEBUG_LOG_FILE_DIR brain damage Netserver child exits because /var/log/netperf doesn't exist. This only happens when adding netperf package to an already installed image. Netserver included in a build will result in /var/log/netperf existing and thus work fine. Brian Norris observed that Debian added a local patch to redefine DEBUG_LOG_FILE_DIR to user /var/log possibly due to: https://security-tracker.debian.org/tracker/CVE-2007-1444 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=413658 Then gentoo added another local patch to create /var/log/netperf and created netperf:netperf user/group as well. That works fine if netperf is included in the original Chromium OS "build_image". As noted above, netserver is broken when we install netperf package on an existing system (e.g. as dependency of autotest) due to missing /var/log/netperf directory. This workaround can be dropped once upstream netperf uses mkstemp(3) to create it's temp files. But for now, given we want WIFI tests to run and netperf is only used for dev and test images, just override gentoo patches and use original /tmp. BUG= chromium:808174 TEST=emerge-$BOARD --verbose netperf # verify "-DDEBUG_LOG..." is used. TEST=cros deploy chromeos15-row1-rack4-host5.cros net-analyzer/netperf test_that -b $BOARD chromeos15-row1-rack4-host5.cros suite:wifi_perf Change-Id: I0fb813aed887ac8d2154374f4f4d02be4b9ee1c9 Reviewed-on: https://chromium-review.googlesource.com/907220 Commit-Ready: Grant Grundler <grundler@chromium.org> Tested-by: Grant Grundler <grundler@chromium.org> Reviewed-by: Grant Grundler <grundler@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Brian Norris <briannorris@chromium.org> (cherry picked from commit 1ad3717e4f13e61be4d77ecd4ccfceab1e953dad) Reviewed-on: https://chromium-review.googlesource.com/908769 Commit-Queue: Grant Grundler <grundler@chromium.org> [add] https://crrev.com/83f842af97ef166e5b9ef4e29182de2d2616f73b/chromeos/config/env/net-analyzer/netperf
,
Mar 3 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/portage-stable/+/dfe96ee828a20489a23b96b970875d26d181996b commit dfe96ee828a20489a23b96b970875d26d181996b Author: Grant Grundler <grundler@google.com> Date: Sat Mar 03 02:38:12 2018 netperf: bump ebuild to pick up DEBUG_LOG fix CQ-DEPEND=CL:908769 BUG= chromium:808174 TEST=cros deploy chromeos15-row1-rack4-host5.cros net-analyzer/netperf test_that -b $BOARD chromeos15-row1-rack4-host5.cros suite:wifi_perf Change-Id: Ia6610525f282a47502e3cb040bccf4e7a553c66b Reviewed-on: https://chromium-review.googlesource.com/908770 Reviewed-by: Grant Grundler <grundler@chromium.org> Tested-by: Grant Grundler <grundler@chromium.org> Commit-Queue: Grant Grundler <grundler@chromium.org> [add] https://crrev.com/dfe96ee828a20489a23b96b970875d26d181996b/net-analyzer/netperf/netperf-2.7.0-r3.ebuild
,
Mar 22 2018
AFAIK all testbed-ap devices are running newer netperf/netserver.
,
Sep 13
Bulk verify old fixed bugs... |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by grundler@chromium.org
, Feb 1 2018