chromeos15-row4-rack11-host2 failing wifi_matfun tests with error "Unhandled AutoservRunError: command execution error" |
|||||
Issue description
Sample failure:
08/24 13:55:34.510 DEBUG| ssh_host:0301| Running (ssh) 'python -c 'import cPickle, glob, sys;cPickle.dump(glob.glob(sys.argv[1]), sys.stdout, 0)'' from 'get_site_crashdumps|fetch_orphaned_crashdumps|_find_orphaned_crashdumps|list_files_glob|run|run_very_slowly'
08/24 13:55:34.909 INFO | site_crashcollect:0221| There are no orphaned crashes; deleting /usr/local/autotest/results/lxc_job_folder/crashinfo.chromeos15-row4-rack11-host2
08/24 13:55:34.913 INFO | site_crashcollect:0100| Trying to generate stack trace locally for /usr/local/autotest/results/lxc_job_folder/sysinfo/var/spool/crash/chrome.20180824.135346.29013.dmp
08/24 13:55:34.914 INFO | site_crashcollect:0034| symbol_dir: /usr/local/autotest/server/../../../lib/debug
08/24 13:55:34.914 DEBUG| utils:0219| Running 'minidump_stackwalk "/usr/local/autotest/results/lxc_job_folder/sysinfo/var/spool/crash/chrome.20180824.135346.29013.dmp" "/usr/local/autotest/server/../../../lib/debug" > "/usr/local/autotest/results/lxc_job_folder/sysinfo/var/spool/crash/chrome.20180824.135346.29013.dmp.txt"'
08/24 13:55:34.924 INFO | site_crashcollect:0107| Failed to generate stack trace locally for dump /usr/local/autotest/results/lxc_job_folder/sysinfo/var/spool/crash/chrome.20180824.135346.29013.dmp (rc=127):
CmdError('minidump_stackwalk "/usr/local/autotest/results/lxc_job_folder/sysinfo/var/spool/crash/chrome.20180824.135346.29013.dmp" "/usr/local/autotest/server/../../../lib/debug" > "/usr/local/autotest/results/lxc_job_folder/sysinfo/var/spool/crash/chrome.20180824.135346.29013.dmp.txt"', * Command:
minidump_stackwalk "/usr/local/autotest/results/lxc_job_folder/sysinfo/va
r/spool/crash/chrome.20180824.135346.29013.dmp"
"/usr/local/autotest/server/../../../lib/debug" > "/usr/local/autotest/res
ults/lxc_job_folder/sysinfo/var/spool/crash/chrome.20180824.135346.29013.d
mp.txt"
Exit status: 127
Duration: 0.00388789176941
https://stainless.corp.google.com/search?exclude_retried=true&first_date=2018-08-22&master_builder_name=&builder_name_number=&shard=&exclude_acts=true&builder_name=&master_builder_name_number=&owner=&retry=&exclude_cts=true&exclude_non_production=false&hostname=&board=&test=&suite=%5Ewifi%5C_release%24&build=%5ER69%5C-10895%5C.34%5C.0%24&status=FAIL&status=ERROR&status=ABORT&reason=&waterfall=&exclude_not_run=false&last_date=2018-08-24&exclude_non_release=true&exclude_au=true&model=&view=list&board_model=%5Enami%5C%2Fpantheon%24
This is a different issue than https://bugs.chromium.org/p/chromium/issues/detail?id=875927, which was failing with exit status 1
,
Aug 28
Bad pcap was causing this issue.Replacing the pcap fixed the issue. ------------------------------------------------------------------------------------------------------------------------------------------------------------------ /tmp/test_that_results_1b6D_g/results-01-network_WiFi_SimpleConnect.wifi_checkRawPMK [ PASSED ] /tmp/test_that_results_1b6D_g/results-01-network_WiFi_SimpleConnect.wifi_checkRawPMK/network_WiFi_SimpleConnect.wifi_checkRawPMK [ PASSED ] /tmp/test_that_results_1b6D_g/results-02-network_WiFi_SimpleConnect.wifi_checkMixedWPA [ PASSED ] /tmp/test_that_results_1b6D_g/results-02-network_WiFi_SimpleConnect.wifi_checkMixedWPA/network_WiFi_SimpleConnect.wifi_checkMixedWPA [ PASSED ] /tmp/test_that_results_1b6D_g/results-03-network_WiFi_SimpleConnect.wifi_check5VHT80_pure [ PASSED ] /tmp/test_that_results_1b6D_g/results-03-network_WiFi_SimpleConnect.wifi_check5VHT80_pure/network_WiFi_SimpleConnect.wifi_check5VHT80_pure [ PASSED ] /tmp/test_that_results_1b6D_g/results-04-network_WiFi_ChromeEndToEnd.transitionWiFiNetworks [ PASSED ] /tmp/test_that_results_1b6D_g/results-04-network_WiFi_ChromeEndToEnd.transitionWiFiNetworks/network_WiFi_ChromeEndToEnd [ PASSED ] /tmp/test_that_results_1b6D_g/results-04-network_WiFi_ChromeEndToEnd.transitionWiFiNetworks/network_WiFi_ChromeEndToEnd/network_ChromeWifiEndToEnd [ PASSED ] /tmp/test_that_results_1b6D_g/results-05-network_WiFi_SimpleConnect.wifi_checkWEP40 [ PASSED ] /tmp/test_that_results_1b6D_g/results-05-network_WiFi_SimpleConnect.wifi_checkWEP40/network_WiFi_SimpleConnect.wifi_checkWEP40 [ PASSED ] /tmp/test_that_results_1b6D_g/results-06-network_WiFi_SimpleConnect.wifi_check11b [ PASSED ] /tmp/test_that_results_1b6D_g/results-06-network_WiFi_SimpleConnect.wifi_check11b/network_WiFi_SimpleConnect.wifi_check11b [ PASSED ] /tmp/test_that_results_1b6D_g/results-07-network_WiFi_SimpleConnect.wifi_check11a [ PASSED ] /tmp/test_that_results_1b6D_g/results-07-network_WiFi_SimpleConnect.wifi_check11a/network_WiFi_SimpleConnect.wifi_check11a [ PASSED ] /tmp/test_that_results_1b6D_g/results-08-network_WiFi_ChromeEndToEnd.enableDisableWiFi [ PASSED ] /tmp/test_that_results_1b6D_g/results-08-network_WiFi_ChromeEndToEnd.enableDisableWiFi/network_WiFi_ChromeEndToEnd [ PASSED ] /tmp/test_that_results_1b6D_g/results-08-network_WiFi_ChromeEndToEnd.enableDisableWiFi/network_WiFi_ChromeEndToEnd/network_ChromeWifiEndToEnd [ PASSED ] /tmp/test_that_results_1b6D_g/results-09-network_WiFi_SimpleConnect.wifi_check24HT20 [ PASSED ] /tmp/test_that_results_1b6D_g/results-09-network_WiFi_SimpleConnect.wifi_check24HT20/network_WiFi_SimpleConnect.wifi_check24HT20 [ PASSED ] /tmp/test_that_results_1b6D_g/results-10-network_WiFi_ChromeEndToEnd.findVerifyWiFiNetworks [ PASSED ] /tmp/test_that_results_1b6D_g/results-10-network_WiFi_ChromeEndToEnd.findVerifyWiFiNetworks/network_WiFi_ChromeEndToEnd [ PASSED ] /tmp/test_that_results_1b6D_g/results-10-network_WiFi_ChromeEndToEnd.findVerifyWiFiNetworks/network_WiFi_ChromeEndToEnd/network_ChromeWifiEndToEnd [ PASSED ] /tmp/test_that_results_1b6D_g/results-11-network_WiFi_SimpleConnect.wifi_checkWPA2_TKIP [ PASSED ] /tmp/test_that_results_1b6D_g/results-11-network_WiFi_SimpleConnect.wifi_checkWPA2_TKIP/network_WiFi_SimpleConnect.wifi_checkWPA2_TKIP [ PASSED ] /tmp/test_that_results_1b6D_g/results-12-network_WiFi_SimpleConnect.wifi_check1x_PEAP [ PASSED ] /tmp/test_that_results_1b6D_g/results-12-network_WiFi_SimpleConnect.wifi_check1x_PEAP/network_WiFi_SimpleConnect.wifi_check1x_PEAP [ PASSED ] /tmp/test_that_results_1b6D_g/results-13-network_WiFi_SimpleConnect.wifi_checkWPA_TKIP [ PASSED ] /tmp/test_that_results_1b6D_g/results-13-network_WiFi_SimpleConnect.wifi_checkWPA_TKIP/network_WiFi_SimpleConnect.wifi_checkWPA_TKIP [ PASSED ] /tmp/test_that_results_1b6D_g/results-14-network_WiFi_ChromeEndToEnd.autoconnectWiFi [ PASSED ] /tmp/test_that_results_1b6D_g/results-14-network_WiFi_ChromeEndToEnd.autoconnectWiFi/network_WiFi_ChromeEndToEnd [ PASSED ] /tmp/test_that_results_1b6D_g/results-14-network_WiFi_ChromeEndToEnd.autoconnectWiFi/network_WiFi_ChromeEndToEnd/network_ChromeWifiEndToEnd [ PASSED ] /tmp/test_that_results_1b6D_g/results-15-network_WiFi_SimpleConnect.wifi_checkHidden [ PASSED ] /tmp/test_that_results_1b6D_g/results-15-network_WiFi_SimpleConnect.wifi_checkHidden/network_WiFi_SimpleConnect.wifi_checkHidden [ PASSED ] /tmp/test_that_results_1b6D_g/results-16-network_WiFi_SimpleConnect.wifi_checkWPA_CCMP [ PASSED ] /tmp/test_that_results_1b6D_g/results-16-network_WiFi_SimpleConnect.wifi_checkWPA_CCMP/network_WiFi_SimpleConnect.wifi_checkWPA_CCMP [ PASSED ] /tmp/test_that_results_1b6D_g/results-17-network_WiFi_SimpleConnect.wifi_checkOddWPAPassphrases [ PASSED ] /tmp/test_that_results_1b6D_g/results-17-network_WiFi_SimpleConnect.wifi_checkOddWPAPassphrases/network_WiFi_SimpleConnect.wifi_checkOddWPAPassphrases [ PASSED ] /tmp/test_that_results_1b6D_g/results-18-network_WiFi_SimpleConnect.wifi_checkHiddenWEP [ PASSED ] /tmp/test_that_results_1b6D_g/results-18-network_WiFi_SimpleConnect.wifi_checkHiddenWEP/network_WiFi_SimpleConnect.wifi_checkHiddenWEP [ PASSED ] /tmp/test_that_results_1b6D_g/results-19-network_WiFi_SimpleConnect.wifi_check1x_TTLS [ PASSED ] /tmp/test_that_results_1b6D_g/results-19-network_WiFi_SimpleConnect.wifi_check1x_TTLS/network_WiFi_SimpleConnect.wifi_check1x_TTLS [ PASSED ] /tmp/test_that_results_1b6D_g/results-20-network_WiFi_SimpleConnect.wifi_checkNonAsciiSSID [ PASSED ] /tmp/test_that_results_1b6D_g/results-20-network_WiFi_SimpleConnect.wifi_checkNonAsciiSSID/network_WiFi_SimpleConnect.wifi_checkNonAsciiSSID [ PASSED ] /tmp/test_that_results_1b6D_g/results-21-network_WiFi_SimpleConnect.wifi_check5HT40 [ PASSED ] /tmp/test_that_results_1b6D_g/results-21-network_WiFi_SimpleConnect.wifi_check5HT40/network_WiFi_SimpleConnect.wifi_check5HT40 [ PASSED ] /tmp/test_that_results_1b6D_g/results-22-network_WiFi_SimpleConnect.wifi_checkDFS [ PASSED ] /tmp/test_that_results_1b6D_g/results-22-network_WiFi_SimpleConnect.wifi_checkDFS/network_WiFi_SimpleConnect.wifi_checkDFS [ PASSED ] /tmp/test_that_results_1b6D_g/results-23-network_WiFi_SimpleConnect.wifi_check5HT20 [ PASSED ] /tmp/test_that_results_1b6D_g/results-23-network_WiFi_SimpleConnect.wifi_check5HT20/network_WiFi_SimpleConnect.wifi_check5HT20 [ PASSED ] /tmp/test_that_results_1b6D_g/results-24-network_WiFi_SimpleConnect.wifi_check1x_WEP [ PASSED ] /tmp/test_that_results_1b6D_g/results-24-network_WiFi_SimpleConnect.wifi_check1x_WEP/network_WiFi_SimpleConnect.wifi_check1x_WEP [ PASSED ] /tmp/test_that_results_1b6D_g/results-25-network_WiFi_SimpleConnect.wifi_check24HT40 [ PASSED ] /tmp/test_that_results_1b6D_g/results-25-network_WiFi_SimpleConnect.wifi_check24HT40/network_WiFi_SimpleConnect.wifi_check24HT40 [ PASSED ] /tmp/test_that_results_1b6D_g/results-26-network_WiFi_SimpleConnect.wifi_checkTruncatedBeacon [ PASSED ] /tmp/test_that_results_1b6D_g/results-26-network_WiFi_SimpleConnect.wifi_checkTruncatedBeacon/network_WiFi_SimpleConnect.wifi_checkTruncatedBeacon [ PASSED ] /tmp/test_that_results_1b6D_g/results-27-network_WiFi_SimpleConnect.wifi_check1x_WPA [ PASSED ] /tmp/test_that_results_1b6D_g/results-27-network_WiFi_SimpleConnect.wifi_check1x_WPA/network_WiFi_SimpleConnect.wifi_check1x_WPA [ PASSED ] /tmp/test_that_results_1b6D_g/results-28-network_WiFi_SimpleConnect.wifi_check11g [ PASSED ] /tmp/test_that_results_1b6D_g/results-28-network_WiFi_SimpleConnect.wifi_check11g/network_WiFi_SimpleConnect.wifi_check11g [ PASSED ] /tmp/test_that_results_1b6D_g/results-29-network_WiFi_SimpleConnect.wifi_checkWPA2 [ PASSED ] /tmp/test_that_results_1b6D_g/results-29-network_WiFi_SimpleConnect.wifi_checkWPA2/network_WiFi_SimpleConnect.wifi_checkWPA2 [ PASSED ] /tmp/test_that_results_1b6D_g/results-30-network_WiFi_SimpleConnect.wifi_checkWPA2_CCMP_PMF [ PASSED ] /tmp/test_that_results_1b6D_g/results-30-network_WiFi_SimpleConnect.wifi_checkWPA2_CCMP_PMF/network_WiFi_SimpleConnect.wifi_checkWPA2_CCMP_PMF [ PASSED ] /tmp/test_that_results_1b6D_g/results-31-network_WiFi_SimpleConnect.wifi_checkHiddenWPA [ PASSED ] /tmp/test_that_results_1b6D_g/results-31-network_WiFi_SimpleConnect.wifi_checkHiddenWPA/network_WiFi_SimpleConnect.wifi_checkHiddenWPA [ PASSED ] /tmp/test_that_results_1b6D_g/results-32-network_WiFi_SimpleConnect.wifi_checkSSIDLimits [ PASSED ] /tmp/test_that_results_1b6D_g/results-32-network_WiFi_SimpleConnect.wifi_checkSSIDLimits/network_WiFi_SimpleConnect.wifi_checkSSIDLimits [ PASSED ] /tmp/test_that_results_1b6D_g/results-33-network_WiFi_SimpleConnect.wifi_checkWPA_multi [ PASSED ] /tmp/test_that_results_1b6D_g/results-33-network_WiFi_SimpleConnect.wifi_checkWPA_multi/network_WiFi_SimpleConnect.wifi_checkWPA_multi [ PASSED ] /tmp/test_that_results_1b6D_g/results-34-network_WiFi_RoamEndToEnd [ PASSED ] /tmp/test_that_results_1b6D_g/results-34-network_WiFi_RoamEndToEnd/network_WiFi_RoamEndToEnd [ PASSED ] /tmp/test_that_results_1b6D_g/results-34-network_WiFi_RoamEndToEnd/network_WiFi_RoamEndToEnd/network_RoamWifiEndToEnd.connect [ PASSED ] /tmp/test_that_results_1b6D_g/results-34-network_WiFi_RoamEndToEnd/network_WiFi_RoamEndToEnd/network_RoamWifiEndToEnd.roam [ PASSED ] /tmp/test_that_results_1b6D_g/results-35-network_WiFi_SimpleConnect.wifi_checkWEP104 [ PASSED ] /tmp/test_that_results_1b6D_g/results-35-network_WiFi_SimpleConnect.wifi_checkWEP104/network_WiFi_SimpleConnect.wifi_checkWEP104 [ PASSED ] /tmp/test_that_results_1b6D_g/results-36-network_WiFi_SimpleConnect.wifi_check5VHT80_mixed [ PASSED ] /tmp/test_that_results_1b6D_g/results-36-network_WiFi_SimpleConnect.wifi_check5VHT80_mixed/network_WiFi_SimpleConnect.wifi_check5VHT80_mixed [ PASSED ] ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total PASS: 78/78 (100%)
,
Aug 30
The pcap image autoupdated and it started failing again,
AutoservRunError: command execution error
* Command:
/usr/bin/ssh -a -x -o Protocol=2 -o StrictHostKeyChecking=no -o
UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o
ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4
-l root -p 22 chromeos15-row4-rack11-host2-pcap "export
LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger
-tag \"autotest\" \"server[stack::get_wlanif|_get_wlanif|run] ->
ssh_run(/bin/ip link set dev monitor0 down)\";fi; /bin/ip link set dev
monitor0 down"
Exit status: 1
Duration: 1.51687693596
stderr:
Cannot find device "monitor0"
pcap -
CHROMEOS_RELEASE_CHROME_MILESTONE=66
CHROMEOS_RELEASE_PATCH_NUMBER=50
CHROMEOS_RELEASE_TRACK=testimage-channel
CHROMEOS_RELEASE_DESCRIPTION=10452.90.50 (Official Build) dev-channel whirlwind test
,
Aug 30
This is not the correct version:
# cat /etc/lsb-release
CHROMEOS_RELEASE_APPID={63E6908B-4769-4985-AD03-72E2C9E77D05}
CHROMEOS_BOARD_APPID={63E6908B-4769-4985-AD03-72E2C9E77D05}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
DEVICETYPE=OTHER
HWID_OVERRIDE=WHIRLWIND DOGFOOD
CHROMEOS_RELEASE_BUILDER_PATH=whirlwind-release/R66-10452.90.50
GOOGLE_RELEASE=10452.90.50
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_BOARD=whirlwind
CHROMEOS_RELEASE_BUILD_NUMBER=10452
CHROMEOS_RELEASE_BRANCH_NUMBER=90
CHROMEOS_RELEASE_CHROME_MILESTONE=66
CHROMEOS_RELEASE_PATCH_NUMBER=50
CHROMEOS_RELEASE_TRACK=testimage-channel
CHROMEOS_RELEASE_DESCRIPTION=10452.90.50 (Official Build) dev-channel whirlwind test
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_VERSION=10452.90.50
CHROMEOS_AUSERVER=https://tools.google.com/service/update2
I see that the A partition (mmcblk0p3) contains the correct test-ap image version, whereas we're currently active on the B partition, with a whirlwind-release (non-test-ap) version.
Maybe something screwed up enough that we rolled back?
Anyway, I switched the image back to the correct one. Probably should work now. Back to Aashutosh to verify. Device is locked, but feel free to unlock when you're done.
,
Dec 1
No recent tests on this device, but it looks like the pcap is in shape now. |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by harpreet@chromium.org
, Aug 27