Incorrect test status uploaded to sponge |
|||
Issue descriptionSome server test results are not collected successfully in after_iteration_hook but we still upload test results to sponge with status of 'PASSED'. For example, https://storage.cloud.google.com/chromeos-autotest-results/224172736-chromeos-test/chromeos6-row3-rack7-host7/platform_BootPerfServer/debug/platform_BootPerfServer.DEBUG The test status is 'PASSED' on sponge - https://sponge.corp.google.com/invocations?searchFor=job_id%3A224172736 ==== 08/05 10:45:20.792 DEBUG| ssh_host:0301| Running (ssh) 'curl --connect-timeout 15 --retry 5 --retry-delay 5 -s http://100.115.219.135:8082/static/veyron_tiger-release/R68-10718.75.0/autotest/packages/client-autotest.tar.bz2 -o /tmp/sysinfo/autoserv-Mt2ejR/packages/client-autotest.tar.bz2' from 'install_pkg|fetch_pkg|fetch_pkg_file|_run_command|run|run_very_slowly' 08/05 11:05:21.473 WARNI| utils:0926| run process timeout (1200) fired on: /usr/bin/ssh -a -x -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22 chromeos6-row3-rack7-host7 "export LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack::fetch_pkg_file|_run_command|run] -> ssh_run(curl --connect-timeout 15 --retry 5 --retry-delay 5 -s http://100.115.219.135:8082/static/veyron_tiger-release/R68-10718.75.0/autotest/packages/client-autotest.tar.bz2 -o /tmp/sysinfo/autoserv-Mt2ejR/packages/client-autotest.tar.bz2)\";fi; curl --connect-timeout 15 --retry 5 --retry-delay 5 -s http://100.115.219.135:8082/static/veyron_tiger-release/R68-10718.75.0/autotest/packages/client-autotest.tar.bz2 -o /tmp/sysinfo/autoserv-Mt2ejR/packages/client-autotest.tar.bz2" 08/05 11:05:22.476 DEBUG| packages:0492| Timeout encountered: /usr/bin/ssh -a -x -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22 chromeos6-row3-rack7-host7 "export LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack::fetch_pkg_file|_run_command|run] -> ssh_run(curl --connect-timeout 15 --retry 5 --retry-delay 5 -s http://100.115.219.135:8082/static/veyron_tiger-release/R68-10718.75.0/autotest/packages/client-autotest.tar.bz2 -o /tmp/sysinfo/autoserv-Mt2ejR/packages/client-autotest.tar.bz2)\";fi; curl --connect-timeout 15 --retry 5 --retry-delay 5 -s http://100.115.219.135:8082/static/veyron_tiger-release/R68-10718.75.0/autotest/packages/client-autotest.tar.bz2 -o /tmp/sysinfo/autoserv-Mt2ejR/packages/client-autotest.tar.bz2" * Command: /usr/bin/ssh -a -x -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22 chromeos6-row3-rack7-host7 "export LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack::fetch_pkg_file|_run_command|run] -> ssh_run(curl --connect-timeout 15 --retry 5 --retry-delay 5 -s http://100.115.219.135:8082/static/veyron_tiger- release/R68-10718.75.0/autotest/packages/client-autotest.tar.bz2 -o /tmp/sysinfo/autoserv-Mt2ejR/packages/client-autotest.tar.bz2)\";fi; curl --connect-timeout 15 --retry 5 --retry-delay 5 -s http://100.115.219.135:8082/static/veyron_tiger- release/R68-10718.75.0/autotest/packages/client-autotest.tar.bz2 -o /tmp/sysinfo/autoserv-Mt2ejR/packages/client-autotest.tar.bz2" Exit status: 143 Duration: 1201.67043304 08/05 11:05:22.506 ERROR| packages:0497| client-autotest.tar.bz2 could not be fetched from any of the repos ['http://100.115.219.135:8082/static/veyron_tiger-release/R68-10718.75.0/autotest/packages'] 08/05 11:05:22.507 INFO | autotest:0358| Could not install autotest using the packaging system: Installation of autotest(type:client) failed : client-autotest.tar.bz2 could not be fetched from any of the repos ['http://100.115.219.135:8082/static/veyron_tiger-release/R68-10718.75.0/autotest/packages']. Trying other methods 08/05 11:05:22.507 DEBUG| abstract_ssh:0520| send_file. source: /usr/local/autotest/client/, dest: /tmp/sysinfo/autoserv-Mt2ejR, delete_dest: True,preserve_symlinks:False 08/05 11:05:22.507 DEBUG| abstract_ssh:0537| Using Rsync. 08/05 11:05:22.507 DEBUG| utils:0215| Running 'rsync -L --delete --timeout=1800 --rsh='/usr/bin/ssh -a -x -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22' -az --no-o --no-g "/usr/local/autotest/client/" "root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR""' 08/05 11:49:59.195 WARNI| abstract_ssh:0546| trying scp, rsync failed: Command <rsync -L --delete --timeout=1800 --rsh='/usr/bin/ssh -a -x -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22' -az --no-o --no-g "/usr/local/autotest/client/" "root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR""> failed, rc=10, Command returned non-zero exit status * Command: rsync -L --delete --timeout=1800 --rsh='/usr/bin/ssh -a -x -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22' -az --no-o --no-g "/usr/local/autotest/client/" "root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR"" Exit status: 10 Duration: 2676.67935586 stderr: Write failed: Broken pipe rsync: [sender] write error: Broken pipe (32) rsync error: error in socket IO (code 10) at io.c(837) [sender=3.1.0] 08/05 11:49:59.201 DEBUG| abstract_ssh:0549| Trying scp. 08/05 11:49:59.219 DEBUG| ssh_host:0301| Running (ssh) 'ls -d /tmp/sysinfo/autoserv-Mt2ejR/' from '_install|install|_install|send_file|run|run_very_slowly' 08/05 11:49:59.555 DEBUG| utils:0283| [stdout] /tmp/sysinfo/autoserv-Mt2ejR/ 08/05 11:49:59.603 DEBUG| ssh_host:0301| Running (ssh) 'rm -rf /tmp/sysinfo/autoserv-Mt2ejR && mkdir /tmp/sysinfo/autoserv-Mt2ejR' from '_install|install|_install|send_file|run|run_very_slowly' 08/05 11:50:02.016 DEBUG| utils:0215| Running 'scp -rq -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmp6mUYqT -P 22 ""/usr/local/autotest/client/"*" 'root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR"'' 08/05 11:50:03.206 DEBUG| abstract_ssh:0571| scp failed: Command <scp -rq -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmp6mUYqT -P 22 ""/usr/local/autotest/client/"*" 'root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR"'> failed, rc=1, Command returned non-zero exit status * Command: scp -rq -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmp6mUYqT -P 22 ""/usr/local/autotest/client/"*" 'root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR"' Exit status: 1 Duration: 1.18295502663 stderr: /usr/local/autotest/client/*: No such file or directory 08/05 11:50:03.207 ERROR| log:0027| post-test iteration server sysinfo error: 08/05 11:50:03.207 ERROR| traceback:0013| Traceback (most recent call last): 08/05 11:50:03.208 ERROR| traceback:0013| File "/usr/local/autotest/client/common_lib/log.py", line 25, in decorated_func 08/05 11:50:03.208 ERROR| traceback:0013| fn(*args, **dargs) 08/05 11:50:03.209 ERROR| traceback:0013| File "/usr/local/autotest/server/test.py", line 66, in wrapper 08/05 11:50:03.209 ERROR| traceback:0013| host, at, outputdir = self._install() 08/05 11:50:03.209 ERROR| traceback:0013| File "/usr/local/autotest/server/test.py", line 136, in _install 08/05 11:50:03.210 ERROR| traceback:0013| self.autotest.install(autodir=autodir) 08/05 11:50:03.239 ERROR| traceback:0013| File "/usr/local/autotest/server/autotest.py", line 259, in install 08/05 11:50:03.239 ERROR| traceback:0013| self._install(host=host, autodir=autodir, use_packaging=use_packaging) 08/05 11:50:03.240 ERROR| traceback:0013| File "/usr/local/autotest/server/autotest.py", line 375, in _install 08/05 11:50:03.240 ERROR| traceback:0013| host.send_file(self.source_material, autodir, delete_dest=True) 08/05 11:50:03.241 ERROR| traceback:0013| File "/usr/local/autotest/server/hosts/abstract_ssh.py", line 572, in send_file 08/05 11:50:03.241 ERROR| traceback:0013| raise error.AutoservRunError(e.args[0], e.args[1]) 08/05 11:50:03.242 ERROR| traceback:0013| AutoservRunError: scp -rq -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmp6mUYqT -P 22 ""/usr/local/autotest/client/"*" 'root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR"' 08/05 11:50:03.242 ERROR| traceback:0013| * Command: 08/05 11:50:03.242 ERROR| traceback:0013| scp -rq -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmp6mUYqT 08/05 11:50:03.242 ERROR| traceback:0013| -P 22 ""/usr/local/autotest/client/"*" 08/05 11:50:03.243 ERROR| traceback:0013| 'root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR"' 08/05 11:50:03.243 ERROR| traceback:0013| Exit status: 1 08/05 11:50:03.243 ERROR| traceback:0013| Duration: 1.18295502663 08/05 11:50:03.243 ERROR| traceback:0013| 08/05 11:50:03.243 ERROR| traceback:0013| stderr: 08/05 11:50:03.244 ERROR| traceback:0013| /usr/local/autotest/client/*: No such file or directory 08/05 11:50:03.245 DEBUG| test:0420| after_iteration_hooks completed 08/05 11:50:03.245 DEBUG| logging_manager:0627| Logging subprocess finished 08/05 11:50:03.250 DEBUG| logging_manager:0627| Logging subprocess finished
,
Sep 12
The tests are all passed though, so WAI? https://storage.cloud.google.com/chromeos-autotest-results/224172736-chromeos-test/chromeos6-row3-rack7-host7/status.log
,
Sep 12
Not sure if this is actionable. If the request is to have after_iteration_hook errors fail the test, that will need to be considered as a feature request. My instinct is that if it's actually something being tested, then it should be a part of the test proper. Maybe hook errors need to be exposed somewhere, but not as test failures.
,
Nov 2
Not currently a priority. |
|||
►
Sign in to add a comment |
|||
Comment 1 by ayatane@chromium.org
, Sep 12