New issue
Advanced search Search tips

Issue 871155 link

Starred by 1 user

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Feature



Sign in to add a comment

Incorrect test status uploaded to sponge

Project Member Reported by cywang@chromium.org, Aug 6

Issue description

Some server test results are not collected successfully in after_iteration_hook but we still upload test results to sponge with status of 'PASSED'.

For example,

https://storage.cloud.google.com/chromeos-autotest-results/224172736-chromeos-test/chromeos6-row3-rack7-host7/platform_BootPerfServer/debug/platform_BootPerfServer.DEBUG

The test status is 'PASSED' on sponge - 
https://sponge.corp.google.com/invocations?searchFor=job_id%3A224172736

====
08/05 10:45:20.792 DEBUG|          ssh_host:0301| Running (ssh) 'curl --connect-timeout 15 --retry 5 --retry-delay 5 -s http://100.115.219.135:8082/static/veyron_tiger-release/R68-10718.75.0/autotest/packages/client-autotest.tar.bz2 -o /tmp/sysinfo/autoserv-Mt2ejR/packages/client-autotest.tar.bz2' from 'install_pkg|fetch_pkg|fetch_pkg_file|_run_command|run|run_very_slowly'
08/05 11:05:21.473 WARNI|             utils:0926| run process timeout (1200) fired on: /usr/bin/ssh -a -x   -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22 chromeos6-row3-rack7-host7 "export LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack::fetch_pkg_file|_run_command|run] -> ssh_run(curl --connect-timeout 15 --retry 5 --retry-delay 5 -s http://100.115.219.135:8082/static/veyron_tiger-release/R68-10718.75.0/autotest/packages/client-autotest.tar.bz2 -o /tmp/sysinfo/autoserv-Mt2ejR/packages/client-autotest.tar.bz2)\";fi; curl --connect-timeout 15 --retry 5 --retry-delay 5 -s http://100.115.219.135:8082/static/veyron_tiger-release/R68-10718.75.0/autotest/packages/client-autotest.tar.bz2 -o /tmp/sysinfo/autoserv-Mt2ejR/packages/client-autotest.tar.bz2"
08/05 11:05:22.476 DEBUG|          packages:0492| Timeout encountered: /usr/bin/ssh -a -x   -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22 chromeos6-row3-rack7-host7 "export LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack::fetch_pkg_file|_run_command|run] -> ssh_run(curl --connect-timeout 15 --retry 5 --retry-delay 5 -s http://100.115.219.135:8082/static/veyron_tiger-release/R68-10718.75.0/autotest/packages/client-autotest.tar.bz2 -o /tmp/sysinfo/autoserv-Mt2ejR/packages/client-autotest.tar.bz2)\";fi; curl --connect-timeout 15 --retry 5 --retry-delay 5 -s http://100.115.219.135:8082/static/veyron_tiger-release/R68-10718.75.0/autotest/packages/client-autotest.tar.bz2 -o /tmp/sysinfo/autoserv-Mt2ejR/packages/client-autotest.tar.bz2"
* Command: 
    /usr/bin/ssh -a -x   -o Protocol=2 -o StrictHostKeyChecking=no -o
    UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o
    ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4
    -l root -p 22 chromeos6-row3-rack7-host7 "export LIBC_FATAL_STDERR_=1; if
    type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\"
    \"server[stack::fetch_pkg_file|_run_command|run] -> ssh_run(curl
    --connect-timeout 15 --retry 5 --retry-delay 5 -s
    http://100.115.219.135:8082/static/veyron_tiger-
    release/R68-10718.75.0/autotest/packages/client-autotest.tar.bz2 -o
    /tmp/sysinfo/autoserv-Mt2ejR/packages/client-autotest.tar.bz2)\";fi; curl
    --connect-timeout 15 --retry 5 --retry-delay 5 -s
    http://100.115.219.135:8082/static/veyron_tiger-
    release/R68-10718.75.0/autotest/packages/client-autotest.tar.bz2 -o
    /tmp/sysinfo/autoserv-Mt2ejR/packages/client-autotest.tar.bz2"
Exit status: 143
Duration: 1201.67043304

08/05 11:05:22.506 ERROR|          packages:0497| client-autotest.tar.bz2 could not be fetched from any of the repos ['http://100.115.219.135:8082/static/veyron_tiger-release/R68-10718.75.0/autotest/packages']
08/05 11:05:22.507 INFO |          autotest:0358| Could not install autotest using the packaging system: Installation of autotest(type:client) failed : client-autotest.tar.bz2 could not be fetched from any of the repos ['http://100.115.219.135:8082/static/veyron_tiger-release/R68-10718.75.0/autotest/packages']. Trying other methods
08/05 11:05:22.507 DEBUG|      abstract_ssh:0520| send_file. source: /usr/local/autotest/client/, dest: /tmp/sysinfo/autoserv-Mt2ejR, delete_dest: True,preserve_symlinks:False
08/05 11:05:22.507 DEBUG|      abstract_ssh:0537| Using Rsync.
08/05 11:05:22.507 DEBUG|             utils:0215| Running 'rsync -L --delete --timeout=1800 --rsh='/usr/bin/ssh -a -x  -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22' -az --no-o --no-g  "/usr/local/autotest/client/" "root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR""'
08/05 11:49:59.195 WARNI|      abstract_ssh:0546| trying scp, rsync failed: Command <rsync -L --delete --timeout=1800 --rsh='/usr/bin/ssh -a -x  -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22' -az --no-o --no-g  "/usr/local/autotest/client/" "root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR""> failed, rc=10, Command returned non-zero exit status
* Command: 
    rsync -L --delete --timeout=1800 --rsh='/usr/bin/ssh -a -x  -o Protocol=2
    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
    BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
    ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22' -az --no-o
    --no-g  "/usr/local/autotest/client/"
    "root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR""
Exit status: 10
Duration: 2676.67935586

stderr:
Write failed: Broken pipe
rsync: [sender] write error: Broken pipe (32)
rsync error: error in socket IO (code 10) at io.c(837) [sender=3.1.0]
08/05 11:49:59.201 DEBUG|      abstract_ssh:0549| Trying scp.
08/05 11:49:59.219 DEBUG|          ssh_host:0301| Running (ssh) 'ls -d /tmp/sysinfo/autoserv-Mt2ejR/' from '_install|install|_install|send_file|run|run_very_slowly'
08/05 11:49:59.555 DEBUG|             utils:0283| [stdout] /tmp/sysinfo/autoserv-Mt2ejR/
08/05 11:49:59.603 DEBUG|          ssh_host:0301| Running (ssh) 'rm -rf /tmp/sysinfo/autoserv-Mt2ejR && mkdir /tmp/sysinfo/autoserv-Mt2ejR' from '_install|install|_install|send_file|run|run_very_slowly'
08/05 11:50:02.016 DEBUG|             utils:0215| Running 'scp -rq  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmp6mUYqT -P 22 ""/usr/local/autotest/client/"*" 'root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR"''
08/05 11:50:03.206 DEBUG|      abstract_ssh:0571| scp failed: Command <scp -rq  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmp6mUYqT -P 22 ""/usr/local/autotest/client/"*" 'root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR"'> failed, rc=1, Command returned non-zero exit status
* Command: 
    scp -rq  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmp6mUYqT
    -P 22 ""/usr/local/autotest/client/"*"
    'root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR"'
Exit status: 1
Duration: 1.18295502663

stderr:
/usr/local/autotest/client/*: No such file or directory
08/05 11:50:03.207 ERROR|               log:0027| post-test iteration server sysinfo error:
08/05 11:50:03.207 ERROR|         traceback:0013| Traceback (most recent call last):
08/05 11:50:03.208 ERROR|         traceback:0013|   File "/usr/local/autotest/client/common_lib/log.py", line 25, in decorated_func
08/05 11:50:03.208 ERROR|         traceback:0013|     fn(*args, **dargs)
08/05 11:50:03.209 ERROR|         traceback:0013|   File "/usr/local/autotest/server/test.py", line 66, in wrapper
08/05 11:50:03.209 ERROR|         traceback:0013|     host, at, outputdir = self._install()
08/05 11:50:03.209 ERROR|         traceback:0013|   File "/usr/local/autotest/server/test.py", line 136, in _install
08/05 11:50:03.210 ERROR|         traceback:0013|     self.autotest.install(autodir=autodir)
08/05 11:50:03.239 ERROR|         traceback:0013|   File "/usr/local/autotest/server/autotest.py", line 259, in install
08/05 11:50:03.239 ERROR|         traceback:0013|     self._install(host=host, autodir=autodir, use_packaging=use_packaging)
08/05 11:50:03.240 ERROR|         traceback:0013|   File "/usr/local/autotest/server/autotest.py", line 375, in _install
08/05 11:50:03.240 ERROR|         traceback:0013|     host.send_file(self.source_material, autodir, delete_dest=True)
08/05 11:50:03.241 ERROR|         traceback:0013|   File "/usr/local/autotest/server/hosts/abstract_ssh.py", line 572, in send_file
08/05 11:50:03.241 ERROR|         traceback:0013|     raise error.AutoservRunError(e.args[0], e.args[1])
08/05 11:50:03.242 ERROR|         traceback:0013| AutoservRunError: scp -rq  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmp6mUYqT -P 22 ""/usr/local/autotest/client/"*" 'root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR"'
08/05 11:50:03.242 ERROR|         traceback:0013| * Command: 
08/05 11:50:03.242 ERROR|         traceback:0013|     scp -rq  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/tmp6mUYqT
08/05 11:50:03.242 ERROR|         traceback:0013|     -P 22 ""/usr/local/autotest/client/"*"
08/05 11:50:03.243 ERROR|         traceback:0013|     'root@chromeos6-row3-rack7-host7:"/tmp/sysinfo/autoserv-Mt2ejR"'
08/05 11:50:03.243 ERROR|         traceback:0013| Exit status: 1
08/05 11:50:03.243 ERROR|         traceback:0013| Duration: 1.18295502663
08/05 11:50:03.243 ERROR|         traceback:0013| 
08/05 11:50:03.243 ERROR|         traceback:0013| stderr:
08/05 11:50:03.244 ERROR|         traceback:0013| /usr/local/autotest/client/*: No such file or directory
08/05 11:50:03.245 DEBUG|              test:0420| after_iteration_hooks completed
08/05 11:50:03.245 DEBUG|   logging_manager:0627| Logging subprocess finished
08/05 11:50:03.250 DEBUG|   logging_manager:0627| Logging subprocess finished
 
Labels: -Type-Bug Type-Feature
Status: Unconfirmed (was: Untriaged)
Not sure if this is actionable.  If the request is to have after_iteration_hook errors fail the test, that will need to be considered as a feature request.  My instinct is that if it's actually something being tested, then it should be a part of the test proper.  Maybe hook errors need to be exposed somewhere, but not as test failures.
Owner: ----
Status: Available (was: Unconfirmed)
Not currently a priority.

Sign in to add a comment