rsync timeout claimed in AU test, but logs don't show the timeout |
||||||
Issue descriptionOak-release build 688 https://uberchromegw.corp.google.com/i/chromeos/builders/oak-release/builds/688 https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/90751024-chromeos-test/chromeos4-row11-rack8-host15/debug/ autoserv.DEBUG snippet: 12/15 02:21:03.550 DEBUG| ssh_host:0177| Running (ssh) 'true' 12/15 02:21:03.817 DEBUG| abstract_ssh:0577| Host chromeos4-row11-rack8-host15 is now up 12/15 02:21:03.818 DEBUG| abstract_ssh:0339| get_file. source: /tmp/sysinfo/autoserv-IrSzmY/results/default/, dest: /usr/local/autotest/results/90751024-chromeos-test/chromeos4-row11-rack8-host15, delete_dest: False,preserve_perm: True, preserve_symlinks:True 12/15 02:21:03.818 DEBUG| abstract_ssh:0350| Using Rsync. 12/15 02:21:03.818 DEBUG| base_utils:0185| Running 'rsync --safe-links --timeout=1800 --rsh='/usr/bin/ssh -a -x -o ControlPath=/tmp/_autotmp_D26fILssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22' -az --no-o --no-g root@chromeos4-row11-rack8-host15:"/tmp/sysinfo/autoserv-IrSzmY/results/default/" "/usr/local/autotest/results/90751024-chromeos-test/chromeos4-row11-rack8-host15"' 12/15 02:21:09.848 DEBUG| base_job:0246| Importing client.steps from state file /usr/local/autotest/results/90751024-chromeos-test/chromeos4-row11-rack8-host15/control.autoserv.state 12/15 02:21:09.849 DEBUG| base_job:0246| Importing client.harness from state file /usr/local/autotest/results/90751024-chromeos-test/chromeos4-row11-rack8-host15/control.autoserv.state 12/15 02:21:09.849 DEBUG| base_job:0243| Value of client.sysinfo is unchanged, skipping import 12/15 02:21:09.849 DEBUG| base_job:0246| Importing client._record_indent from state file /usr/local/autotest/results/90751024-chromeos-test/chromeos4-row11-rack8-host15/control.autoserv.state 12/15 02:21:09.849 DEBUG| base_job:0243| Value of public.client_success is unchanged, skipping import 12/15 02:21:09.849 DEBUG| base_job:0243| Value of global_properties.tag is unchanged, skipping import 12/15 02:21:09.849 DEBUG| base_job:0243| Value of global_properties.test_retry is unchanged, skipping import 12/15 02:21:09.850 DEBUG| monitors:0191| ts_mon: sending 20 metrics to PubSub 12/15 02:21:09.850 DEBUG| base_job:0392| Persistent state client.* deleted 12/15 02:21:09.851 DEBUG| abstract_ssh:0339| get_file. source: /tmp/autoserv-PJG2le/sysinfo.pickle, dest: /tmp/tmpMbc3zP, delete_dest: False,preserve_perm: True, preserve_symlinks:False 12/15 02:21:09.851 DEBUG| abstract_ssh:0350| Using Rsync. 12/15 02:21:09.851 DEBUG| base_utils:0185| Running 'rsync -L --timeout=1800 --rsh='/usr/bin/ssh -a -x -o ControlPath=/tmp/_autotmp_D26fILssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22' -az --no-o --no-g root@chromeos4-row11-rack8-host15:"/tmp/autoserv-PJG2le/sysinfo.pickle" "/tmp/tmpMbc3zP"' 12/15 02:21:10.676 WARNI| test:0606| Autotest caught exception when running test: Traceback (most recent call last): File "/usr/local/autotest/client/common_lib/test.py", line 600, in _exec _call_test_function(self.execute, *p_args, **p_dargs) File "/usr/local/autotest/client/common_lib/test.py", line 804, in _call_test_function return func(*args, **dargs) File "/usr/local/autotest/client/common_lib/test.py", line 461, in execute dargs) File "/usr/local/autotest/client/common_lib/test.py", line 347, in _call_run_once_with_retry postprocess_profiled_run, args, dargs) File "/usr/local/autotest/client/common_lib/test.py", line 376, in _call_run_once self.run_once(*args, **dargs) File "/usr/local/autotest/server/site_tests/autoupdate_EndToEndTest/autoupdate_EndToEndTest.py", line 1816, in run_once self.run_update_test(test_platform, test_conf) File "/usr/local/autotest/server/site_tests/autoupdate_EndToEndTest/autoupdate_EndToEndTest.py", line 1686, in run_update_test log_verifier.verify_expected_events_chain(chain) File "/usr/local/autotest/server/site_tests/autoupdate_EndToEndTest/autoupdate_EndToEndTest.py", line 334, in verify_expected_events_chain expected_event_chain.verify(self._get_next_log_event) File "/usr/local/autotest/server/site_tests/autoupdate_EndToEndTest/autoupdate_EndToEndTest.py", line 279, in verify raise ExpectedUpdateEventChainFailed(err_msg) ExpectedUpdateEventChainFailed: Failed to receive a download finished notification (download_finished) within 600 seconds. This could be a problem with the updater or a connectivity issue. For more details, check the update_engine log (in sysinfo or on the DUT, also included in the test log). The 10-minute timeout exception happens less than 1 second after the rsync starts. Either this is a lie, or the logs are unacceptably confusing.
,
Jan 4 2017
adlr for triage. Looks like network flake to me.
,
Jan 11 2017
,
Feb 1 2017
,
Feb 1 2017
This log is soooooooo confusing that takes me hours to find clues from it :( Adding more logging into it. Basically the above 'Rsync' has nothing to do with the traceback. The error happens in autoupdate_end_to_end test, and the above 'rsync' command is just a post-processing step for all tests.
,
Feb 21 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/02b6fee0016c10331831a1a7d0463d0c2c07f8b6 commit 02b6fee0016c10331831a1a7d0463d0c2c07f8b6 Author: xixuan <xixuan@chromium.org> Date: Tue Feb 21 22:07:49 2017 autotest: add logging in testing control files. This CL adds more loggings for running tests by control files. BUG= chromium:674598 TEST=Run autoupdate_End_to_End test in local AFE and verify loggings. Change-Id: I7c6696d2d46c8e2eff9dc744f9be53fa6045f5f1 Reviewed-on: https://chromium-review.googlesource.com/435964 Commit-Ready: Xixuan Wu <xixuan@chromium.org> Tested-by: Xixuan Wu <xixuan@chromium.org> Reviewed-by: Luigi Semenzato <semenzato@chromium.org> [modify] https://crrev.com/02b6fee0016c10331831a1a7d0463d0c2c07f8b6/server/test.py [modify] https://crrev.com/02b6fee0016c10331831a1a7d0463d0c2c07f8b6/server/autotest.py [modify] https://crrev.com/02b6fee0016c10331831a1a7d0463d0c2c07f8b6/client/common_lib/test.py
,
Jun 21 2017
,
Sep 19 2017
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by autumn@chromium.org
, Jan 4 2017