New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 674598 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Jun 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

rsync timeout claimed in AU test, but logs don't show the timeout

Project Member Reported by semenzato@chromium.org, Dec 15 2016

Issue description

Oak-release build 688

https://uberchromegw.corp.google.com/i/chromeos/builders/oak-release/builds/688
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/90751024-chromeos-test/chromeos4-row11-rack8-host15/debug/

autoserv.DEBUG snippet:

12/15 02:21:03.550 DEBUG|          ssh_host:0177| Running (ssh) 'true'
12/15 02:21:03.817 DEBUG|      abstract_ssh:0577| Host chromeos4-row11-rack8-host15 is now up
12/15 02:21:03.818 DEBUG|      abstract_ssh:0339| get_file. source: /tmp/sysinfo/autoserv-IrSzmY/results/default/, dest: /usr/local/autotest/results/90751024-chromeos-test/chromeos4-row11-rack8-host15, delete_dest: False,preserve_perm: True, preserve_symlinks:True
12/15 02:21:03.818 DEBUG|      abstract_ssh:0350| Using Rsync.
12/15 02:21:03.818 DEBUG|        base_utils:0185| Running 'rsync --safe-links  --timeout=1800 --rsh='/usr/bin/ssh -a -x   -o ControlPath=/tmp/_autotmp_D26fILssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22' -az --no-o --no-g root@chromeos4-row11-rack8-host15:"/tmp/sysinfo/autoserv-IrSzmY/results/default/" "/usr/local/autotest/results/90751024-chromeos-test/chromeos4-row11-rack8-host15"'
12/15 02:21:09.848 DEBUG|          base_job:0246| Importing client.steps from state file /usr/local/autotest/results/90751024-chromeos-test/chromeos4-row11-rack8-host15/control.autoserv.state
12/15 02:21:09.849 DEBUG|          base_job:0246| Importing client.harness from state file /usr/local/autotest/results/90751024-chromeos-test/chromeos4-row11-rack8-host15/control.autoserv.state
12/15 02:21:09.849 DEBUG|          base_job:0243| Value of client.sysinfo is unchanged, skipping import
12/15 02:21:09.849 DEBUG|          base_job:0246| Importing client._record_indent from state file /usr/local/autotest/results/90751024-chromeos-test/chromeos4-row11-rack8-host15/control.autoserv.state
12/15 02:21:09.849 DEBUG|          base_job:0243| Value of public.client_success is unchanged, skipping import
12/15 02:21:09.849 DEBUG|          base_job:0243| Value of global_properties.tag is unchanged, skipping import
12/15 02:21:09.849 DEBUG|          base_job:0243| Value of global_properties.test_retry is unchanged, skipping import
12/15 02:21:09.850 DEBUG|          monitors:0191| ts_mon: sending 20 metrics to PubSub
12/15 02:21:09.850 DEBUG|          base_job:0392| Persistent state client.* deleted
12/15 02:21:09.851 DEBUG|      abstract_ssh:0339| get_file. source: /tmp/autoserv-PJG2le/sysinfo.pickle, dest: /tmp/tmpMbc3zP, delete_dest: False,preserve_perm: True, preserve_symlinks:False
12/15 02:21:09.851 DEBUG|      abstract_ssh:0350| Using Rsync.
12/15 02:21:09.851 DEBUG|        base_utils:0185| Running 'rsync -L  --timeout=1800 --rsh='/usr/bin/ssh -a -x   -o ControlPath=/tmp/_autotmp_D26fILssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22' -az --no-o --no-g root@chromeos4-row11-rack8-host15:"/tmp/autoserv-PJG2le/sysinfo.pickle" "/tmp/tmpMbc3zP"'
12/15 02:21:10.676 WARNI|              test:0606| Autotest caught exception when running test:
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/test.py", line 600, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 804, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 461, in execute
    dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 347, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 376, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/server/site_tests/autoupdate_EndToEndTest/autoupdate_EndToEndTest.py", line 1816, in run_once
    self.run_update_test(test_platform, test_conf)
  File "/usr/local/autotest/server/site_tests/autoupdate_EndToEndTest/autoupdate_EndToEndTest.py", line 1686, in run_update_test
    log_verifier.verify_expected_events_chain(chain)
  File "/usr/local/autotest/server/site_tests/autoupdate_EndToEndTest/autoupdate_EndToEndTest.py", line 334, in verify_expected_events_chain
    expected_event_chain.verify(self._get_next_log_event)
  File "/usr/local/autotest/server/site_tests/autoupdate_EndToEndTest/autoupdate_EndToEndTest.py", line 279, in verify
    raise ExpectedUpdateEventChainFailed(err_msg)
ExpectedUpdateEventChainFailed: Failed to receive a download finished notification (download_finished) within 600 seconds. This could be a problem with the updater or a connectivity issue. For more details, check the update_engine log (in sysinfo or on the DUT, also included in the test log).

The 10-minute timeout exception happens less than 1 second after the rsync starts.  Either this is a lie, or the logs are unacceptably confusing.
  

 
Owner: de...@chromium.org
deymo - who's the best owner for autoupdate issues?

Comment 2 by de...@chromium.org, Jan 4 2017

Owner: adlr@chromium.org
adlr for triage. 

Looks like network flake to me.
Labels: Hotlist-Fixit
Owner: ----
Owner: xixuan@chromium.org
Status: Started (was: Untriaged)
This log is soooooooo confusing that takes me hours to find clues from it :(  Adding more logging into it.

Basically the above 'Rsync' has nothing to do with the traceback. The error happens in autoupdate_end_to_end test, and the above 'rsync' command is just a post-processing step for all tests.
Project Member

Comment 6 by bugdroid1@chromium.org, Feb 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/02b6fee0016c10331831a1a7d0463d0c2c07f8b6

commit 02b6fee0016c10331831a1a7d0463d0c2c07f8b6
Author: xixuan <xixuan@chromium.org>
Date: Tue Feb 21 22:07:49 2017

autotest: add logging in testing control files.

This CL adds more loggings for running tests by control files.

BUG= chromium:674598 
TEST=Run autoupdate_End_to_End test in local AFE and verify loggings.

Change-Id: I7c6696d2d46c8e2eff9dc744f9be53fa6045f5f1
Reviewed-on: https://chromium-review.googlesource.com/435964
Commit-Ready: Xixuan Wu <xixuan@chromium.org>
Tested-by: Xixuan Wu <xixuan@chromium.org>
Reviewed-by: Luigi Semenzato <semenzato@chromium.org>

[modify] https://crrev.com/02b6fee0016c10331831a1a7d0463d0c2c07f8b6/server/test.py
[modify] https://crrev.com/02b6fee0016c10331831a1a7d0463d0c2c07f8b6/server/autotest.py
[modify] https://crrev.com/02b6fee0016c10331831a1a7d0463d0c2c07f8b6/client/common_lib/test.py

Comment 7 by xixuan@chromium.org, Jun 21 2017

Status: Fixed (was: Started)
Status: Verified (was: Fixed)

Sign in to add a comment