New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 696837 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Feb 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug

Blocking:
issue 696633



Sign in to add a comment

test_push: Failing provision_AutoUpdate.double because autoupdate fails rsync silently

Project Member Reported by pprabhu@chromium.org, Feb 28 2017

Issue description

Failed test: http://chromeos-autotest.hot.corp.google.com/afe/#tab_id=view_job&object_id=5901

Symptom:
    File "/usr/local/autotest/client/common_lib/test.py", line 804, in _call_test_function
      return func(*args, **dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 461, in execute
      dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 347, in _call_run_once_with_retry
      postprocess_profiled_run, args, dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 376, in _call_run_once
      self.run_once(*args, **dargs)
    File "/usr/local/autotest/server/site_tests/provision_AutoUpdate/provision_AutoUpdate.py", line 147, in run_once
      force_full_update=force)
    File "/usr/local/autotest/server/afe_utils.py", line 237, in machine_install_and_update_labels
      *args, **dargs)
    File "/usr/local/autotest/server/hosts/cros_host.py", line 719, in machine_install_by_devserver
      full_update=force_full_update)
    File "/usr/local/autotest/client/common_lib/cros/dev_server.py", line 1813, in auto_update
      error_msg % (host_name, 'has reached AU retry limit.'))
  DevServerException: CrOS auto-update failed for host chromeos2-row9-rack1-host9: has reached AU retry limit.


Looking at the client logs for the test, we tried autoupdate via the old flow with devserver 100.115.245.198 twice, then bailed.

The devserver's (collected) autoupdate logs suddenly go silent when trying to copy stateful.tgz to the DUT.

Note that the DUT was successfully provisioned to gandof-release/R54-8743.76.0 before the test. The test tries to autoupdate the DUT twice from this provisioned version to the target version, and once again a trivial autoupdate to the same version. The test failed when attempting the first autoupdate to version R54-8743.76.0

Logs to successful provision: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos2-row9-rack1-host9/11758-provision
Logs to failed test: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/5901-chromeos-test/chromeos2-row9-rack1-host9/


 
Status: Started (was: Untriaged)
In test push, we use the stable version as the release version to test against. Unlike in prod, the stable version is not automatically updated.

gandof's stable version was very old. 

$ chromeos-test@chromeos-autotest:/usr/local/autotest$ ./cli/atest stable_version modify --board gandof --version R57-9202.18.0
Stable version for board gandof is changed from R54-8743.76.0 to R57-9202.18.0.


We need an automated way to update stable versions in test servers as well.

I've rekicked test push to see if this is fixed by a new version / get better failure reasons.
Cc: shuqianz@chromium.org
Labels: -Pri-2 Pri-0
Status: Fixed (was: Started)
test_push passed after bumping stable version.

Filed issue 697141 for follow up.

Comment 4 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 5 by dchan@google.com, May 30 2017

Labels: VerifyIn-60

Comment 6 by dchan@chromium.org, Aug 1 2017

Labels: VerifyIn-61

Comment 7 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)

Sign in to add a comment