New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 688410 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

no retries for autoupdate_Rollback?

Project Member Reported by semenzato@chromium.org, Feb 3 2017

Issue description

https://uberchromegw.corp.google.com/i/chromeos/builders/daisy-release/builds/820

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/99490555-chromeos-test/chromeos2-row4-rack7-host8/debug/

autoserv.DEBUG:

02/03 05:52:11.381 INFO |autoupdate_Rollbac:0030| Initial device state: active kernel {'kernel': 2, 'root': 3, 'name': 'KERN-A'}, inactive kernel {'kernel': 4, 'root': 5, 'name': 'KERN-B'}.
02/03 05:52:11.382 INFO |autoupdate_Rollbac:0032| Performing an update.
02/03 05:52:11.383 INFO |       autoupdater:0268| Updating image via: /usr/bin/update_engine_client --update --omaha_url=http://100.115.245.197:8082/update/daisy-release/R58-9247.0.0
02/03 05:52:11.384 DEBUG|          ssh_host:0272| Running (ssh) '/usr/bin/update_engine_client --update --omaha_url=http://100.115.245.197:8082/update/daisy-release/R58-9247.0.0'
02/03 05:52:12.120 ERROR|        base_utils:0280| [stderr] [0203/055211:INFO:update_engine_client.cc(471)] Forcing an update by setting app_version to ForcedUpdate.
02/03 05:52:12.122 ERROR|        base_utils:0280| [stderr] [0203/055211:INFO:update_engine_client.cc(473)] Initiating update check and install.
02/03 05:52:12.123 ERROR|        base_utils:0280| [stderr] [0203/055211:INFO:update_engine_client.cc(502)] Waiting for update to complete.
02/03 05:53:05.004 ERROR|           metrics:0429| Caught exception while flushing: No module named pyasn1.codec.ber
02/03 06:21:18.871 ERROR|        base_utils:0280| [stderr] Write failed: Broken pipe
02/03 06:21:18.876 DEBUG|       autoupdater:0176| exception in update handler: command execution error
* Command: 
    /usr/bin/ssh -a -x     -o StrictHostKeyChecking=no -o
    UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o
    ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4
    -o Protocol=2 -l root -p 22 chromeos2-row4-rack7-host8 "export
    LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger
    -tag \"autotest\" \"server[stack::_base_update_handler|GenericRetry|_base_
    update_handler_no_retry] -> ssh_run(/usr/bin/update_engine_client --update
    --omaha_url=http://100.115.245.197:8082/update/daisy-
    release/R58-9247.0.0)\";fi; /usr/bin/update_engine_client --update
    --omaha_url=http://100.115.245.197:8082/update/daisy-release/R58-9247.0.0"
Exit status: 255
Duration: 1747.41930985

stderr:
[0203/055211:INFO:update_engine_client.cc(471)] Forcing an update by setting app_version to ForcedUpdate.
[0203/055211:INFO:update_engine_client.cc(473)] Initiating update check and install.
[0203/055211:INFO:update_engine_client.cc(502)] Waiting for update to complete.
Write failed: Broken pipe
02/03 06:21:18.880 DEBUG|        retry_util:0129| ending retries with error: <class 'autotest_lib.client.common_lib.error.AutoservRunError'>(command execution error

I am confused by two things.

1. I thought we were running these tests with the AU server running on the DUT, but the update_engine_client command points to a devserver.

2. update_engine_client exits with "broken pipe" after almost 30m, and is not retried.  Is this intended?  Would it make sense to retry?
 
As to why this particular update failed, I don't know, but it wasn't devserver load.  Devserver2-chromeos5 was somewhat loaded, but never near disk or CPU saturation.  The network is a little strange: two out of four interfaces aren't used.  I'll file a separate bug for that (just search for the devserver name).
I meant chromeos2-devserver5, sorry.

c#0, thing #1: the AU server running on the DUT is only for provisioning tests.  The AU tests still have the AU server running remotely.

Comment 4 by aut...@google.com, Feb 9 2017

Labels: -current-issue
Owner: semenzato@chromium.org
@semenzato - can you clarify what you need? If you'd like to modify the test to add a re-try, that seems fine. 
Status: Archived (was: Untriaged)

Sign in to add a comment