New issue
Advanced search Search tips

Issue 754507 link

Starred by 1 user
Status: Duplicate
Merged: issue 754314
Owner: ----
Closed: Aug 2017
Cc:
xixuan@chromium.org
dgarr...@chromium.org
kathrelk...@chromium.org
dhadd...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Unhandled DevServerException: CrOS auto-update failed for host <host>: SSHConnectionError: Connection to <ip> timed out while waiting to read

Project Member Reported by dhadd...@chromium.org, Aug 10 2017 
I am seeing this on the paygen_au_* suites occasionally

The test stages the files on the devserver correctly. 

It then calls cros_au via dev_server.auto_update()



08/10 15:45:28.521 WARNI|        dev_server:2022| Unable to parse build name canary-channel/snappy/9828.0.0 for metrics. Continuing anyway.
08/10 15:45:28.521 DEBUG|        dev_server:2084| Start CrOS auto-update for host chromeos2-row4-rack2-host2 at 1 time(s).
08/10 15:45:28.522 DEBUG|             utils:0212| Running 'ssh 100.115.245.197 'curl "http://100.115.245.197:8082/cros_au?full_update=True&payload_filename=payloads/chromeos_9828.0.0_snappy_canary-channel_full_test.bin-6409561c70e6dd0f7868f4bc4eadeca3&force_update=True&build_name=canary-channel/snappy/9828.0.0&host_name=chromeos2-row4-rack2-host2&async=True&clobber_stateful=True"''
08/10 15:45:34.456 INFO |        dev_server:1816| Received response from devserver for cros_au call: '[true, 1089]'
08/10 15:45:34.458 DEBUG|        dev_server:1935| start process 1089 for auto_update in devserver
08/10 15:45:34.458 DEBUG|        dev_server:1837| Check the progress for auto-update process 1089
08/10 15:45:34.459 DEBUG|             utils:0212| Running 'ssh 100.115.245.197 'curl "http://100.115.245.197:8082/get_au_status?full_update=True&payload_filename=payloads/chromeos_9828.0.0_snappy_canary-channel_full_test.bin-6409561c70e6dd0f7868f4bc4eadeca3&force_update=True&pid=1089&build_name=canary-channel/snappy/9828.0.0&host_name=chromeos2-row4-rack2-host2&clobber_stateful=True"''
08/10 15:45:35.429 DEBUG|        dev_server:1873| Current CrOS auto-update status: CrOS update is just started.
08/10 15:45:45.472 DEBUG|             utils:0212| Running 'ssh 100.115.245.197 'curl "http://100.115.245.197:8082/get_au_status?full_update=True&payload_filename=payloads/chromeos_9828.0.0_snappy_canary-channel_full_test.bin-6409561c70e6dd0f7868f4bc4eadeca3&force_update=True&pid=1089&build_name=canary-channel/snappy/9828.0.0&host_name=chromeos2-row4-rack2-host2&clobber_stateful=True"''
08/10 15:45:46.419 DEBUG|        dev_server:1873| Current CrOS auto-update status: CrOS update is just started.
08/10 15:45:56.471 DEBUG|             utils:0212| Running 'ssh 100.115.245.197 'curl "http://100.115.245.197:8082/get_au_status?full_update=True&payload_filename=payloads/chromeos_9828.0.0_snappy_canary-channel_full_test.bin-6409561c70e6dd0f7868f4bc4eadeca3&force_update=True&pid=1089&build_name=canary-channel/snappy/9828.0.0&host_name=chromeos2-row4-rack2-host2&clobber_stateful=True"''
08/10 15:45:57.433 DEBUG|        dev_server:1873| Current CrOS auto-update status: CrOS update is just started.
08/10 15:46:07.467 DEBUG|             utils:0212| Running 'ssh 100.115.245.197 'curl "http://100.115.245.197:8082/get_au_status?full_update=True&payload_filename=payloads/chromeos_9828.0.0_snappy_canary-channel_full_test.bin-6409561c70e6dd0f7868f4bc4eadeca3&force_update=True&pid=1089&build_name=canary-channel/snappy/9828.0.0&host_name=chromeos2-row4-rack2-host2&clobber_stateful=True"''
08/10 15:46:08.489 DEBUG|        dev_server:1938| Failed to trigger auto-update process on devserver

Comment 1 by dhadd...@chromium.org, Aug 10 2017 

All tests that have the "timed out wiating to read" fail in this same exact spot.
Before failing they seemed to be able to ssh into the DUT to stage just fine. 

08/10 15:45:11.339 DEBUG|             utils:0212| Running 'ssh 100.115.245.197 'curl "http://100.115.245.197:8082/stage?artifacts=&files=stateful.tgz&async=True&archive_url=gs://chromeos-releases/canary-channel/snappy/9828.0.0"''
08/10 15:45:12.456 DEBUG|        dev_server:1056| response for RPC: 'Success'

Comment 2 by dhadd...@chromium.org, Aug 11 2017 

For lars, this has failed 5 of the last 7 runs:
https://wmatrix.googleplex.com/platform/paygen_au_canary?platforms=lars

Interestingly it has been the same DUT each time:
chromeos4-row11-rack7-host11

Comment 3 by dhadd...@chromium.org, Aug 11 2017 

For reks it has failed 4 times recently:
https://wmatrix.googleplex.com/platform/paygen_au_canary?platforms=reks

And all the failures were one DUT:
chromeos4-row11-rack6-host2

Comment 4 by dhadd...@chromium.org, Aug 11 2017 

Snappy:
https://wmatrix.googleplex.com/platform/paygen_au_dev?platforms=snappy

DUT: chromeos2-row4-rack2-host2

Comment 5 by dhadd...@chromium.org, Aug 11 2017

Cc: kathrelk...@chromium.org
Labels: M-61
+MO

Comment 6 by dhadd...@chromium.org, Aug 11 2017

Components: -Internals>Installer Infra>Client>ChromeOS

Comment 7 by dhadd...@chromium.org, Aug 11 2017

Cc: dgarr...@chromium.org dhadd...@chromium.org
Owner: ----
+infra deputy FYI

Comment 8 by dgarr...@chromium.org, Aug 11 2017

Mergedinto: 754314
Status: Duplicate (was: Untriaged)
All of these builds had DUTs attached with a known bad model of USB Dongle. I've locked/filed tickets for all I know about, and we are working on a process to identify them automatically.

Here is where I looked into the DUTS in question.

 https://crbug.com/754314
►

Sign in to add a comment