dev_server:0791| Error occurred with exit_code -15 when executing the ssh call |
||||||||
Issue descriptionDue to some unknown error, all Cyan pool:cts DUTs were blocked for 12 hours and 50% of CTS didn't run after yesterday afternoon. Parent job: https://ubercautotest.corp.google.com/afe/#tab_id=view_job&object_id=85881044 Failing child jobs: http://cautotest/afe/#tab_id=view_job&object_id=85891104 http://cautotest/afe/#tab_id=view_job&object_id=85891154 11/16 22:39:35.134 DEBUG| base_utils:0185| Running 'ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"'' 11/16 22:40:35.368 WARNI| base_utils:0912| run process timeout (60) fired on: ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"' 11/16 22:40:36.374 DEBUG| dev_server:0791| Error occurred with exit_code -15 when executing the ssh call: . 11/16 22:40:36.375 WARNI| retry:0181| <class 'autotest_lib.client.common_lib.error.CmdTimeoutError'>(Command <ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"'> failed, rc=-15, Command(s) did not complete within 60 seconds * Command: ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=au totest_packages&files=&async=True&archive_url=gs://chromeos-image-archive /cyan-release/R54-8743.85.0"' Exit status: -15 Duration: 61.2213640213 ) 11/16 22:40:36.376 WARNI| retry:0148| Retrying in 2.492768 seconds... 11/16 22:40:38.879 DEBUG| base_utils:0185| Running 'ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"'' 11/16 22:41:39.126 WARNI| base_utils:0912| run process timeout (60) fired on: ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"' 11/16 22:41:40.128 DEBUG| dev_server:0791| Error occurred with exit_code -15 when executing the ssh call: . 11/16 22:41:40.128 WARNI| retry:0181| <class 'autotest_lib.client.common_lib.error.CmdTimeoutError'>(Command <ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"'> failed, rc=-15, Command(s) did not complete within 60 seconds * Command: ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=au totest_packages&files=&async=True&archive_url=gs://chromeos-image-archive /cyan-release/R54-8743.85.0"' Exit status: -15 Duration: 61.2330970764 ) 11/16 22:41:40.129 WARNI| retry:0148| Retrying in 2.863819 seconds... 11/16 22:41:43.005 DEBUG| base_utils:0185| Running 'ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"'' 11/16 22:42:43.250 WARNI| base_utils:0912| run process timeout (60) fired on: ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"' 11/16 22:42:44.255 DEBUG| dev_server:0791| Error occurred with exit_code -15 when executing the ssh call: . 11/16 22:42:44.256 WARNI| retry:0181| <class 'autotest_lib.client.common_lib.error.CmdTimeoutError'>(Command <ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"'> failed, rc=-15, Command(s) did not complete within 60 seconds * Command: ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=au totest_packages&files=&async=True&archive_url=gs://chromeos-image-archive /cyan-release/R54-8743.85.0"' Exit status: -15 Duration: 61.2336370945 ) 11/16 22:42:44.256 WARNI| retry:0148| Retrying in 3.972167 seconds... 11/16 22:42:48.246 DEBUG| base_utils:0185| Running 'ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"'' 11/16 22:43:48.492 WARNI| base_utils:0912| run process timeout (60) fired on: ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"' 11/16 22:43:49.495 DEBUG| dev_server:0791| Error occurred with exit_code -15 when executing the ssh call: . 11/16 22:43:49.495 WARNI| retry:0181| <class 'autotest_lib.client.common_lib.error.CmdTimeoutError'>(Command <ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"'> failed, rc=-15, Command(s) did not complete within 60 seconds * Command: ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=au totest_packages&files=&async=True&archive_url=gs://chromeos-image-archive /cyan-release/R54-8743.85.0"' Exit status: -15 Duration: 61.2318110466 ) 11/16 22:43:49.496 WARNI| retry:0148| Retrying in 3.934465 seconds... 11/16 22:43:53.447 DEBUG| base_utils:0185| Running 'ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"'' 11/16 22:44:53.695 WARNI| base_utils:0912| run process timeout (60) fired on: ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"' 11/16 22:44:54.699 DEBUG| dev_server:0791| Error occurred with exit_code -15 when executing the ssh call: . 11/16 22:44:54.699 WARNI| retry:0181| <class 'autotest_lib.client.common_lib.error.CmdTimeoutError'>(Command <ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"'> failed, rc=-15, Command(s) did not complete within 60 seconds * Command: ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=au totest_packages&files=&async=True&archive_url=gs://chromeos-image-archive /cyan-release/R54-8743.85.0"' Exit status: -15 Duration: 61.2334671021 ) 11/16 22:44:54.700 WARNI| retry:0148| Retrying in 2.948491 seconds... 11/16 22:44:57.662 DEBUG| base_utils:0185| Running 'ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"'' 11/16 22:45:57.879 WARNI| base_utils:0912| run process timeout (60) fired on: ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"' 11/16 22:45:58.887 DEBUG| dev_server:0791| Error occurred with exit_code -15 when executing the ssh call: . 11/16 22:45:58.888 WARNI| retry:0181| <class 'autotest_lib.client.common_lib.error.CmdTimeoutError'>(Command <ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"'> failed, rc=-15, Command(s) did not complete within 60 seconds * Command: ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=au totest_packages&files=&async=True&archive_url=gs://chromeos-image-archive /cyan-release/R54-8743.85.0"' Exit status: -15 Duration: 61.2091860771 ) 11/16 22:45:58.888 WARNI| retry:0148| Retrying in 2.046674 seconds... 11/16 22:46:00.942 DEBUG| base_utils:0185| Running 'ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"'' 11/16 22:47:01.190 WARNI| base_utils:0912| run process timeout (60) fired on: ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"' 11/16 22:47:02.196 DEBUG| dev_server:0791| Error occurred with exit_code -15 when executing the ssh call: . 11/16 22:47:02.196 WARNI| retry:0181| <class 'autotest_lib.client.common_lib.error.CmdTimeoutError'>(Command <ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/cyan-release/R54-8743.85.0"'> failed, rc=-15, Command(s) did not complete within 60 seconds * Command: ssh 100.115.219.137 'curl "http://100.115.219.137:8082/stage?artifacts=au totest_packages&files=&async=True&archive_url=gs://chromeos-image-archive /cyan-release/R54-8743.85.0"' Exit status: -15 Duration: 61.2359850407 )
,
Nov 17 2016
[Auto-generated comment by a script] We noticed that this issue is targeted for M-54; it appears the fix may have landed after branch point, meaning a merge might be required. Please confirm if a merge is required here - if so add Merge-Request-54 label, otherwise remove Merge-TBD label. Thanks.
,
Nov 17 2016
Thanks. I will file a different issue for unblocking the tests if something like this happens. Blocking tests and devices for 12-15 hours doesn't sounds like a good thing.
,
Nov 17 2016
I'd think the problem here is that we retry too much. If a devserver is unreachable. It's actually not very useful to keep retrying for 12-15 hours. But yes, you should file a separate bug for that.
,
Nov 17 2016
Filed Issue 666454
,
Dec 30 2016
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Oct 14 2017
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by pprabhu@chromium.org
, Nov 17 2016