Devservers in Restricted Subnet |
||||
Issue descriptionHi, We're receiving some errors in regards to devservers in restricted subnets. Can we remove the devservers from the list of restricted subnets? Getting devservers for host: chromeos15-audiobox3-host1-servo 02/08 15:33:27.812 DEBUG| dev_server:0732| The host chromeos15-audiobox3-host1-servo (100.115.127.228) is in a restricted subnet. Try to locate a devserver inside subnet 100.115.124.0:22. 02/08 15:33:27.815 DEBUG| dev_server:0683| Pick one healthy devserver from ['http://100.115.127.250:8082', 'http://100.115.127.248:8082', 'http://100.115.127.249:8082', 'http://100.115.127.246:8082', 'http://100.115.127.245:8082'] 02/08 15:33:27.816 DEBUG| dev_server:0687| Check health for http://100.115.127.245:8082 02/08 15:33:27.817 DEBUG| utils:0214| Running 'ssh 100.115.127.245 'curl "http://100.115.127.245:8082/check_health?"'' 02/08 15:33:37.037 DEBUG| dev_server:0692| Pick http://100.115.127.245:8082
,
Feb 12 2018
Hi jashur, I don't see the error in the log you quoted. Those logs are all DEBUG level.
,
Feb 14 2018
Here's a couple logs. It says that it's in a restricted subnet:
02/13 09:51:56.719 WARNI| test:0637| The test failed with the following exception
Traceback (most recent call last):
File "/usr/local/autotest/client/common_lib/test.py", line 631, in _exec
_call_test_function(self.execute, *p_args, **p_dargs)
File "/usr/local/autotest/client/common_lib/test.py", line 837, in _call_test_function
raise error.UnhandledTestFail(e)
UnhandledTestFail: Unhandled DevServerException: CrOS auto-update failed for host chromeos15-audiobox3-host1: 0) SSHConnectionError: ssh: Could not resolve hostname chromeos15-audiobox3-host1: Name or service not known, 1) RootfsUpdateError: Device has not rebooted, still boot_id 8d0680c8-0cea-4573-89b1-a4d55f6bee15
Traceback (most recent call last):
File "/usr/local/autotest/client/common_lib/test.py", line 831, in _call_test_function
return func(*args, **dargs)
File "/usr/local/autotest/client/common_lib/test.py", line 495, in execute
dargs)
File "/usr/local/autotest/client/common_lib/test.py", line 362, in _call_run_once_with_retry
postprocess_profiled_run, args, dargs)
File "/usr/local/autotest/client/common_lib/test.py", line 400, in _call_run_once
self.run_once(*args, **dargs)
File "/usr/local/autotest/server/site_tests/provision_AutoUpdate/provision_AutoUpdate.py", line 127, in run_once
with_cheets=with_cheets)
File "/usr/local/autotest/server/afe_utils.py", line 124, in machine_install_and_update_labels
*args, **dargs)
File "/usr/local/autotest/server/hosts/cros_host.py", line 829, in machine_install_by_devserver
quick_provision=quick_provision)
File "/usr/local/autotest/client/common_lib/cros/dev_server.py", line 2526, in auto_update
error_msg % (host_name, real_error))
DevServerException: CrOS auto-update failed for host chromeos15-audiobox3-host1: 0) SSHConnectionError: ssh: Could not resolve hostname chromeos15-audiobox3-host1: Name or service not known, 1) RootfsUpdateError: Device has not rebooted, still boot_id 8d0680c8-0cea-4573-89b1-a4d55f6bee15
02/13 17:26:10.656 DEBUG| utils:0282| [stdout] CHROMEOS_AUSERVER=https://tools.google.com/service/update2
02/13 17:26:11.390 INFO | dev_server:0770| Getting devservers for host: chromeos15-audiobox3-host1-servo
02/13 17:26:11.451 DEBUG| dev_server:0791| The host chromeos15-audiobox3-host1-servo (100.115.127.228) is in a restricted subnet. Try to locate a devserver inside subnet 100.115.124.0:22.
02/13 17:26:11.452 DEBUG| dev_server:0742| Pick one healthy devserver from ['http://100.115.127.250:8082', 'http://100.115.127.248:8082', 'http://100.115.127.249:8082', 'http://100.115.127.246:8082', 'http://100.115.127.245:8082']
02/13 17:26:11.452 DEBUG| dev_server:0746| Check health for http://100.115.127.245:8082
02/13 17:26:11.453 DEBUG| utils:0214| Running 'ssh 100.115.127.245 'curl "http://100.115.127.245:8082/check_health?"''
02/13 17:26:19.982 DEBUG| dev_server:0751| Pick http://100.115.127.245:8082
02/13 17:26:19.988 DEBUG| ssh_host:0301| Running (ssh) 'cat "/etc/lsb-release"' from 'update_image|_check_for_reboot|_get_release_version|run|run|run_very_slowly'
,
Feb 15 2018
+kalin
,
Feb 15 2018
+kalin
,
Feb 16 2018
Related here is https://b/73127902 From host page https://buganizer.corp.google.com/issues/73127902 check debug logs for any of the failed jobs. E.g. in https://storage.cloud.google.com/chromeos-autotest-results/177022052-chromeos-test/chromeos15-audiobox3-host1/debug/autoserv.DEBUG I see several exceptions. SSHConnectionError: ssh: Could not resolve hostname chromeos15-audiobox3-host1: Name or service not known ... RootfsUpdateError: Device has not rebooted, still boot_id 6ec699a2-4576-44d8-8793-25c10679a433 ... DevServerException: CrOS auto-update failed for host chromeos15-audiobox3-host1: 0) SSHConnectionError: ssh: Could not resolve hostname chromeos15-audiobox3-host1: Name or service not known, 1) RootfsUpdateError: Device has not rebooted, still boot_id 6ec699a2-4576-44d8-8793-25c10679a433 Specifically for the restructed subnet I see several log snippets like 02/15 15:26:45.714 DEBUG| cros_host:0771| Resolving a devserver for auto-update 02/15 15:26:45.714 INFO | dev_server:0770| Getting devservers for host: chromeos15-audiobox3-host1 02/15 15:26:45.775 DEBUG| dev_server:0791| The host chromeos15-audiobox3-host1 (100.115.127.227) is in a restricted subnet. Try to locate a devserver inside subnet 100.115.124.0:22. 02/15 15:26:45.776 DEBUG| dev_server:0742| Pick one healthy devserver from ['http://100.115.127.250:8082', 'http://100.115.127.248:8082', 'http://100.115.127.249:8082', 'http://100.115.127.246:8082', 'http://100.115.127.245:8082']
,
Feb 22 2018
**friendly ping**
,
Feb 22 2018
Why does this have anything to do with restricted subnet? The DUT has five different devservers to pick from in its subnet. It's not obvious to me that the restricted subnet is causing any problems.
,
Feb 22 2018
@Kalin, could this possibly be an issue with the DUT itself? Let us know.
,
Feb 22 2018
As the originating issue b/73127902 states -------------------------- Test jobs Failed, b/c autotest server jobs (part of any special jobs too) fail with DevServerException: CrOS auto-update failed for host chromeos15-audiobox3-host1: 0) SSHConnectionError: ssh: Could not resolve hostname chromeos15-audiobox3-host1 -------------------------- I can have the host reimaged, but I doubt anything will improve.
,
Mar 5 2018
This was resolved in https://bugs.chromium.org/p/chromium/issues/detail?id=817379
,
Mar 5 2018
Thanks for the update. Jobs status look good(complete). |
||||
►
Sign in to add a comment |
||||
Comment 1 by xixuan@chromium.org
, Feb 10 2018