New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 810968 link

Starred by 3 users

Issue metadata

Status: Verified
Owner:
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Devservers in Restricted Subnet

Project Member Reported by jashur@chromium.org, Feb 10 2018

Issue description

Hi,

We're receiving some errors in regards to devservers in restricted subnets. Can we remove the devservers from the list of restricted subnets?

Getting devservers for host: chromeos15-audiobox3-host1-servo
02/08 15:33:27.812 DEBUG|        dev_server:0732| The host chromeos15-audiobox3-host1-servo (100.115.127.228) is in a restricted subnet. Try to locate a devserver inside subnet 100.115.124.0:22.
02/08 15:33:27.815 DEBUG|        dev_server:0683| Pick one healthy devserver from ['http://100.115.127.250:8082', 'http://100.115.127.248:8082', 'http://100.115.127.249:8082', 'http://100.115.127.246:8082', 'http://100.115.127.245:8082']
02/08 15:33:27.816 DEBUG|        dev_server:0687| Check health for http://100.115.127.245:8082
02/08 15:33:27.817 DEBUG|             utils:0214| Running 'ssh 100.115.127.245 'curl "http://100.115.127.245:8082/check_health?"''
02/08 15:33:37.037 DEBUG|        dev_server:0692| Pick http://100.115.127.245:8082

 

Comment 1 by xixuan@chromium.org, Feb 10 2018

Where's the error you receive?

Which devserver you want to remove from the subnet 100.115.124.0?

If you removed all, you cannot resolve a devserver for the DUTs that in the same subnet with it.

Comment 2 by pho...@chromium.org, Feb 12 2018

Hi jashur, I don't see the error in the log you quoted. Those logs are all DEBUG level.

Comment 3 by jashur@chromium.org, Feb 14 2018

Here's a couple logs. It says that it's in a restricted subnet:

02/13 09:51:56.719 WARNI|              test:0637| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/test.py", line 631, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 837, in _call_test_function
    raise error.UnhandledTestFail(e)
UnhandledTestFail: Unhandled DevServerException: CrOS auto-update failed for host chromeos15-audiobox3-host1: 0) SSHConnectionError: ssh: Could not resolve hostname chromeos15-audiobox3-host1: Name or service not known, 1) RootfsUpdateError: Device has not rebooted, still boot_id 8d0680c8-0cea-4573-89b1-a4d55f6bee15
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/test.py", line 831, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 495, in execute
    dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 362, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 400, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/server/site_tests/provision_AutoUpdate/provision_AutoUpdate.py", line 127, in run_once
    with_cheets=with_cheets)
  File "/usr/local/autotest/server/afe_utils.py", line 124, in machine_install_and_update_labels
    *args, **dargs)
  File "/usr/local/autotest/server/hosts/cros_host.py", line 829, in machine_install_by_devserver
    quick_provision=quick_provision)
  File "/usr/local/autotest/client/common_lib/cros/dev_server.py", line 2526, in auto_update
    error_msg % (host_name, real_error))
DevServerException: CrOS auto-update failed for host chromeos15-audiobox3-host1: 0) SSHConnectionError: ssh: Could not resolve hostname chromeos15-audiobox3-host1: Name or service not known, 1) RootfsUpdateError: Device has not rebooted, still boot_id 8d0680c8-0cea-4573-89b1-a4d55f6bee15


02/13 17:26:10.656 DEBUG|             utils:0282| [stdout] CHROMEOS_AUSERVER=https://tools.google.com/service/update2
02/13 17:26:11.390 INFO |        dev_server:0770| Getting devservers for host: chromeos15-audiobox3-host1-servo
02/13 17:26:11.451 DEBUG|        dev_server:0791| The host chromeos15-audiobox3-host1-servo (100.115.127.228) is in a restricted subnet. Try to locate a devserver inside subnet 100.115.124.0:22.
02/13 17:26:11.452 DEBUG|        dev_server:0742| Pick one healthy devserver from ['http://100.115.127.250:8082', 'http://100.115.127.248:8082', 'http://100.115.127.249:8082', 'http://100.115.127.246:8082', 'http://100.115.127.245:8082']
02/13 17:26:11.452 DEBUG|        dev_server:0746| Check health for http://100.115.127.245:8082
02/13 17:26:11.453 DEBUG|             utils:0214| Running 'ssh 100.115.127.245 'curl "http://100.115.127.245:8082/check_health?"''
02/13 17:26:19.982 DEBUG|        dev_server:0751| Pick http://100.115.127.245:8082
02/13 17:26:19.988 DEBUG|          ssh_host:0301| Running (ssh) 'cat "/etc/lsb-release"' from 'update_image|_check_for_reboot|_get_release_version|run|run|run_very_slowly'

Comment 4 by jashur@chromium.org, Feb 15 2018

Cc: ka...@chromium.org
+kalin

Comment 5 by jashur@chromium.org, Feb 15 2018

+kalin

Comment 6 by ka...@chromium.org, Feb 16 2018

Related here is https://b/73127902

From host page https://buganizer.corp.google.com/issues/73127902 check debug logs for any of the failed jobs. 

E.g. in https://storage.cloud.google.com/chromeos-autotest-results/177022052-chromeos-test/chromeos15-audiobox3-host1/debug/autoserv.DEBUG


I see several exceptions.

SSHConnectionError: ssh: Could not resolve hostname chromeos15-audiobox3-host1: Name or service not known
...
RootfsUpdateError: Device has not rebooted, still boot_id 6ec699a2-4576-44d8-8793-25c10679a433
...
  DevServerException: CrOS auto-update failed for host chromeos15-audiobox3-host1: 0) SSHConnectionError: ssh: Could not resolve hostname chromeos15-audiobox3-host1: Name or service not known, 1) RootfsUpdateError: Device has not rebooted, still boot_id 6ec699a2-4576-44d8-8793-25c10679a433


Specifically for the restructed subnet I see several log snippets like
02/15 15:26:45.714 DEBUG|         cros_host:0771| Resolving a devserver for auto-update
02/15 15:26:45.714 INFO |        dev_server:0770| Getting devservers for host: chromeos15-audiobox3-host1
02/15 15:26:45.775 DEBUG|        dev_server:0791| The host chromeos15-audiobox3-host1 (100.115.127.227) is in a restricted subnet. Try to locate a devserver inside subnet 100.115.124.0:22.
02/15 15:26:45.776 DEBUG|        dev_server:0742| Pick one healthy devserver from ['http://100.115.127.250:8082', 'http://100.115.127.248:8082', 'http://100.115.127.249:8082', 'http://100.115.127.246:8082', 'http://100.115.127.245:8082']

Comment 7 by jashur@chromium.org, Feb 22 2018

**friendly ping**
Why does this have anything to do with restricted subnet?  The DUT has five different devservers to pick from in its subnet.  It's not obvious to me that the restricted subnet is causing any problems.

Comment 9 by jashur@chromium.org, Feb 22 2018

@Kalin, could this possibly be an issue with the DUT itself? Let us know.

Comment 10 by ka...@chromium.org, Feb 22 2018

As the originating issue b/73127902 states
--------------------------
Test jobs Failed, b/c autotest server jobs (part of any special jobs too) fail with 
DevServerException: CrOS auto-update failed for host chromeos15-audiobox3-host1: 0) SSHConnectionError: ssh: Could not resolve hostname chromeos15-audiobox3-host1
--------------------------

I can have the host reimaged, but I doubt anything will improve.
Status: Fixed (was: Untriaged)
This was resolved in https://bugs.chromium.org/p/chromium/issues/detail?id=817379
Status: Verified (was: Fixed)
Thanks for the update. Jobs status look good(complete).

Sign in to add a comment