New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 747186 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Jul 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug-Regression



Sign in to add a comment

Test "network_WiFi_UpdateRouter" fails with Name Error

Project Member Reported by aashuto...@chromium.org, Jul 21 2017

Issue description


Traceback (most recent call last):
  File "/usr/local/autotest/server/hosts/cros_host.py", line 920, in machine_install
    update_url, force_update, updater)
  File "/usr/local/autotest/server/hosts/cros_host.py", line 498, in _try_stateful_update
    self.prepare_for_update()
  File "/usr/local/autotest/server/hosts/jetstream_host.py", line 90, in prepare_for_update
    except AutoservRunError:
NameError: global name 'AutoservRunError' is not defined

The test started failing around June 14th,
https://tests.corp.google.com/invocations?searchFor=network_WiFi_UpdateRouter%20lars


After throwing the name error, the server tried to reboot the router and tries to ssh back which it fails. So the test fails with  "Host did not return from reboot" Error. 

07/19 01:54:52.797 INFO |        server_job:0206| 	START	----	reboot	timestamp=1500454492	localtime=Jul 19 01:54:52	
07/19 01:54:52.799 INFO |        server_job:0206| 		GOOD	----	reboot.start	timestamp=1500454492	localtime=Jul 19 01:54:52	

07/19 01:54:53.504 DEBUG|          ssh_host:0296| Running (ssh) 'if [ -f '/proc/sys/kernel/random/boot_id' ]; then cat '/proc/sys/kernel/random/boot_id'; else echo 'no boot_id available'; fi' from 'wait_for_restart|wait_down|get_boot_id|run|wrapper|run_very_slowly'
07/19 01:54:53.826 DEBUG|             utils:0280| [stdout] d8e63aec-2b7c-4654-b6da-04d9e172bdf0
07/19 01:54:54.858 DEBUG|          ssh_host:0296| Running (ssh) 'if [ -f '/proc/sys/kernel/random/boot_id' ]; then cat '/proc/sys/kernel/random/boot_id'; else echo 'no boot_id available'; fi' from 'wait_for_restart|wait_down|get_boot_id|run|wrapper|run_very_slowly'
07/19 01:55:27.335 DEBUG|             utils:0280| [stdout] 08d7cc50-2ded-4744-b95a-224776d4ead2
07/19 01:55:27.343 DEBUG|      abstract_ssh:0684| Host chromeos9-row1-cart1-host4-router now has boot_id 08d7cc50-2ded-4744-b95a-224776d4ead2 and so must have rebooted
07/19 01:55:27.366 DEBUG|          ssh_host:0296| Running (ssh) 'true' from 'wait_up|is_up|ssh_ping|run|wrapper|run_very_slowly'


Running (ssh) '{ ps -e || ps; } | grep 'ap-controller'' from 'wait_for_restart|wait_up|are_wait_up_processes_up|run|wrapper|run_very_slowly'
07/19 02:03:27.360 DEBUG|      abstract_ssh:0601| Host chromeos9-row1-cart1-host4-router is still down after waiting 480 seconds
07/19 02:03:27.362 INFO |        server_job:0206| 		ABORT	----	reboot.verify	timestamp=1500455007	localtime=Jul 19 02:03:27	Host did not return from reboot
07/19 02:03:27.364 INFO |        server_job:0206| 	END FAIL	----	reboot	timestamp=1500455007	localtime=Jul 19 02:03:27	Host did not return from reboot
  Traceback (most recent call last):
    File "/usr/local/autotest/server/server_job.py", line 1062, in run_op
      op_func()
    File "/usr/local/autotest/server/hosts/remote.py", line 160, in reboot
      **dargs)
    File "/usr/local/autotest/server/hosts/remote.py", line 229, in wait_for_restart
      self.log_op(self.OP_REBOOT, op_func)
    File "/usr/local/autotest/client/common_lib/hosts/base_classes.py", line 548, in log_op
      op_func()
    File "/usr/local/autotest/server/hosts/remote.py", line 228, in op_func
      super(RemoteHost, self).wait_for_restart(timeout=timeout, **dargs)
    File "/usr/local/autotest/client/common_lib/hosts/base_classes.py", line 309, in wait_for_restart
      raise error.AutoservRebootError("Host did not return from reboot")
  AutoservRebootError: Host did not return from reboot

 
Components: -Infra Infra>Client>ChromeOS
Owner: lgoo...@chromium.org
Status: Assigned (was: Unconfirmed)
I suspect this happened when jetstream_host autodetection was turned on, https://chromium-review.googlesource.com/c/528487/.

It looks like the ap-configure command is not found, which is not expected.
And the ap-controller process is not found, which is also unexpected.

Is this a modified Jetstream host?

> It looks like the ap-configure command is not found, which is not expected.

That may be causing the root error, but the summary also complains
of this:
    def prepare_for_update(self):
        """Prepare the host for an update."""
        logging.debug('Jetstream: Prepare for update')
        try:
            self.cleanup_services()
        except AutoservRunError:
            logging.exception('Failed to reset host')

"AutoservRunError" needs to be "error.AutoservRunError"

Comment 4 by ecgh@chromium.org, Jul 21 2017

Cc: -ecgh@google.com e...@chromium.com
The whirlwind routers in the ChromeOS lab wifi cells use the whirlwind testbed-ap profile, which does not include the jetstream ap-daemons package. See  bug 702007  for more details.

Comment 5 by ecgh@chromium.org, Jul 21 2017

Cc: -e...@chromium.com ecgh@chromium.org
JetstreamHost assumes the device is running standard jetstream images. Since you are running a different stack, you may want to force the test to use CrosHost instead by passing in a host_class argument to create_host:

https://chromium.git.corp.google.com/chromiumos/third_party/autotest/+/5a2cac10377d910d6d7f2553b99b595cb570dae2/server/site_tests/network_WiFi_UpdateRouter/network_WiFi_UpdateRouter.py#80

Project Member

Comment 7 by bugdroid1@chromium.org, Jul 22 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/af7e16dc7685b5d09338069b3514b5dd9abdadff

commit af7e16dc7685b5d09338069b3514b5dd9abdadff
Author: Laurence Goodby <lgoodby@google.com>
Date: Sat Jul 22 07:09:12 2017

autotest: Handle jetstream_host cleanup errors

This fixes a NameError that is raised when jetstream host cleanup fails.

BUG= chromium:747186 
TEST=run unit test.

Change-Id: I6cc2edc474be3e31f08873c46419b9ab30799ff4
Reviewed-on: https://chromium-review.googlesource.com/580703
Commit-Ready: Laurence Goodby <lgoodby@chromium.org>
Tested-by: Laurence Goodby <lgoodby@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@google.com>
Reviewed-by: Laurence Goodby <lgoodby@chromium.org>

[modify] https://crrev.com/af7e16dc7685b5d09338069b3514b5dd9abdadff/server/hosts/jetstream_host.py
[add] https://crrev.com/af7e16dc7685b5d09338069b3514b5dd9abdadff/server/hosts/jetstream_host_unittest.py

The NameError issue is fixed, but the test is likely to continue failing as long as the AP is not running ap-controller. This can be worked around by modifying the test per comment #6.

Status: Fixed (was: Assigned)

Comment 10 by ecgh@chromium.org, Jul 24 2017

I've created  issue 748151  for fixing the tests as suggested here (#c6).

Comment 11 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Sign in to add a comment