New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 724899 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner: ----
Closed: May 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

moblab: Cannot run CTS test ("container base_05 is not defined.")

Project Member Reported by drinkcat@chromium.org, May 21 2017

Issue description

CHROMEOS_RELEASE_BUILDER_PATH=guado_moblab-release/R58-9334.68.0

Trying to run CTS tests on moblab, e.g. run arc-cts suite. All tests fail, for example cheets_CTS_N.7.1_r5.arm.CtsSampleDeviceTestCases:

ssp_logs/debug/autoserv.DEBUG looks like this:
05/22 04:42:27.866 DEBUG|        base_utils:0185| Running 'sudo lxc-clone -p /mnt/moblab/containers -P /mnt/moblab/containers base_05 test_1162_1495399347_2329  '
05/22 04:42:27.875 DEBUG|        base_utils:0185| Running 'sudo lxc-ls --active'
05/22 04:42:27.896 DEBUG|        base_utils:0185| Running 'sudo test -e "/mnt/moblab/containers/base_05/rootfs"'
05/22 04:42:27.906 DEBUG|        base_utils:0185| Running 'sudo test -e "/mnt/moblab/containers/base_02/rootfs"'
05/22 04:42:27.928 INFO |        server_job:0184| FAIL	----	----	timestamp=1495399347	localtime=May 22 04:42:27	Failed to setup container for test: Command <sudo lxc-clone -p /mnt/moblab/containers -P /mnt/moblab/containers base_05 test_1162_1495399347_2329  > failed, rc=1, Command returned non-zero exit status
  * Command: 
      sudo lxc-clone -p /mnt/moblab/containers -P /mnt/moblab/containers
      base_05 test_1162_1495399347_2329
  Exit status: 1
  Duration: 0.00598216056824
  
  stderr:
  Error: container base_05 is not defined. Check logs in ssp_logs folder for more details.
05/22 04:42:27.929 DEBUG|        base_utils:0185| Running 'sudo -n chown -R 246 "/usr/local/autotest/results/1162-moblab/192.168.231.101"'
05/22 04:42:27.942 DEBUG|        base_utils:0185| Running 'sudo -n chgrp -R 246 "/usr/local/autotest/results/1162-moblab/192.168.231.101"'
05/22 04:42:27.961 ERROR|         traceback:0013| Traceback (most recent call last):
05/22 04:42:27.961 ERROR|         traceback:0013|   File "/usr/local/autotest/server/autoserv", line 530, in run_autoserv
05/22 04:42:27.962 ERROR|         traceback:0013|     machines)
05/22 04:42:27.963 ERROR|         traceback:0013|   File "/usr/local/autotest/server/autoserv", line 163, in _run_with_ssp
05/22 04:42:27.963 ERROR|         traceback:0013|     dut_name=dut_name)
05/22 04:42:27.964 ERROR|         traceback:0013|   File "/usr/lib64/python2.7/site-packages/chromite/lib/metrics.py", line 274, in wrapper
05/22 04:42:27.965 ERROR|         traceback:0013|     return fn(*args, **kwargs)
05/22 04:42:27.965 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc.py", line 235, in func_cleanup_if_fail
05/22 04:42:27.966 ERROR|         traceback:0013|     return func(*args, **kwargs)
05/22 04:42:27.967 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc.py", line 922, in setup_test
05/22 04:42:27.967 ERROR|         traceback:0013|     container = self.create_from_base(name)
05/22 04:42:27.967 ERROR|         traceback:0013|   File "/usr/lib64/python2.7/site-packages/chromite/lib/metrics.py", line 274, in wrapper
05/22 04:42:27.967 ERROR|         traceback:0013|     return fn(*args, **kwargs)
05/22 04:42:27.968 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc.py", line 791, in create_from_base
05/22 04:42:27.968 ERROR|         traceback:0013|     utils.run(cmd)
05/22 04:42:27.968 ERROR|         traceback:0013|   File "/usr/local/autotest/client/common_lib/base_utils.py", line 740, in run
05/22 04:42:27.970 ERROR|         traceback:0013|     "Command returned non-zero exit status")
05/22 04:42:27.970 ERROR|         traceback:0013| CmdError: Command <sudo lxc-clone -p /mnt/moblab/containers -P /mnt/moblab/containers base_05 test_1162_1495399347_2329  > failed, rc=1, Command returned non-zero exit status
05/22 04:42:27.971 ERROR|         traceback:0013| * Command: 
05/22 04:42:27.971 ERROR|         traceback:0013|     sudo lxc-clone -p /mnt/moblab/containers -P /mnt/moblab/containers
05/22 04:42:27.971 ERROR|         traceback:0013|     base_05 test_1162_1495399347_2329
05/22 04:42:27.971 ERROR|         traceback:0013| Exit status: 1
05/22 04:42:27.972 ERROR|         traceback:0013| Duration: 0.00598216056824
05/22 04:42:27.972 ERROR|         traceback:0013| 
05/22 04:42:27.972 ERROR|         traceback:0013| stderr:
05/22 04:42:27.973 ERROR|         traceback:0013| Error: container base_05 is not defined
05/22 04:42:27.974 ERROR|          autoserv:0765| 1
Traceback (most recent call last):
  File "/usr/local/autotest/server/autoserv", line 761, in main
    use_ssp)
  File "/usr/local/autotest/server/autoserv", line 574, in run_autoserv
    sys.exit(exit_code)
SystemExit: 1
05/22 04:42:28.032 DEBUG|   logging_manager:0627| Logging subprocess finished
05/22 04:42:28.032 DEBUG|   logging_manager:0627| Logging subprocess finished

Details (ssh tunneling over my workstation):
Job: http://drinkcat.tpe.corp.google.com:8080/afe/#tab_id=view_job&object_id=1162
Logs: http://drinkcat.tpe.corp.google.com:8080/results/1162-moblab/192.168.231.101/ssp_logs/debug/autoserv.DEBUG
 

Comment 1 by dshi@chromium.org, May 22 2017

Can you try to reboot the moblab. Moblab sets up base container during boot up. It seems that the issue is that base_05 is not downloaded and staged properly in /usr/local/autotest/containers.

Reboot it can force the container setup to rerun.
Status: WontFix (was: Untriaged)
Indeed, works after a reboot. Thanks!

Sign in to add a comment