New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 820664 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Blocking:
issue 796626



Sign in to add a comment

lxc container is failing to create with a permissions error

Project Member Reported by haddowk@chromium.org, Mar 10 2018

Issue description


Some failures in the lab show the following error:

03/09 14:16:29.169 INFO |            config:0024| Configuration file does not exist, ignoring: /etc/chrome-infra/ts-mon.json
03/09 14:16:29.170 ERROR|            config:0244| ts_mon monitoring is disabled because the endpoint provided is invalid or not supported: 
03/09 14:16:29.171 NOTIC|      cros_logging:0038| ts_mon was set up.
03/09 14:16:29.171 DEBUG|          autoserv:0264| Trying to start servod.
03/09 14:16:29.260 WARNI|          autoserv:0272| Starting servod is aborted. The dut's servo_host attribute is not set to localhost.
03/09 14:16:29.261 DEBUG|             utils:0214| Running 'sudo lxc-ls --active'
03/09 14:16:29.296 DEBUG|             utils:0214| Running 'sudo test -e "/mnt/moblab/containers/base_05/rootfs"'
03/09 14:16:29.318 DEBUG|             utils:0214| Running 'cp /usr/local/autotest/results/drone_tmp/attach.12 /usr/local/autotest/results/4-moblab/192.168.231.100/attach.12'
03/09 14:16:29.328 DEBUG| container_factory:0068| Creating new container (name: test_4_1520633788_17090, lxc_path: /mnt/moblab/containers)
03/09 14:16:29.329 DEBUG|             utils:0214| Running 'sudo test -e "/mnt/moblab/containers/test_4_1520633788_17090"'
03/09 14:16:29.350 DEBUG|             utils:0214| Running 'sudo -n virt-what'
03/09 14:16:29.412 DEBUG|             utils:2344| virt-what output: 
03/09 14:16:29.413 DEBUG|             utils:0214| Running 'sudo lxc-clone --lxcpath /mnt/moblab/containers --newpath /mnt/moblab/containers --orig base_05 --new test_4_1520633788_17090  '
03/09 14:16:39.739 DEBUG|             utils:0214| Running 'sudo lxc-info -P /mnt/moblab/containers -n test_4_1520633788_17090 -c lxc.rootfs'
03/09 14:16:39.765 DEBUG|             utils:0214| Running 'sudo lxc-ls --active'
03/09 14:16:39.786 DEBUG|             utils:0214| Running 'sudo test -e "/mnt/moblab/containers/test_4_1520633788_17090/rootfs"'
03/09 14:16:39.796 DEBUG|             utils:0214| Running 'sudo test -e "/mnt/moblab/containers/base_05/rootfs"'
03/09 14:16:39.816 DEBUG|         container:0390| Destroying container /mnt/moblab/containers/test_4_1520633788_17090
03/09 14:16:39.817 DEBUG|             utils:0214| Running 'sudo lxc-destroy -P /mnt/moblab/containers -n test_4_1520633788_17090 -f'
03/09 14:16:42.103 INFO |        server_job:0218| FAIL	----	----	timestamp=1520633802	localtime=Mar 09 14:16:42	Failed to setup container for test: [Errno 13] Permission denied: '/mnt/moblab/containers/test_4_1520633788_17090/rootfs/container_id.json'. Check logs in ssp_logs folder for more details.
03/09 14:16:42.104 DEBUG|             utils:0214| Running 'sudo -n chown -R 246 "/usr/local/autotest/results/4-moblab/192.168.231.100"'
03/09 14:16:42.127 DEBUG|             utils:0214| Running 'sudo -n chgrp -R 246 "/usr/local/autotest/results/4-moblab/192.168.231.100"'
03/09 14:16:42.149 ERROR|         traceback:0013| Traceback (most recent call last):
03/09 14:16:42.150 ERROR|         traceback:0013|   File "/usr/local/autotest/server/autoserv", line 507, in run_autoserv
03/09 14:16:42.151 ERROR|         traceback:0013|     machines)
03/09 14:16:42.151 ERROR|         traceback:0013|   File "/usr/local/autotest/server/autoserv", line 168, in _run_with_ssp
03/09 14:16:42.152 ERROR|         traceback:0013|     dut_name=dut_name)
03/09 14:16:42.152 ERROR|         traceback:0013|   File "/usr/lib64/python2.7/site-packages/chromite/lib/metrics.py", line 483, in wrapper
03/09 14:16:42.153 ERROR|         traceback:0013|     return fn(*args, **kwargs)
03/09 14:16:42.154 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc/cleanup_if_fail.py", line 40, in func_cleanup_if_fail
03/09 14:16:42.154 ERROR|         traceback:0013|     return func(*args, **kwargs)
03/09 14:16:42.155 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc/container_bucket.py", line 202, in setup_test
03/09 14:16:42.155 ERROR|         traceback:0013|     container = self._factory.create_container(container_id)
03/09 14:16:42.156 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc/container_factory.py", line 78, in create_container
03/09 14:16:42.156 ERROR|         traceback:0013|     new_container.id = cid
03/09 14:16:42.157 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc/container.py", line 548, in id
03/09 14:16:42.157 ERROR|         traceback:0013|     self._id.save(self.rootfs)
03/09 14:16:42.158 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc/container.py", line 55, in save
03/09 14:16:42.158 ERROR|         traceback:0013|     with open(dst, 'w') as f:
03/09 14:16:42.158 ERROR|         traceback:0013| IOError: [Errno 13] Permission denied: '/mnt/moblab/containers/test_4_1520633788_17090/rootfs/container_id.json'
03/09 14:16:42.169 ERROR|          autoserv:0759| Uncaught SystemExit with code 1

 

Comment 1 by jkop@chromium.org, Mar 10 2018

Appears to be a result of crrev.com/c/954064. Reverted.
Owner: jkop@chromium.org
Status: Fixed (was: Untriaged)
Thanks for the quick revert - sorry the moblab-paladin was marked experimental because of a separate networking issue on moblab.

Comment 3 by jkop@chromium.org, Mar 10 2018

Could you point me to where I can grab logs for this?
http://chromeos2-row1-rack8-host3.cros.corp.google.com/results/4-moblab/192.168.231.100/ssp_logs/debug/

They will not stick around for long I will try to download them and attach
logs.tar.gz
7.7 KB Download

Comment 6 by jkop@chromium.org, Mar 10 2018

Thanks

Comment 7 by jkop@chromium.org, Mar 12 2018

Blocking: 796626

Sign in to add a comment