provision failure on skunk-1 |
|||
Issue description./bin/run_server_task ShardProvisionTask --host_server chromeos-skunk-1.mtv.corp.google.com failed with the error output, what's the proper way to fix this and move forward ? Can I manually remove (sudo rm /var/lib/apt/lists/lock) and rerun the command ? [chromeos-skunk-1.mtv.corp.google.com] sudo: virt-what update_kernel is only applicable to Ganeti instances. 2017-08-10 10:29:57,297 INFO| [chan 3] Opened sftp connection (server version 3) [chromeos-skunk-1.mtv.corp.google.com] put: /usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/netrc -> /root/.netrc 2017-08-10 10:29:57,725 INFO| [chan 3] sftp session closed. [chromeos-skunk-1.mtv.corp.google.com] sudo: apt-get -y update Fatal error: sudo() received nonzero return code 100 while executing! Requested: apt-get -y update Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "apt-get -y update" ========================================================= Standard output ========================================================= Reading package lists... Done E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable) E: Unable to lock directory /var/lib/apt/lists/ ==================================================================================================================================== Aborting. 2017-08-10 10:29:58,941 ERRO| Traceback (most recent call last): File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/tasks/task.py", line 66, in run self._run() File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/tasks/atomic_shard.py", line 72, in _run self._deploy_puppet() File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/tasks/atomic_common.py", line 93, in decorated_func return func(self) File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/tasks/atomic_common.py", line 164, in _deploy_puppet fabfile.deploy_puppet(server_type=self.server_type) File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/fabfile.py", line 162, in deploy_puppet _bootstrap_puppet() File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/fabfile.py", line 175, in _bootstrap_puppet sudo('apt-get -y update') File "/usr/local/google/home/nxia/.cache/cros_venv/venv-2.7.6-2acae930a65a57e1e4b565419962f29e/local/lib/python2.7/site-packages/fabric/network.py", line 677, in host_prompting_wrapper return func(*args, **kwargs) File "/usr/local/google/home/nxia/.cache/cros_venv/venv-2.7.6-2acae930a65a57e1e4b565419962f29e/local/lib/python2.7/site-packages/fabric/operations.py", line 1146, in sudo capture_buffer_size=capture_buffer_size, File "/usr/local/google/home/nxia/.cache/cros_venv/venv-2.7.6-2acae930a65a57e1e4b565419962f29e/local/lib/python2.7/site-packages/fabric/operations.py", line 952, in _run_command error(message=msg, stdout=out, stderr=err) File "/usr/local/google/home/nxia/.cache/cros_venv/venv-2.7.6-2acae930a65a57e1e4b565419962f29e/local/lib/python2.7/site-packages/fabric/utils.py", line 358, in error return func(message) File "/usr/local/google/home/nxia/.cache/cros_venv/venv-2.7.6-2acae930a65a57e1e4b565419962f29e/local/lib/python2.7/site-packages/fabric/utils.py", line 54, in abort raise env.abort_exception(msg) FabricException: sudo() received nonzero return code 100 while executing! Requested: apt-get -y update Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "apt-get -y update" ========================================================= Standard output ========================================================= Reading package lists... Done E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable) E: Unable to lock directory /var/lib/apt/lists/ ==================================================================================================================================== 2017-08-10 10:29:58,941 ERRO| sudo() received nonzero return code 100 while executing! Requested: apt-get -y update Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "apt-get -y update" ========================================================= Standard output ========================================================= Reading package lists... Done E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable) E: Unable to lock directory /var/lib/apt/lists/ ==================================================================================================================================== 2017-08-10 10:29:58,942 INFO| Printing out task report. { "sub_reports": [], "exception": "FabricException('sudo() received nonzero return code 100 while executing!\\n\\nRequested: apt-get -y update\\nExecuted: sudo -S -p \\'sudo password:\\' /bin/bash -l -c \"apt-get -y update\"\\n\\n========================================================= Standard output =========================================================\\n\\nReading package lists... 0%\\r\\rReading package lists... 0%\\r\\rReading package lists... 1%\\r\\rReading package lists... 49%\\r\\rReading package lists... 49%\\r\\rReading package lists... 81%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 99%\\r\\rReading package lists... 99%\\r\\rReading package lists... 99%\\r\\rReading package lists... 99%\\r\\rReading package lists... 99%\\r\\rReading package lists... 99%\\r\\rReading package lists... Done\\r\\r\\nE: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)\\r\\nE: Unable to lock directory /var/lib/apt/lists/\\n\\n====================================================================================================================================',)", "is_successful": false, "description": "sudo() received nonzero return code 100 while executing!\n\nRequested: apt-get -y update\nExecuted: sudo -S -p 'sudo password:' /bin/bash -l -c \"apt-get -y update\"\n\n========================================================= Standard output =========================================================\n\nReading package lists... 0%\r\rReading package lists... 0%\r\rReading package lists... 1%\r\rReading package lists... 49%\r\rReading package lists... 49%\r\rReading package lists... 81%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 99%\r\rReading package lists... 99%\r\rReading package lists... 99%\r\rReading package lists... 99%\r\rReading package lists... 99%\r\rReading package lists... 99%\r\rReading package lists... Done\r\r\nE: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)\r\nE: Unable to lock directory /var/lib/apt/lists/\n\n====================================================================================================================================", "arguments_used": { "host_server": "chromeos-skunk-1.mtv.corp.google.com" }, "task_name": "ShardProvisionTask" }
,
Aug 10 2017
I logged into the other machines, they'll have /var/lib/apt/lists/lock, do I need to remove them first and run the provision command ? can the provision command clean them up automatically?
,
Aug 10 2017
you'd better manually run the command on the serve first.
,
Aug 10 2017
I login each machine and delete the lock manually. provision skunk-2 gave an error, should I just re-run the provision command ? [chromeos-skunk-2.mtv.corp.google.com] out: HEAD is now at 11f11de... Add backports.shutil_which [chromeos-skunk-2.mtv.corp.google.com] out: Error: Syntax error at 'user' at /root/chromeos-admin/puppet/modules/lab/manifests/core_cluster.pp:24:5 on node chromeos-skunk-2.mtv.corp.google.com [chromeos-skunk-2.mtv.corp.google.com] out: Error: Syntax error at 'user' at /root/chromeos-admin/puppet/modules/lab/manifests/core_cluster.pp:24:5 on node chromeos-skunk-2.mtv.corp.google.com [chromeos-skunk-2.mtv.corp.google.com] out: Fatal error: sudo() received nonzero return code 1 while executing! Requested: HOME=/root/ FACTER_server_type="autotest-shard" /root/chromeos-admin/puppet/sync_and_run_puppet -f Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "HOME=/root/ FACTER_server_type=\"autotest-shard\" /root/chromeos-admin/puppet/sync_and_run_puppet -f"
,
Aug 10 2017
,
Aug 10 2017
,
Sep 18 2017
,
Sep 18 2017
also need to remove /var/lib/dpkg/lock Hit:7 https://rapture-prod.corp.google.com goobuntu-puppet-addons InRelease E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable) E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it? |
|||
►
Sign in to add a comment |
|||
Comment 1 by shuqianz@chromium.org
, Aug 10 2017