New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 754337 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Aug 2017
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 1
Type: Bug

Blocked on:
issue 754492

Blocking:
issue 754036



Sign in to add a comment

provision failure on skunk-1

Project Member Reported by nxia@chromium.org, Aug 10 2017

Issue description

 ./bin/run_server_task ShardProvisionTask --host_server chromeos-skunk-1.mtv.corp.google.com

failed with the error output, what's the proper way to fix this and move forward ? Can I manually remove (sudo rm /var/lib/apt/lists/lock) and rerun the command ?


[chromeos-skunk-1.mtv.corp.google.com] sudo: virt-what
update_kernel is only applicable to Ganeti instances.
2017-08-10 10:29:57,297 INFO| [chan 3] Opened sftp connection (server version 3)
[chromeos-skunk-1.mtv.corp.google.com] put: /usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/netrc -> /root/.netrc
2017-08-10 10:29:57,725 INFO| [chan 3] sftp session closed.
[chromeos-skunk-1.mtv.corp.google.com] sudo: apt-get -y update

Fatal error: sudo() received nonzero return code 100 while executing!

Requested: apt-get -y update
Executed: sudo -S -p 'sudo password:'  /bin/bash -l -c "apt-get -y update"

========================================================= Standard output =========================================================

Reading package lists... Done
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/

====================================================================================================================================

Aborting.
2017-08-10 10:29:58,941 ERRO| Traceback (most recent call last):
  File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/tasks/task.py", line 66, in run
    self._run()
  File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/tasks/atomic_shard.py", line 72, in _run
    self._deploy_puppet()
  File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/tasks/atomic_common.py", line 93, in decorated_func
    return func(self)
  File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/tasks/atomic_common.py", line 164, in _deploy_puppet
    fabfile.deploy_puppet(server_type=self.server_type)
  File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/fabfile.py", line 162, in deploy_puppet
    _bootstrap_puppet()
  File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/fabfile.py", line 175, in _bootstrap_puppet
    sudo('apt-get -y update')
  File "/usr/local/google/home/nxia/.cache/cros_venv/venv-2.7.6-2acae930a65a57e1e4b565419962f29e/local/lib/python2.7/site-packages/fabric/network.py", line 677, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/usr/local/google/home/nxia/.cache/cros_venv/venv-2.7.6-2acae930a65a57e1e4b565419962f29e/local/lib/python2.7/site-packages/fabric/operations.py", line 1146, in sudo
    capture_buffer_size=capture_buffer_size,
  File "/usr/local/google/home/nxia/.cache/cros_venv/venv-2.7.6-2acae930a65a57e1e4b565419962f29e/local/lib/python2.7/site-packages/fabric/operations.py", line 952, in _run_command
    error(message=msg, stdout=out, stderr=err)
  File "/usr/local/google/home/nxia/.cache/cros_venv/venv-2.7.6-2acae930a65a57e1e4b565419962f29e/local/lib/python2.7/site-packages/fabric/utils.py", line 358, in error
    return func(message)
  File "/usr/local/google/home/nxia/.cache/cros_venv/venv-2.7.6-2acae930a65a57e1e4b565419962f29e/local/lib/python2.7/site-packages/fabric/utils.py", line 54, in abort
    raise env.abort_exception(msg)
FabricException: sudo() received nonzero return code 100 while executing!

Requested: apt-get -y update
Executed: sudo -S -p 'sudo password:'  /bin/bash -l -c "apt-get -y update"

========================================================= Standard output =========================================================

Reading package lists... Done
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/

====================================================================================================================================

2017-08-10 10:29:58,941 ERRO| sudo() received nonzero return code 100 while executing!

Requested: apt-get -y update
Executed: sudo -S -p 'sudo password:'  /bin/bash -l -c "apt-get -y update"

========================================================= Standard output =========================================================

Reading package lists... Done
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/

====================================================================================================================================
2017-08-10 10:29:58,942 INFO| Printing out task report.
{
  "sub_reports": [],
  "exception": "FabricException('sudo() received nonzero return code 100 while executing!\\n\\nRequested: apt-get -y update\\nExecuted: sudo -S -p \\'sudo password:\\'  /bin/bash -l -c \"apt-get -y update\"\\n\\n========================================================= Standard output =========================================================\\n\\nReading package lists... 0%\\r\\rReading package lists... 0%\\r\\rReading package lists... 1%\\r\\rReading package lists... 49%\\r\\rReading package lists... 49%\\r\\rReading package lists... 81%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 98%\\r\\rReading package lists... 99%\\r\\rReading package lists... 99%\\r\\rReading package lists... 99%\\r\\rReading package lists... 99%\\r\\rReading package lists... 99%\\r\\rReading package lists... 99%\\r\\rReading package lists... Done\\r\\r\\nE: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)\\r\\nE: Unable to lock directory /var/lib/apt/lists/\\n\\n====================================================================================================================================',)",
  "is_successful": false,
  "description": "sudo() received nonzero return code 100 while executing!\n\nRequested: apt-get -y update\nExecuted: sudo -S -p 'sudo password:'  /bin/bash -l -c \"apt-get -y update\"\n\n========================================================= Standard output =========================================================\n\nReading package lists... 0%\r\rReading package lists... 0%\r\rReading package lists... 1%\r\rReading package lists... 49%\r\rReading package lists... 49%\r\rReading package lists... 81%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 98%\r\rReading package lists... 99%\r\rReading package lists... 99%\r\rReading package lists... 99%\r\rReading package lists... 99%\r\rReading package lists... 99%\r\rReading package lists... 99%\r\rReading package lists... Done\r\r\nE: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)\r\nE: Unable to lock directory /var/lib/apt/lists/\n\n====================================================================================================================================",
  "arguments_used": {
    "host_server": "chromeos-skunk-1.mtv.corp.google.com"
  },
  "task_name": "ShardProvisionTask"
}

 
Status: Fixed (was: Untriaged)
Searched around, it seems the apt-get cmd is busy with apt-get virt-what. Run the following cmd to kill the process apt-get processes and run again:
sudo rm /var/lib/apt/lists/* -vf 
sudo apt-get update

I've kicked off task to provision the server. You don't need to run the task again

Comment 2 by nxia@chromium.org, Aug 10 2017

I logged into the other machines, they'll have /var/lib/apt/lists/lock, do I need to remove them first and run the provision command ?

can the provision command clean them up automatically?
you'd better manually run the command on the serve first.

Comment 4 by nxia@chromium.org, Aug 10 2017

I login each machine and delete the lock manually. 

provision skunk-2 gave an error, should I just re-run the provision command ?

[chromeos-skunk-2.mtv.corp.google.com] out: HEAD is now at 11f11de... Add backports.shutil_which
[chromeos-skunk-2.mtv.corp.google.com] out: Error: Syntax error at 'user' at /root/chromeos-admin/puppet/modules/lab/manifests/core_cluster.pp:24:5 on node chromeos-skunk-2.mtv.corp.google.com
[chromeos-skunk-2.mtv.corp.google.com] out: Error: Syntax error at 'user' at /root/chromeos-admin/puppet/modules/lab/manifests/core_cluster.pp:24:5 on node chromeos-skunk-2.mtv.corp.google.com
[chromeos-skunk-2.mtv.corp.google.com] out: 


Fatal error: sudo() received nonzero return code 1 while executing!

Requested: HOME=/root/ FACTER_server_type="autotest-shard" /root/chromeos-admin/puppet/sync_and_run_puppet -f
Executed: sudo -S -p 'sudo password:'  /bin/bash -l -c "HOME=/root/ FACTER_server_type=\"autotest-shard\" /root/chromeos-admin/puppet/sync_and_run_puppet -f"


Comment 6 by nxia@chromium.org, Aug 10 2017

Blockedon: 754492
Cc: -nxia@chromium.org shuqianz@chromium.org
Owner: nxia@chromium.org

Comment 7 by nxia@chromium.org, Sep 18 2017

Blocking: 754036
Cc: -shuqianz@chromium.org

Comment 8 by nxia@chromium.org, Sep 18 2017

also need to remove /var/lib/dpkg/lock

Hit:7 https://rapture-prod.corp.google.com goobuntu-puppet-addons InRelease
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?

Sign in to add a comment