New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 811959 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Feb 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocking:
issue 810584



Sign in to add a comment

Can't provision DB servers.

Project Member Reported by dgarr...@chromium.org, Feb 13 2018

Issue description

This command:

bin/run_server_task DatabaseProvisionTask --host_server cros-bighd-0001.mtv.corp.google.com


Fails with this puppet error:

[cros-bighd-0001.mtv.corp.google.com] sudo: HOME=/root/ FACTER_server_type="autotest-db" /root/chromeos-admin/puppet/sync_and_run_puppet -f
[cros-bighd-0001.mtv.corp.google.com] out: HEAD is now at 500052d Fix virtualenv bug when VIRTUALENV_ALWAYS_COPY is set
[cros-bighd-0001.mtv.corp.google.com] out: puppet/run_puppet: line 26: GEM: readonly variable
[cros-bighd-0001.mtv.corp.google.com] out: /root/chromeos-admin//puppet/bin/ensure_gem_deps: GEM=gem
[cros-bighd-0001.mtv.corp.google.com] out: /root/chromeos-admin//puppet/bin/ensure_gem_deps: GEM_HOME not set; installing for user
[cros-bighd-0001.mtv.corp.google.com] out: Error: Could not find data item classes in any Hiera data file and no default supplied at /root/chromeos-admin/puppet/manifests/site.pp:3 on node cros-bighd-0001.mtv.corp.google.com
[cros-bighd-0001.mtv.corp.google.com] out: Error: Could not find data item classes in any Hiera data file and no default supplied at /root/chromeos-admin/puppet/manifests/site.pp:3 on node cros-bighd-0001.mtv.corp.google.com
[cros-bighd-0001.mtv.corp.google.com] out: 


Fatal error: sudo() received nonzero return code 1 while executing!

Requested: HOME=/root/ FACTER_server_type="autotest-db" /root/chromeos-admin/puppet/sync_and_run_puppet -f
Executed: sudo -S -p 'sudo password:'  /bin/bash -l -c "HOME=/root/ FACTER_server_type=\"autotest-db\" /root/chromeos-admin/puppet/sync_and_run_puppet -f"

Aborting.
2018-02-13 12:42:32,812 ERRO| Traceback (most recent call last):
  File "/usr/local/google/home/dgarrett/sand/chromeos/chromeos-admin/venv/server_management_lib/tasks/task.py", line 66, in run
    self._run()
  File "/usr/local/google/home/dgarrett/sand/chromeos/chromeos-admin/venv/server_management_lib/tasks/atomic_common.py", line 203, in _run
    self._deploy_puppet()
  File "/usr/local/google/home/dgarrett/sand/chromeos/chromeos-admin/venv/server_management_lib/tasks/atomic_common.py", line 94, in decorated_func
    return func(self)
  File "/usr/local/google/home/dgarrett/sand/chromeos/chromeos-admin/venv/server_management_lib/tasks/atomic_common.py", line 178, in _deploy_puppet
    fabfile.deploy_puppet(server_type=self.server_type)
  File "/usr/local/google/home/dgarrett/sand/chromeos/chromeos-admin/venv/server_management_lib/fabfile.py", line 177, in deploy_puppet
    _run_puppet(server_type=server_type)
  File "/usr/local/google/home/dgarrett/sand/chromeos/chromeos-admin/venv/server_management_lib/fabfile.py", line 211, in _run_puppet
    sudo('HOME=/root/ FACTER_server_type="%s" %s' % (server_type, puppet_cmd))
  File "/usr/local/google/home/dgarrett/.cache/cros_venv/venv-2.7.13-95de6b4f9b30bb6fc148ee4eccd758dc/local/lib/python2.7/site-packages/fabric/network.py", line 677, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/usr/local/google/home/dgarrett/.cache/cros_venv/venv-2.7.13-95de6b4f9b30bb6fc148ee4eccd758dc/local/lib/python2.7/site-packages/fabric/operations.py", line 1146, in sudo
    capture_buffer_size=capture_buffer_size,
  File "/usr/local/google/home/dgarrett/.cache/cros_venv/venv-2.7.13-95de6b4f9b30bb6fc148ee4eccd758dc/local/lib/python2.7/site-packages/fabric/operations.py", line 952, in _run_command
    error(message=msg, stdout=out, stderr=err)
  File "/usr/local/google/home/dgarrett/.cache/cros_venv/venv-2.7.13-95de6b4f9b30bb6fc148ee4eccd758dc/local/lib/python2.7/site-packages/fabric/utils.py", line 358, in error
    return func(message)
  File "/usr/local/google/home/dgarrett/.cache/cros_venv/venv-2.7.13-95de6b4f9b30bb6fc148ee4eccd758dc/local/lib/python2.7/site-packages/fabric/utils.py", line 54, in abort
    raise env.abort_exception(msg)
FabricException: sudo() received nonzero return code 1 while executing!

Requested: HOME=/root/ FACTER_server_type="autotest-db" /root/chromeos-admin/puppet/sync_and_run_puppet -f
Executed: sudo -S -p 'sudo password:'  /bin/bash -l -c "HOME=/root/ FACTER_server_type=\"autotest-db\" /root/chromeos-admin/puppet/sync_and_run_puppet -f"

2018-02-13 12:42:32,812 ERRO| sudo() received nonzero return code 1 while executing!

Requested: HOME=/root/ FACTER_server_type="autotest-db" /root/chromeos-admin/puppet/sync_and_run_puppet -f
Executed: sudo -S -p 'sudo password:'  /bin/bash -l -c "HOME=/root/ FACTER_server_type=\"autotest-db\" /root/chromeos-admin/puppet/sync_and_run_puppet -f"
2018-02-13 12:42:32,813 INFO| Printing out task report.
{
  "sub_reports": [],
  "exception": "FabricException('sudo() received nonzero return code 1 while executing!\\n\\nRequested: HOME=/root/ FACTER_server_type=\"autotest-db\" /root/chromeos-admin/puppet/sync_and_run_puppet -f\\nExecuted: sudo -S -p \\'sudo password:\\'  /bin/bash -l -c \"HOME=/root/ FACTER_server_type=\\\\\"autotest-db\\\\\" /root/chromeos-admin/puppet/sync_and_run_puppet -f\"',)",
  "is_successful": false,
  "description": "sudo() received nonzero return code 1 while executing!\n\nRequested: HOME=/root/ FACTER_server_type=\"autotest-db\" /root/chromeos-admin/puppet/sync_and_run_puppet -f\nExecuted: sudo -S -p 'sudo password:'  /bin/bash -l -c \"HOME=/root/ FACTER_server_type=\\\"autotest-db\\\" /root/chromeos-admin/puppet/sync_and_run_puppet -f\"",
  "arguments_used": {
    "host_server": "cros-bighd-0001.mtv.corp.google.com"
  },
  "task_name": "DatabaseProvisionTask"
}

 
Blocking: 810584
As Allen pointed out:

admin/venv/server_management_lib/tasks/atomic_database.py

has a hardcoded role of "autotest-db", which isn't valid.

I'm working around this by locally hard coding to "autotest-db-slave-secondary" which matches the role listed in puppet.

Then leaving this bug around, since pulling the real role from autotest seems long and complicated.
Cc: -shuqianz@chromium.org ayatane@chromium.org
Owner: shuqianz@chromium.org
Project Member

Comment 4 by bugdroid1@chromium.org, Feb 13 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/c2ca631ecf71e95d73bfdd0cd1a661e36d0bd00e

commit c2ca631ecf71e95d73bfdd0cd1a661e36d0bd00e
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Tue Feb 13 22:10:23 2018

Status: Fixed (was: Assigned)

Sign in to add a comment