New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 814920 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 812935
Owner: ----
Closed: Feb 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Automated push to cautotest-prod is failing

Reported by jrbarnette@chromium.org, Feb 22 2018

Issue description

I just observed this command to fail:
    $ site_utils/deploy_server.py -x cros-full-0036.mtv.corp.google.com --force_update --skip-update

That command is essentially the same command used by
'site_utils/automated_deploy.py', meaning the errors
below can be expected to occur on the next push to prod.

So, this bug is blocking push...

Relevant error output follows:
======== cros-full-0036.mtv.corp.google.com ========
Running '/usr/local/autotest/site_utils/deploy_server_local.py --force_update --skip-update' on cros-full-0036.mtv.corp.google.com
scheduler stop/waiting
scheduler start/running, process 150528
host-scheduler stop/waiting
host-scheduler start/running, process 150787
rpc-logserver stop/waiting
rpc-logserver start/running, process 150810
 * Restarting web server apache2
[Thu Feb 22 13:49:52.821923 2018] [google_sso:emerg] [pid 150969:tid 140478507714432] (4)Interrupted system call: WARNING: Found obsolete directive 'RpMode'
[Thu Feb 22 13:49:52.821982 2018] [google_sso:emerg] [pid 150969:tid 140478507714432] (4)Interrupted system call: WARNING: Found obsolete directive 'TicketSigningKey'
[Thu Feb 22 13:49:52.822006 2018] [google_sso:emerg] [pid 150969:tid 140478507714432] (4)Interrupted system call: WARNING: Found obsolete directive 'RpMode'
[Thu Feb 22 13:49:52.822009 2018] [google_sso:emerg] [pid 150969:tid 140478507714432] (4)Interrupted system call: WARNING: Found obsolete directive 'TicketSigningKey'
   ...done.
job_aborter: unrecognized service
Will skip service check for pushing servers in prod.
Checking tree status:
Tree status: clean
Running update commands: afe, build_externals, tko
Running: afe: /usr/local/autotest/utils/compile_gwt_clients.py -c autotest.AfeClient
Running: build_externals: /usr/local/autotest/utils/build_externals.py
Running: tko: /usr/local/autotest/utils/compile_gwt_clients.py -c autotest.TkoClient
Restarting Services: scheduler, host-scheduler, rpc-logserver, apache2, job_aborter, sysmon
Restarting: scheduler
Restarting: host-scheduler
Restarting: rpc-logserver
Restarting: apache2
Restarting: job_aborter
Traceback (most recent call last):
  File "/usr/local/autotest/site_utils/deploy_server_local.py", line 538, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/usr/local/autotest/site_utils/deploy_server_local.py", line 530, in main
    use_chromite_master=behaviors.update_push_servers)
  File "/usr/local/autotest/site_utils/deploy_server_local.py", line 365, in run_deploy_actions
    skip_service_status=skip_service_status)
  File "/usr/local/autotest/site_utils/deploy_server_local.py", line 317, in restart_services
    restart_service(name)
  File "/usr/local/autotest/site_utils/deploy_server_local.py", line 274, in restart_service
    subprocess.check_call(cmd, stderr=subprocess.STDOUT)
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', 'service', 'job_aborter', 'restart']' returned non-zero exit status 1

The servers that failed were:
cros-full-0036.mtv.corp.google.com

 
$ ssh cros-full-0036.mtv.corp.google.com service status job_aborter
status: unrecognized service

IOW, our current config says restart a service named "job_aborter",
but the server isn't actually present on the system.

That is the only server in "atest server list" with the roles "host_scheduler,scheduler".

I would suspect that the problem is related to it's unique status.
Mergedinto: 812935
Status: Duplicate (was: Available)
https://chrome-internal-review.googlesource.com/c/chromeos/chromeos-admin/+/573852

Sign in to add a comment