Issue metadata
Sign in to add a comment
|
Automated push to cautotest-prod is failing
Reported by
jrbarnette@chromium.org,
Feb 22 2018
|
||||||||||||||||||||||||
Issue description
I just observed this command to fail:
$ site_utils/deploy_server.py -x cros-full-0036.mtv.corp.google.com --force_update --skip-update
That command is essentially the same command used by
'site_utils/automated_deploy.py', meaning the errors
below can be expected to occur on the next push to prod.
So, this bug is blocking push...
Relevant error output follows:
======== cros-full-0036.mtv.corp.google.com ========
Running '/usr/local/autotest/site_utils/deploy_server_local.py --force_update --skip-update' on cros-full-0036.mtv.corp.google.com
scheduler stop/waiting
scheduler start/running, process 150528
host-scheduler stop/waiting
host-scheduler start/running, process 150787
rpc-logserver stop/waiting
rpc-logserver start/running, process 150810
* Restarting web server apache2
[Thu Feb 22 13:49:52.821923 2018] [google_sso:emerg] [pid 150969:tid 140478507714432] (4)Interrupted system call: WARNING: Found obsolete directive 'RpMode'
[Thu Feb 22 13:49:52.821982 2018] [google_sso:emerg] [pid 150969:tid 140478507714432] (4)Interrupted system call: WARNING: Found obsolete directive 'TicketSigningKey'
[Thu Feb 22 13:49:52.822006 2018] [google_sso:emerg] [pid 150969:tid 140478507714432] (4)Interrupted system call: WARNING: Found obsolete directive 'RpMode'
[Thu Feb 22 13:49:52.822009 2018] [google_sso:emerg] [pid 150969:tid 140478507714432] (4)Interrupted system call: WARNING: Found obsolete directive 'TicketSigningKey'
...done.
job_aborter: unrecognized service
Will skip service check for pushing servers in prod.
Checking tree status:
Tree status: clean
Running update commands: afe, build_externals, tko
Running: afe: /usr/local/autotest/utils/compile_gwt_clients.py -c autotest.AfeClient
Running: build_externals: /usr/local/autotest/utils/build_externals.py
Running: tko: /usr/local/autotest/utils/compile_gwt_clients.py -c autotest.TkoClient
Restarting Services: scheduler, host-scheduler, rpc-logserver, apache2, job_aborter, sysmon
Restarting: scheduler
Restarting: host-scheduler
Restarting: rpc-logserver
Restarting: apache2
Restarting: job_aborter
Traceback (most recent call last):
File "/usr/local/autotest/site_utils/deploy_server_local.py", line 538, in <module>
sys.exit(main(sys.argv[1:]))
File "/usr/local/autotest/site_utils/deploy_server_local.py", line 530, in main
use_chromite_master=behaviors.update_push_servers)
File "/usr/local/autotest/site_utils/deploy_server_local.py", line 365, in run_deploy_actions
skip_service_status=skip_service_status)
File "/usr/local/autotest/site_utils/deploy_server_local.py", line 317, in restart_services
restart_service(name)
File "/usr/local/autotest/site_utils/deploy_server_local.py", line 274, in restart_service
subprocess.check_call(cmd, stderr=subprocess.STDOUT)
File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', 'service', 'job_aborter', 'restart']' returned non-zero exit status 1
The servers that failed were:
cros-full-0036.mtv.corp.google.com
,
Feb 22 2018
That is the only server in "atest server list" with the roles "host_scheduler,scheduler". I would suspect that the problem is related to it's unique status.
,
Feb 22 2018
https://chrome-internal-review.googlesource.com/c/chromeos/chromeos-admin/+/573852 |
|||||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||||
Comment 1 by jrbarnette@chromium.org
, Feb 22 2018