New issue
Advanced search Search tips

Issue 837959 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Jul 27
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Ansible commands broken for devservers.

Project Member Reported by dgarr...@chromium.org, Apr 29 2018

Issue description

Attempting to force a puppet run via ansible is now broken for devservers.

chromeos$runlocalssh ansible-playbook -i bin/di sync_and_run_puppet.yml


SSO password for dgarrett: 
Please touch your security key.
Touch registered. Getting cookie.
LOAS2 cert for dgarrett expires 2018-04-29 14:12 (19h 59m from now)
LOAS cert for dgarrett expires 2018-04-29 14:12 (19h 59m from now)
Prod SSH cert for dgarrett expires 2018-04-29 14:12 (19h 59m from now)
Corp SSH cert for dgarrett expires 2018-04-29 14:12 (19h 59m from now)
[DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and make sure become_method is 'sudo' (default). This feature will
 be removed in version 2.6. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [Sync and run Puppet on all hosts] ****************************************************************************************************

TASK [Sync and run puppet] *****************************************************************************************************************
fatal: [android1758-infra-devserver5.cros.corp.google.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname android1758-infra-devserver5.cros.corp.google.com: No address associated with hostname\r\n", "unreachable": true}
fatal: [chromeos2-devserver5.cros.corp.google.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: key_load_public: invalid format\r\nWarning: Permanently added 'chromeos2-devserver5.cros.corp.google.com,100.115.245.197' (ECDSA) to the list of known hosts.\r\nLoad key \"/usr/local/google/home/dgarrett/.ssh/identity\": invalid format\r\nReceived disconnect from 100.115.245.197 port 22:2: Too many authentication failures for chromeos-test\r\nDisconnected from 100.115.245.197 port 22\r\n", "unreachable": true}

<many devserver connection failures>


PLAY RECAP *********************************************************************************************************************************
android1758-infra-devserver5.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos-golo-server-test.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-golo-server1.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-golo-server2.hot.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-golo-server3.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-golo-server4.hot.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-golo-server5.hot.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-gt-devserver11.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-gt-devserver12.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-gt-devserver13.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-gt-devserver14.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-gt-devserver15.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-gt-devserver16.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-gt-devserver17.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-gt-devserver18.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-gt-devserver19.hot.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-gt-devserver20.hot.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-gt-devserver21.hot.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-gt-devserver5.hot.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-gt-devserver6.hot.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-gt-devserver7.hot.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-server-tester1.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-server-tester2.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-server125.hot.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-server126.hot.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-server129.hot.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-server134.hot.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-server151.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-server155.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-server156.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-server158.cbf.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-skunk-1.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-skunk-2.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-skunk-3.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-skunk-4.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-skunk-5.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-staging-drone2.hot.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
chromeos-staging-master2.hot.corp.google.com : ok=0    changed=0    unreachable=0    failed=1   
chromeos-staging-shard2.cbf.corp.google.com : ok=0    changed=0    unreachable=0    failed=1   
chromeos1-dev-infra-devserver.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos1-dev-infra-devserver1.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos1-infra-devserver.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos1-infra-devserver4.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos1-infra-devserver5.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos1-infra-devserver7.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos15-infra-devserver1.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos15-infra-devserver2.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos15-infra-devserver3.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos15-infra-devserver4.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos15-infra-devserver5.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos2-devserver5.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos2-devserver8.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos2-devserver9.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos2-devservertest.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos3-infra-devserver.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos3-infra-devserver1.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos3-infra-devserver2.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos3-infra-devserver3.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos4-devserver1.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos4-devserver11.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos4-devserver2.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos4-devserver4.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos4-devserver5.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos4-devserver6.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos4-devserver7.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos4-devserver8.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos4-devserver9.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos6-devserver2.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos6-devserver3.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos6-devserver4.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos9-infra-devserver.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos9-infra-devserver1.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
chromeos9-infra-devserver7.cros.corp.google.com : ok=0    changed=0    unreachable=1    failed=0   
cros-bighd-0001.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-bighd-0002.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-bighd-0003.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0001.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0002.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0003.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0004.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0005.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0006.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0007.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0008.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0009.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0010.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0011.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0012.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0013.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0014.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0015.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0016.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0017.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0018.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0019.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0020.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0021.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0022.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0023.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0024.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0025.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0026.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0027.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0028.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0029.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0030.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0031.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0032.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0033.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0034.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0035.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0036.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0037.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0039.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   
cros-full-0040.mtv.corp.google.com : ok=1    changed=1    unreachable=0    failed=0   

 
Cc: -jkop@chromium.org
Status: Started (was: Untriaged)
Project Member

Comment 2 by bugdroid1@chromium.org, May 2 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/fce25f0073b51be4606bc4a8f46a28cbefc5dd18

commit fce25f0073b51be4606bc4a8f46a28cbefc5dd18
Author: Allen Li <ayatane@chromium.org>
Date: Wed May 02 22:08:11 2018

Status: Fixed (was: Started)

Sign in to add a comment