New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 804875 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Jan 2018
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug

Blocking:
issue 804442



Sign in to add a comment

gcloud sql instances patch tko --authorized-networks XXXX failed with 400

Project Member Reported by nxia@chromium.org, Jan 23 2018

Issue description

Fatal error: local() encountered an error (return code 1) while executing '/usr/local/autotest//site_utils/sync_cloudsql_access.py --project google.com:chromeos-lab --instance tko --afe cautotest --extra_servers 172.25.66.97'

Aborting.
2018-01-23 06:32:13,294 ERRO| Traceback (most recent call last):
  File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/tasks/task.py", line 66, in run
    self._run()
  File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/tasks/atomic_shard.py", line 279, in _run
    self._update_cloudsql_server_whitelist()
  File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/tasks/atomic_common.py", line 113, in decorated_func
    return func(self)
  File "/usr/local/google/home/nxia/chromiumos/chromeos-admin/venv/server_management_lib/tasks/atomic_common.py", line 430, in _update_cloudsql_server_whitelist
    api.local(command, capture=True)
  File "/usr/local/google/home/nxia/.cache/cros_venv/venv-2.7.6-95de6b4f9b30bb6fc148ee4eccd758dc/local/lib/python2.7/site-packages/fabric/operations.py", line 1237, in local
    error(message=msg, stdout=out, stderr=err)
  File "/usr/local/google/home/nxia/.cache/cros_venv/venv-2.7.6-95de6b4f9b30bb6fc148ee4eccd758dc/local/lib/python2.7/site-packages/fabric/utils.py", line 358, in error
    return func(message)
  File "/usr/local/google/home/nxia/.cache/cros_venv/venv-2.7.6-95de6b4f9b30bb6fc148ee4eccd758dc/local/lib/python2.7/site-packages/fabric/utils.py", line 54, in abort
    raise env.abort_exception(msg)
FabricException: local() encountered an error (return code 1) while executing '/usr/local/autotest//site_utils/sync_cloudsql_access.py --project google.com:chromeos-lab --instance tko --afe cautotest --extra_servers 172.25.66.97'






nxia@nxia:~/chromiumos/chromeos-admin$ /usr/local/autotest//site_utils/sync_cloudsql_access.py --project google.com:chromeos-lab --instance tko --afe cautotest --extra_servers 172.25.66.97



DEBUG:root:Running 'gcloud sql instances patch tko --authorized-networks 172.24.26.7,172.24.176.42,172.24.26.29,172.24.26.30,172.25.71.54,172.25.71.53,172.24.176.41,100.108.96.2,100.108.96.5,100.108.96.8,172.24.26.11,172.25.90.3,172.24.26.45,172.25.90.2,172.24.98.51,172.24.98.58,172.24.98.59,172.24.98.60,172.25.90.33,100.108.96.9,172.25.90.27,172.25.90.29,172.25.90.31,172.25.90.30,172.24.51.9,172.24.59.9,172.24.59.11,172.24.51.13,172.24.51.12,172.24.59.14,172.24.51.8,172.24.59.10,172.24.59.7,172.24.51.14,172.24.59.12,172.24.51.10,172.24.59.13,172.25.90.25,172.25.90.28,100.108.133.186,100.108.133.206,100.109.178.143,100.109.175.139,100.109.175.141,100.109.175.140,100.109.178.147,100.109.175.142,100.108.133.201,100.109.178.145,100.108.133.202,100.108.133.203,100.109.178.146,100.109.175.138,100.116.60.160,100.116.60.161,100.116.60.162,100.116.60.163,100.116.60.159,172.24.26.28,172.24.51.3,172.24.51.15,172.25.93.13,100.109.169.113,100.109.25.88,172.25.66.97 -q'
The following message will be used for the patch API method.
{"project": "google.com:chromeos-lab", "name": "tko", "settings": {"ipConfiguration": {"authorizedNetworks": [{"value": "172.24.26.7"}, {"value": "172.24.176.42"}, {"value": "172.24.26.29"}, {"value": "172.24.26.30"}, {"value": "172.25.71.54"}, {"value": "172.25.71.53"}, {"value": "172.24.176.41"}, {"value": "100.108.96.2"}, {"value": "100.108.96.5"}, {"value": "100.108.96.8"}, {"value": "172.24.26.11"}, {"value": "172.25.90.3"}, {"value": "172.24.26.45"}, {"value": "172.25.90.2"}, {"value": "172.24.98.51"}, {"value": "172.24.98.58"}, {"value": "172.24.98.59"}, {"value": "172.24.98.60"}, {"value": "172.25.90.33"}, {"value": "100.108.96.9"}, {"value": "172.25.90.27"}, {"value": "172.25.90.29"}, {"value": "172.25.90.31"}, {"value": "172.25.90.30"}, {"value": "172.24.51.9"}, {"value": "172.24.59.9"}, {"value": "172.24.59.11"}, {"value": "172.24.51.13"}, {"value": "172.24.51.12"}, {"value": "172.24.59.14"}, {"value": "172.24.51.8"}, {"value": "172.24.59.10"}, {"value": "172.24.59.7"}, {"value": "172.24.51.14"}, {"value": "172.24.59.12"}, {"value": "172.24.51.10"}, {"value": "172.24.59.13"}, {"value": "172.25.90.25"}, {"value": "172.25.90.28"}, {"value": "100.108.133.186"}, {"value": "100.108.133.206"}, {"value": "100.109.178.143"}, {"value": "100.109.175.139"}, {"value": "100.109.175.141"}, {"value": "100.109.175.140"}, {"value": "100.109.178.147"}, {"value": "100.109.175.142"}, {"value": "100.108.133.201"}, {"value": "100.109.178.145"}, {"value": "100.108.133.202"}, {"value": "100.108.133.203"}, {"value": "100.109.178.146"}, {"value": "100.109.175.138"}, {"value": "100.116.60.160"}, {"value": "100.116.60.161"}, {"value": "100.116.60.162"}, {"value": "100.116.60.163"}, {"value": "100.116.60.159"}, {"value": "172.24.26.28"}, {"value": "172.24.51.3"}, {"value": "172.24.51.15"}, {"value": "172.25.93.13"}, {"value": "100.109.169.113"}, {"value": "100.109.25.88"}, {"value": "172.25.66.97"}]}, "databaseFlags": [{"name": "innodb_file_per_table", "value": "on"}, {"name": "slow_query_log", "value": "on"}, {"name": "long_query_time", "value": "90"}, {"name": "wait_timeout", "value": "300"}]}}
ERROR: (gcloud.sql.instances.patch) HTTPError 400: Invalid parameter: Backup specification (every day every tuesday 09:00).
Traceback (most recent call last):
  File "/usr/local/autotest//site_utils/sync_cloudsql_access.py", line 120, in <module>
    main()
  File "/usr/local/autotest//site_utils/sync_cloudsql_access.py", line 116, in main
    options.extra_servers)
  File "/usr/local/autotest//site_utils/sync_cloudsql_access.py", line 88, in update_allowed_networks
    utils.run(cmd, stdout_tee=sys.stdout, stderr_tee=sys.stderr)
  File "/usr/local/autotest/client/common_lib/utils.py", line 748, in run
    "Command returned non-zero exit status")


 

Comment 1 by nxia@chromium.org, Jan 23 2018

Checked the tko db, it's creating the backups

"
Creating backup. This may take a few minutes. While this operation is running, you may continue to view information about the instance.
"

Comment 2 by nxia@chromium.org, Jan 23 2018

created a bug at https://b.corp.google.com/issues/72371839

Comment 3 by nxia@chromium.org, Jan 23 2018

The backup has completed, the ip still can't be authorized. I also can't add the ip via the pantheon UI. I guess the ip can't be recognized by the cloud sql tko

chromeos-server-tester2.mtv.corp.google.com
100.109.25.88

Comment 4 Deleted

Comment 5 by nxia@chromium.org, Jan 23 2018

same failure for chromeos-server-tester1.mtv.corp.google.com 100.109.25.87

Comment 6 by nxia@chromium.org, Jan 30 2018

Blocking: 804442

Comment 7 by nxia@chromium.org, Jan 30 2018

resolved by enabling db auto backup 

Comment 8 by nxia@chromium.org, Jan 30 2018

Status: Fixed (was: Untriaged)

Sign in to add a comment