restarting mysql causes mysql metrics to stop emitting |
||
Issue descriptione.g. https://viceroy.corp.google.com/chromeos/afe_db?duration=1h&host_name=chromeos-server102&utc_end=1510095778.92 My guess it the mysql metrics daemon is crashing when I restart mysql, and upstart isn't bringing it back to life. This was on chromeos-server102. I also did the same manual mysql restart on 98
,
Mar 29 2018
|
||
►
Sign in to add a comment |
||
Comment 1 by pho...@chromium.org
, Nov 7 2017We already do retries on the MySQL connection - looking at the log, it turns out this is caused by the metrics not being reset: ERROR:root:Caught an exception while running ('CounterMetric',).('set',)((0,)) Traceback (most recent call last): File "/usr/local/autotest/site-packages/chromite/lib/ts_mon_config.py", line 276, in _CallMetric **message.method_kwargs) File "/usr/local/autotest/site-packages/chromite/third_party/infra_libs/ts_mon/common/metrics.py", line 372, in set self._set(fields, target_fields, value, enforce_ge=True) File "/usr/local/autotest/site-packages/chromite/third_party/infra_libs/ts_mon/common/metrics.py", line 300, in _set value, enforce_ge=enforce_ge) File "/usr/local/autotest/site-packages/chromite/third_party/infra_libs/ts_mon/common/metric_store.py", line 209, in set raise errors.MonitoringDecreasingValueError(name, old_value, value) MonitoringDecreasingValueError: Monotonically increasing metric "chromeos/autotest/afe_db/slow_queries" was given value "0", which is not greater than or equal to "781".