kill_slow_queries sends out repeated email spam when it crashes |
|||||
Issue descriptionIt will send out a flurry of emails like: """ Error occurred during killing slow db queries: (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)") Detailed logs can be found in /var/log/slow_queries.log on db backup server. To avoid db crash, please check ASAP. """ We can avoid this by using a retrying/reconnecting wrapper around the MySQL connection/cursor object, like how autotest_lib.site_utils.stats.mysql_stats does.
,
Aug 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/b87be557886f7490a480a2b6b805a26886fb034c commit b87be557886f7490a480a2b6b805a26886fb034c Author: Paul Hobbs <phobbs@google.com> Date: Fri Aug 04 00:10:59 2017 kill_slow_queries: Retry connections Use mysql_stats' RetryingConnection class to try reconnecting when queries fail. BUG= chromium:750986 TEST=None Change-Id: If3a68b77c0f0cb8aa55266f61057df96d81dfdd4 Reviewed-on: https://chromium-review.googlesource.com/595118 Commit-Ready: Paul Hobbs <phobbs@google.com> Tested-by: Paul Hobbs <phobbs@google.com> Reviewed-by: Shuqian Zhao <shuqianz@chromium.org> [modify] https://crrev.com/b87be557886f7490a480a2b6b805a26886fb034c/site_utils/kill_slow_queries.py [modify] https://crrev.com/b87be557886f7490a480a2b6b805a26886fb034c/site_utils/stats/mysql_stats.py
,
Aug 4 2017
,
Aug 4 2017
Paul, I think these changes introduce a bug: Error occurred during killing slow db queries: 'RetryingConnection' object has no attribute 'fetchall' Detailed logs can be found in /var/log/slow_queries.log on db backup server. To avoid db crash, please check ASAP. Can you take a look?
,
Aug 4 2017
Hm, yeah. I'll upload a fix.
,
Aug 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/09ea0e5bb2a4384b2713d47f0c73b9dc37c204f3 commit 09ea0e5bb2a4384b2713d47f0c73b9dc37c204f3 Author: Paul Hobbs <phobbs@google.com> Date: Fri Aug 04 22:13:54 2017 kill_slow_queries: Fix usage of RetryingConnection BUG= chromium:750986 TEST=ran ./kill_slow_queries.py locally. Change-Id: I66252f994d52f4283eac474c8a96e4b452a822e3 Reviewed-on: https://chromium-review.googlesource.com/602872 Reviewed-by: Shuqian Zhao <shuqianz@chromium.org> Tested-by: Paul Hobbs <phobbs@google.com> [modify] https://crrev.com/09ea0e5bb2a4384b2713d47f0c73b9dc37c204f3/site_utils/kill_slow_queries.py [modify] https://crrev.com/09ea0e5bb2a4384b2713d47f0c73b9dc37c204f3/site_utils/stats/mysql_stats.py
,
Aug 4 2017
Fix has been chumped. I checked chromeos-server25.mtv, and lucky for us the broken CL hadn't been pushed there yet. What is the hostname of the backup server?
,
Aug 9 2017
,
Jan 22 2018
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by bugdroid1@chromium.org
, Aug 4 2017