New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 750986 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Last visit > 30 days ago
Closed: Aug 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

kill_slow_queries sends out repeated email spam when it crashes

Project Member Reported by pho...@chromium.org, Aug 1 2017

Issue description

It will send out a flurry of emails like:

"""
Error occurred during killing slow db queries:
(2002, "Can't connect to local MySQL server through socket
'/var/run/mysqld/mysqld.sock' (2)")
Detailed logs can be found in /var/log/slow_queries.log on db backup server.
To avoid db crash, please check ASAP.
"""

We can avoid this by using a retrying/reconnecting wrapper around the MySQL connection/cursor object, like how autotest_lib.site_utils.stats.mysql_stats does.
 
Project Member

Comment 1 by bugdroid1@chromium.org, Aug 4 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/eb55c9522f076cd402d12e8463b99a03e7502b57

commit eb55c9522f076cd402d12e8463b99a03e7502b57
Author: Paul Hobbs <phobbs@google.com>
Date: Fri Aug 04 00:10:59 2017

mysql_stats: Make RetryingConnection more reusable

BUG= chromium:750986 
TEST=unit tests pass.

Change-Id: Icae91be2edb7cc1c1152cf97c72e96fde502405d
Reviewed-on: https://chromium-review.googlesource.com/595117
Commit-Ready: Paul Hobbs <phobbs@google.com>
Tested-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Shuqian Zhao <shuqianz@chromium.org>

[modify] https://crrev.com/eb55c9522f076cd402d12e8463b99a03e7502b57/site_utils/stats/mysql_stats_unittest.py
[modify] https://crrev.com/eb55c9522f076cd402d12e8463b99a03e7502b57/site_utils/stats/mysql_stats.py

Project Member

Comment 2 by bugdroid1@chromium.org, Aug 4 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/b87be557886f7490a480a2b6b805a26886fb034c

commit b87be557886f7490a480a2b6b805a26886fb034c
Author: Paul Hobbs <phobbs@google.com>
Date: Fri Aug 04 00:10:59 2017

kill_slow_queries: Retry connections

Use mysql_stats' RetryingConnection class to try reconnecting when queries fail.

BUG= chromium:750986 
TEST=None

Change-Id: If3a68b77c0f0cb8aa55266f61057df96d81dfdd4
Reviewed-on: https://chromium-review.googlesource.com/595118
Commit-Ready: Paul Hobbs <phobbs@google.com>
Tested-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Shuqian Zhao <shuqianz@chromium.org>

[modify] https://crrev.com/b87be557886f7490a480a2b6b805a26886fb034c/site_utils/kill_slow_queries.py
[modify] https://crrev.com/b87be557886f7490a480a2b6b805a26886fb034c/site_utils/stats/mysql_stats.py

Status: Fixed (was: Started)
Cc: akes...@chromium.org
Status: Assigned (was: Fixed)
Paul, I think these changes introduce a bug:

Error occurred during killing slow db queries:
'RetryingConnection' object has no attribute 'fetchall'
Detailed logs can be found in /var/log/slow_queries.log on db backup server.
To avoid db crash, please check ASAP.


Can you take a look?
Hm, yeah. I'll upload a fix.
Project Member

Comment 6 by bugdroid1@chromium.org, Aug 4 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/09ea0e5bb2a4384b2713d47f0c73b9dc37c204f3

commit 09ea0e5bb2a4384b2713d47f0c73b9dc37c204f3
Author: Paul Hobbs <phobbs@google.com>
Date: Fri Aug 04 22:13:54 2017

kill_slow_queries: Fix usage of RetryingConnection

BUG= chromium:750986 
TEST=ran ./kill_slow_queries.py locally.

Change-Id: I66252f994d52f4283eac474c8a96e4b452a822e3
Reviewed-on: https://chromium-review.googlesource.com/602872
Reviewed-by: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Paul Hobbs <phobbs@google.com>

[modify] https://crrev.com/09ea0e5bb2a4384b2713d47f0c73b9dc37c204f3/site_utils/kill_slow_queries.py
[modify] https://crrev.com/09ea0e5bb2a4384b2713d47f0c73b9dc37c204f3/site_utils/stats/mysql_stats.py

Fix has been chumped. I checked chromeos-server25.mtv, and lucky for us the broken CL hadn't been pushed there yet. What is the hostname of the backup server?
Status: Fixed (was: Assigned)

Comment 9 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Sign in to add a comment