New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 712388 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Last visit > 30 days ago
Closed: May 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Add metrics for number of Apache error log messages

Project Member Reported by pho...@chromium.org, Apr 17 2017

Issue description

This would be useful if there is a spike in them, and to try to keep them clean. Right now there are a bunch of noisey messages in there. After we clean those up, the only way to keep it clean is to increase the visibility of those messages.
 
I have CLs in flight to clean up the top 3 causes of log spam (sigalrm on wsgi illegal, 2 django deprecations) so let's not delay this. 

Comment 2 by pho...@chromium.org, Apr 18 2017

Status: Started (was: Assigned)
I'm working on this now
Project Member

Comment 3 by bugdroid1@chromium.org, Apr 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/d14cb7e71c558f5faba29e126817cabdc577ff96

commit d14cb7e71c558f5faba29e126817cabdc577ff96
Author: Paul Hobbs <phobbs@google.com>
Date: Fri Apr 21 17:24:19 2017

Added an apache error log metrics script.

The script just emits a count of how many lines it has seen, along with whether
the log message was an error or warning, and whether it was emitted from
within the mod_wsgi module.

BUG= chromium:712388 
TEST=added unit tests, and manual testing.

Change-Id: I8a2304ce4676364117b11f4560d6cad8b2fc2aa2
Reviewed-on: https://chromium-review.googlesource.com/480992
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Trybot-Ready: Paul Hobbs <phobbs@google.com>
Commit-Queue: Paul Hobbs <phobbs@google.com>
Tested-by: Paul Hobbs <phobbs@google.com>

[add] https://crrev.com/d14cb7e71c558f5faba29e126817cabdc577ff96/site_utils/stats/apache_error_log_example.txt
[add] https://crrev.com/d14cb7e71c558f5faba29e126817cabdc577ff96/site_utils/stats/apache_error_stats.py
[add] https://crrev.com/d14cb7e71c558f5faba29e126817cabdc577ff96/site_utils/stats/apache_error_stats_unittest.py

Project Member

Comment 4 by bugdroid1@chromium.org, Apr 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/d14cb7e71c558f5faba29e126817cabdc577ff96

commit d14cb7e71c558f5faba29e126817cabdc577ff96
Author: Paul Hobbs <phobbs@google.com>
Date: Fri Apr 21 17:24:19 2017

Added an apache error log metrics script.

The script just emits a count of how many lines it has seen, along with whether
the log message was an error or warning, and whether it was emitted from
within the mod_wsgi module.

BUG= chromium:712388 
TEST=added unit tests, and manual testing.

Change-Id: I8a2304ce4676364117b11f4560d6cad8b2fc2aa2
Reviewed-on: https://chromium-review.googlesource.com/480992
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Trybot-Ready: Paul Hobbs <phobbs@google.com>
Commit-Queue: Paul Hobbs <phobbs@google.com>
Tested-by: Paul Hobbs <phobbs@google.com>

[add] https://crrev.com/d14cb7e71c558f5faba29e126817cabdc577ff96/site_utils/stats/apache_error_log_example.txt
[add] https://crrev.com/d14cb7e71c558f5faba29e126817cabdc577ff96/site_utils/stats/apache_error_stats.py
[add] https://crrev.com/d14cb7e71c558f5faba29e126817cabdc577ff96/site_utils/stats/apache_error_stats_unittest.py

Project Member

Comment 5 by bugdroid1@chromium.org, Apr 26 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/7e0d7ed2cbd69738a96c9d72e0e30521a0021d28

commit 7e0d7ed2cbd69738a96c9d72e0e30521a0021d28
Author: Paul Hobbs <phobbs@google.com>
Date: Wed Apr 26 01:40:03 2017

apache_error_stats: Fix to constant and logging

- Fix a bug in the PATTERNS constant, which should be a list of tuples
  instead of a dictionary.
- Change the matching loop to only log that a message is being emitted
  if the .match result is not None.

BUG= chromium:712388 
TEST=None

Change-Id: Ic9eb0ddb025006cc89c753f1cc576919aee358c9
Reviewed-on: https://chromium-review.googlesource.com/486121
Commit-Ready: Paul Hobbs <phobbs@google.com>
Tested-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/7e0d7ed2cbd69738a96c9d72e0e30521a0021d28/site_utils/stats/apache_error_stats.py

Project Member

Comment 6 by bugdroid1@chromium.org, Apr 26 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/7c2895cb0c16fbdb2d72c187cec9202117d362aa

commit 7c2895cb0c16fbdb2d72c187cec9202117d362aa
Author: Paul Hobbs <phobbs@google.com>
Date: Wed Apr 26 18:26:00 2017

Add "tail_until_writer_finished" script

Copied a script from chromiumos/src/platform/dev which is useful for
tailing log files which will be rotated. This is necessary for the
apache_error_stats script.

BUG= chromium:712388 
TEST=Tests pass, and are properly skipped if "inotifywait" is missing.

Change-Id: I56047cf3bd4eaa48346c51d08f7441378e251f6b
Reviewed-on: https://chromium-review.googlesource.com/486122
Commit-Ready: Paul Hobbs <phobbs@google.com>
Tested-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Shuqian Zhao <shuqianz@chromium.org>

[add] https://crrev.com/7c2895cb0c16fbdb2d72c187cec9202117d362aa/site_utils/stats/tail_until_writer_finished.py
[add] https://crrev.com/7c2895cb0c16fbdb2d72c187cec9202117d362aa/site_utils/stats/tail_until_writer_finished_unittest.py

Project Member

Comment 7 by bugdroid1@chromium.org, Apr 28 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/e0279f226ec6618f3ef6381a7f00cf81fdb50fdc

commit e0279f226ec6618f3ef6381a7f00cf81fdb50fdc
Author: Paul Hobbs <phobbs@google.com>
Date: Fri Apr 28 23:13:21 2017

Project Member

Comment 8 by bugdroid1@chromium.org, May 3 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/a0b3ecc53a258f5cf2ebb7cf72ea39a58354a1ce

commit a0b3ecc53a258f5cf2ebb7cf72ea39a58354a1ce
Author: Paul Hobbs <phobbs@google.com>
Date: Wed May 03 20:32:03 2017

Project Member

Comment 9 by bugdroid1@chromium.org, May 3 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/a0b3ecc53a258f5cf2ebb7cf72ea39a58354a1ce

commit a0b3ecc53a258f5cf2ebb7cf72ea39a58354a1ce
Author: Paul Hobbs <phobbs@google.com>
Date: Wed May 03 20:32:03 2017

Project Member

Comment 10 by bugdroid1@chromium.org, May 4 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/447901ccd555f575c6301b5886e626478eb30542

commit 447901ccd555f575c6301b5886e626478eb30542
Author: Paul Hobbs <phobbs@google.com>
Date: Thu May 04 17:10:28 2017

tail_until_writer_finished: Remove "-u" argument

Passing flags to python doesn't work when python is invoked as "/usr/bin/env
python", so remove the flag. Added a sys.stdout.flush() in each iteration of the
main loop to replicate the unbuffered behavior enabled by "python -u"

BUG= chromium:712388 
TEST=./tail_until_writer_finished succeeds.

Change-Id: I1f1d4ad992f3156df90d22295a453832dff6492d
Reviewed-on: https://chromium-review.googlesource.com/495572
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Paul Hobbs <phobbs@google.com>

[modify] https://crrev.com/447901ccd555f575c6301b5886e626478eb30542/site_utils/stats/tail_until_writer_finished.py

Project Member

Comment 11 by bugdroid1@chromium.org, May 5 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/a8b3df5216f419e6e2faac66d2b541b23222fadc

commit a8b3df5216f419e6e2faac66d2b541b23222fadc
Author: Paul Hobbs <phobbs@google.com>
Date: Fri May 05 19:14:22 2017

Project Member

Comment 12 by bugdroid1@chromium.org, May 5 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/a8b3df5216f419e6e2faac66d2b541b23222fadc

commit a8b3df5216f419e6e2faac66d2b541b23222fadc
Author: Paul Hobbs <phobbs@google.com>
Date: Fri May 05 19:14:22 2017

Project Member

Comment 13 by bugdroid1@chromium.org, May 9 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/c527908c8ea7a5884bdda4b781eefd9d709b1d3f

commit c527908c8ea7a5884bdda4b781eefd9d709b1d3f
Author: Paul Hobbs <phobbs@google.com>
Date: Tue May 09 02:16:43 2017

Project Member

Comment 14 by bugdroid1@chromium.org, May 13 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/5b0c8bcb92665f8eeafb0013994d81eb6c9ea879

commit 5b0c8bcb92665f8eeafb0013994d81eb6c9ea879
Author: Paul Hobbs <phobbs@google.com>
Date: Sat May 13 03:57:58 2017

[autotest] apache_error_stats: Make regexp more generic

Some AFE servers have a "[remote $IP_ADDRESS]" section after the pid section.
Others have a "tid" entry in the pid group (like "[pid 123456:tid 12345]").
Adjust the error log matcher regular expression to account for these cases.

BUG= chromium:712388 
TEST=new unit tests

Change-Id: I7dbcf926d0aae9ee42c8b5955198e5a6f1876bd0
Reviewed-on: https://chromium-review.googlesource.com/500993
Commit-Ready: Paul Hobbs <phobbs@google.com>
Tested-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Paul Hobbs <phobbs@google.com>

[modify] https://crrev.com/5b0c8bcb92665f8eeafb0013994d81eb6c9ea879/site_utils/stats/apache_error_log_example.txt
[modify] https://crrev.com/5b0c8bcb92665f8eeafb0013994d81eb6c9ea879/site_utils/stats/apache_error_stats.py
[modify] https://crrev.com/5b0c8bcb92665f8eeafb0013994d81eb6c9ea879/site_utils/stats/apache_error_stats_unittest.py

Project Member

Comment 15 by bugdroid1@chromium.org, May 13 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/5c2ced33986dd7682e2f136e20ee3958b90739ca

commit 5c2ced33986dd7682e2f136e20ee3958b90739ca
Author: Paul Hobbs <phobbs@google.com>
Date: Sat May 13 03:57:59 2017

[autotest] apache_error_log_metrics: Add matcher for all lines

Add a default regexp which matches all log lines. This is useful for servers
which don't match any of the lines.

BUG= chromium:712388 
TEST=unit tests still pass

Change-Id: Id86ebbfa6f2db02b7cc99b4d7715958fa4d156e5
Reviewed-on: https://chromium-review.googlesource.com/500995
Commit-Ready: Paul Hobbs <phobbs@google.com>
Tested-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Paul Hobbs <phobbs@google.com>

[modify] https://crrev.com/5c2ced33986dd7682e2f136e20ee3958b90739ca/site_utils/stats/apache_error_log_metrics.py

Project Member

Comment 16 by bugdroid1@chromium.org, May 17 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/ec25c053d3ef57ee6504485099d58737427093c1

commit ec25c053d3ef57ee6504485099d58737427093c1
Author: Paul Hobbs <phobbs@google.com>
Date: Wed May 17 03:56:50 2017

Status: Fixed (was: Started)
Labels: VerifyIn-61

Comment 19 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Sign in to add a comment