Huge log files on devservers: devserver_apache_access_log_metrics.log, apache_log_metrics.log |
||||||
Issue descriptionchromeos-test@chromeos2-devserver6:/var/log/devserver$ hostname chromeos2-devserver6 chromeos-test@chromeos2-devserver6:/var/log/devserver$ ls -lh devserver_apache_access_log_metrics.log -rw-r--r-- 1 root root 20G Mar 8 18:05 devserver_apache_access_log_metrics.log Another instance: chromeos-test@chromeos4-devserver1:/var/log/devserver$ hostname chromeos4-devserver1 chromeos-test@chromeos4-devserver1:/var/log/devserver$ ls -lh devserver_apache_access_log_metrics.log -rw-r--r-- 1 root root 17G Mar 8 18:06 devserver_apache_access_log_metrics.log The reason is due to permission. These log files are non-writable for non-root users, so our script cannot rotate them.
,
Mar 9 2018
apache_log_metrics.log is also very huge. It should be rotated, but seems be broken by something wrong.
chromeos-test@chromeos4-devserver1:/var/log/devserver$ ls -lh *.log
-rw-r--r-- 1 chromeos-test chromeos-test 6.8G May 26 2017 apache_log_metrics.log
-rw-r--r-- 1 root root 288M Mar 8 18:18 console.log
-rw-r--r-- 1 root root 17G Mar 8 18:18 devserver_apache_access_log_metrics.log
-rw-r--r-- 1 chromeos-test chromeos-test 6.6M Mar 8 18:18 server.log
-rw-r--r-- 1 root root 8.0K Mar 8 17:28 workqueue-console.log
-rw-r--r-- 1 chromeos-test chromeos-test 51K Mar 8 18:11 workqueue.log
chromeos-test@chromeos4-devserver1:/var/log/devserver$ tail apache_log_metrics.log
ReadChunks(fh)
File "/home/chromeos-test/chromiumos/src/platform/dev/tail_until_writer_finished.py", line 53, in ReadChunks
print(chunk, end='', file=outfile)
IOError: [Errno 32] Broken pipe
Fri May 26 14:49:45 PDT 2017 upstart_apache_log_metrics: starting.
Fri May 26 14:49:45 PDT 2017 apache_log_metrics: started.
Traceback (most recent call last):
File "/home/chromeos-test/chromiumos/src/platform/dev/apache_log_metrics.py", line 20, in <module>
from devserver_lib.devserver import MakeLogHandler
ImportError: No module named devserver_lib.devserver
But seems this log file is useless anymore because we have /var/log/apache_access_log_metrics_daemon.log and /var/log/apache_error_log_metrics_daemon.log
,
Mar 9 2018
,
Mar 9 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/c05fb55a9979f51c174efa1d6c8ecb90547324c7 commit c05fb55a9979f51c174efa1d6c8ecb90547324c7 Author: Congbin Guo <guocb@google.com> Date: Fri Mar 09 23:56:17 2018
,
Mar 12 2018
seems the issue for devserver_apache_access_log_metrics.log is not resolved completely. The log file is still growing and no rotation happened after the CL merged two more days ago. This might be a bug of TimedRotatingFileHandler. Maybe we need to resolve this by manually stop the metric daemon and remove the current log file? Assign this issue to Paul for more investigation.
,
Apr 21 2018
assigning to akeshet for triage
,
Apr 26 2018
-> guocb who is overhauling devservers. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by gu...@chromium.org
, Mar 9 2018Thu Mar 8 18:12:14 PST 2018 devserver_apache_access_log_metrics: started. Traceback (most recent call last): File "/home/chromeos-test/chromiumos/src/platform/dev/apache_log_metrics.py", line 220, in <module> main() File "/home/chromeos-test/chromiumos/src/platform/dev/apache_log_metrics.py", line 212, in main backupCount=_LOG_ROTATION_BACKUP)) File "/usr/lib/python2.7/logging/handlers.py", line 171, in __init__ BaseRotatingHandler.__init__(self, filename, 'a', encoding, delay) File "/usr/lib/python2.7/logging/handlers.py", line 64, in __init__ logging.FileHandler.__init__(self, filename, mode, encoding, delay) File "/usr/lib/python2.7/logging/__init__.py", line 903, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/lib/python2.7/logging/__init__.py", line 928, in _open stream = open(self.baseFilename, self.mode) IOError: [Errno 13] Permission denied: '/var/log/devserver/devserver_apache_access_log_metrics.log'