New issue
Advanced search Search tips

Issue 905870 link

Starred by 1 user

Issue metadata

Status: Closed
Owner:
Closed: Dec 10
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Feature



Sign in to add a comment

tsmon log is too verbose

Project Member Reported by no...@chromium.org, Nov 15

Issue description

on the event that tsmon is unable to upload metrics, it logs a warning/error. If flushing runs in a tight loop, like it does in CQ, and there is a long period of time (hours) where it is impossible to send metrics, tsmon spams logs to the extent that it hits limits of other systems, e.g. logging system.
This caused https://irm.corp.google.com/incidents/a.fnj66Wxpx1YGDKxdvYQX/investigate

There is no value in logging the same message every 2sec. At the very lease intervals between such messages should be exponential. 
 
Owner: slewiskelly@chromium.org
Scott, this is probably a good bug for you to take.
Owner: slewiskelly@google.com
I wonder if this was fixed by https://chromium-review.googlesource.com/c/infra/infra/+/1043566.

It looks like the implementations of ts_mon that were running in a tight loop, now backoff exponentially.
Status: Closed (was: Untriaged)

Sign in to add a comment