tsmon log is too verbose |
|||
Issue descriptionon the event that tsmon is unable to upload metrics, it logs a warning/error. If flushing runs in a tight loop, like it does in CQ, and there is a long period of time (hours) where it is impossible to send metrics, tsmon spams logs to the extent that it hits limits of other systems, e.g. logging system. This caused https://irm.corp.google.com/incidents/a.fnj66Wxpx1YGDKxdvYQX/investigate There is no value in logging the same message every 2sec. At the very lease intervals between such messages should be exponential.
,
Nov 20
,
Nov 22
I wonder if this was fixed by https://chromium-review.googlesource.com/c/infra/infra/+/1043566. It looks like the implementations of ts_mon that were running in a tight loop, now backoff exponentially.
,
Dec 10
|
|||
►
Sign in to add a comment |
|||
Comment 1 by abennetts@google.com
, Nov 16