false_rejections_minus_exonerations was provided a negative increment, causing it to fail |
|||
Issue descriptionbuild https://luci-milo.appspot.com/buildbot/chromeos/master-paladin/17553 stage https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fmaster-paladin%2F17553%2F%2B%2Frecipes%2Fsteps%2FCommitQueueCompletion%2F0%2Fstdout Metric was provided a negative increment value, resulting in this failure: 22:14:19: INFO: Recording submission metrics about 91 CLs to monarch. [1;31m22:14:21: ERROR: Caught an exception while running ('CumulativeSmallIntegerDistribution',).('add',)((-1,), fields={'submission_strategy': 'strategy:cq-success'}) Traceback (most recent call last): File "/b/c/cbuild/repository/chromite/lib/ts_mon_config.py", line 276, in _CallMetric **message.method_kwargs) File "/b/c/cbuild/repository/chromite/third_party/infra_libs/ts_mon/common/metrics.py", line 496, in add self._incr(fields, target_fields, value, modify_fn=modify_fn) File "/b/c/cbuild/repository/chromite/third_party/infra_libs/ts_mon/common/metrics.py", line 305, in _incr delta, modify_fn=modify_fn) File "/b/c/cbuild/repository/chromite/third_party/infra_libs/ts_mon/common/metric_store.py", line 215, in incr raise errors.MonitoringDecreasingValueError(name, None, delta) MonitoringDecreasingValueError: Monotonically increasing metric "chromeos/cbuildbot/submitted_change/false_rejections_minus_exonerations" was given value "-1", which is not greater than or equal to "None".[0m P2 metrics outage
,
Jan 23 2018
Don't know, worth investigating. But regardless, the metric emittor should have a sanity check for nonnegative incrementing.
,
Jan 23 2018
OK, that's easy enough. This should be at most an ERROR: log, not actually breaking the other metrics.
,
Jan 27 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/a9dfdbf29e9036214247aa175cc838072882ad0c commit a9dfdbf29e9036214247aa175cc838072882ad0c Author: Paul Hobbs <phobbs@google.com> Date: Sat Jan 27 02:17:17 2018 clactions: Workaround for exonerations > rejections Add a workaround for a bug where sometimes the number of exonerations for a CL is larger than the number of rejections of that CL: - Log an error message when we notice this case - Don't try to increment a metrics Counter with a negative increment, which causes an exception. TEST=clactions_unittest BUG= chromium:804900 Change-Id: If9f1a9067e0e13ce4b4dc0a7e8707112b0b80ea7 Reviewed-on: https://chromium-review.googlesource.com/887359 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Paul Hobbs <phobbs@google.com> Reviewed-by: Aviv Keshet <akeshet@chromium.org> [modify] https://crrev.com/a9dfdbf29e9036214247aa175cc838072882ad0c/lib/clactions.py
,
Jan 27 2018
Still worth investigating - the above CL will provide better log messages for debugging when it happens.
,
Mar 29 2018
|
|||
►
Sign in to add a comment |
|||
Comment 1 by pho...@chromium.org
, Jan 23 2018