[cr-audit-commits] Report commits stuck retrying audit. |
||
Issue descriptionCurrently, if a rule panics while auditing a commit, the audit will be retried indefinitely. This can make a commit with a violation remain unnoticed indefinitely. We don't want to alert on every single panic, as they could be caused by transient errors on the network or the backend services. One possible solution is to retry for about 3 hours and alert if the audit is still panicking. 3 hours is chosen as a reasonable timeout because it is slightly more than the expected downtime per quarter of a service with a standard sla of 99.9%.
,
Sep 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/6a575263b80273cada354c6247baea2bc19da1ec commit 6a575263b80273cada354c6247baea2bc19da1ec Author: Roberto Carrillo <robertocn@google.com> Date: Wed Sep 20 23:03:06 2017 [cr-audit-commits] Keep track of retried audits and notify. This change makes the commit auditor increment a counter when an attempt to audit a commit results in the panic of one or more rules. If, after 30 minutes of retrying (6 attempts) some rules still panic, file a bug to notify. R=stgao,chanli BUG= 766359 Change-Id: Ie0c0ab4184bc480b7f1dbcd0397c87a895a91f29 Reviewed-on: https://chromium-review.googlesource.com/671668 Commit-Queue: Roberto Carrillo <robertocn@chromium.org> Reviewed-by: Shuotao Gao <stgao@chromium.org> [modify] https://crrev.com/6a575263b80273cada354c6247baea2bc19da1ec/go/src/infra/appengine/cr-audit-commits/app/rules_config.go [modify] https://crrev.com/6a575263b80273cada354c6247baea2bc19da1ec/go/src/infra/appengine/cr-audit-commits/app/notifier_test.go [modify] https://crrev.com/6a575263b80273cada354c6247baea2bc19da1ec/go/src/infra/appengine/cr-audit-commits/app/commit_auditor.go [modify] https://crrev.com/6a575263b80273cada354c6247baea2bc19da1ec/go/src/infra/appengine/cr-audit-commits/app/notifier.go [modify] https://crrev.com/6a575263b80273cada354c6247baea2bc19da1ec/go/src/infra/appengine/cr-audit-commits/app/findit_rules.go [modify] https://crrev.com/6a575263b80273cada354c6247baea2bc19da1ec/go/src/infra/appengine/cr-audit-commits/app/model.go
,
Sep 20 2017
|
||
►
Sign in to add a comment |
||
Comment 1 by robert...@chromium.org
, Sep 19 2017