New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 773806 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Last visit > 30 days ago
Closed: Oct 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Make sustained high provision failure rate into an alert

Project Member Reported by dgarr...@chromium.org, Oct 11 2017

Issue description

It's already an Omen.
 
Cc: pho...@chromium.org
Labels: -Pri-3 Chase-Pending Pri-1
Owner: ----
Status: Available (was: Untriaged)
Labels: -Chase-Pending Chase
Owner: jkop@chromium.org
Status: Assigned (was: Available)

Comment 3 by jkop@chromium.org, Oct 17 2017

Status: Started (was: Assigned)

Comment 4 by jkop@chromium.org, Oct 19 2017

Status: Fixed (was: Started)

Comment 5 by jkop@chromium.org, Oct 23 2017

Status: Started (was: Fixed)
There was a typo that made the threshold 100x too low (0.16% instead of 16%). Should be fixed today.

Comment 6 by jkop@chromium.org, Oct 24 2017

Another change collided and made the metric sources change before this landed. Waiting on another change to check that it's fixed.

Comment 7 by jkop@chromium.org, Oct 24 2017

Status: Fixed (was: Started)
Tentatively declaring this fixed. Changes are up, and the alert is tracking the metric.

Declaration is tentative because the Pcon graph for the alert, http://shortn/_EnNA3BNB4a, and the corresponding viceroy Omen, vi/chromeos/provision#_VG_huYBJmlb, don't seem to be tightly coupled.

Comment 8 by pho...@chromium.org, Oct 25 2017

Re #7: I think you're looking at the wrong graph - here it is side by side with the Omen's underlying precomputation: http://shortn/_wY1i7w2w69

Comment 9 by jkop@chromium.org, Oct 25 2017

Oh, excellent. Yes, those match up, that looks like the differences are entirely about how they're doing averaging.

I'm not sure how to match up the queries more precisely. That's desirable but, I think, a low priority.

Comment 10 by jkop@chromium.org, Oct 31 2017

Status: Verified (was: Fixed)
Alert fired correctly today.

Sign in to add a comment