New issue
Advanced search Search tips

Issue 651497 link

Starred by 1 user
Status: Available
Owner: ----
Cc:
seanmccullough@chromium.org
zhangtiff@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocking:
issue 630006



Sign in to add a comment

SOM doesn't handle a large number of alerts well

Project Member Reported by martiniss@chromium.org, Sep 29 2016 
This is both a systematic issue and a UX issue.

It is systematic in that we shouldn't really be generating more than ~10-20 alerts per tree. More than that is just too spammy, and indicates a problem with both our infrastructure and the alerts dispatcher process.

It is a UX issue in that the UI is terrible for handling this many alerts. We should try to handle it a bit better; it's basically impossible to deal with more than 10 alerts.

I'll be working on this, as this blocks perfbot sheriffs using sheriff-o-matic.

Comment 1 by seanmccullough@google.com, Sep 29 2016 

It might be useful to think of this as a ranking problem too. Only show the "top" 10 at any given time, even though there may be hundreds identified.  You'd have to come up with a ranking function (perhaps tree- or just perf-specific) to sort the alerts that's more advanced than our current sorting logic.  Have it take into account the number of builders affected, the number of consecutive builds affected, the size of the queues on those builders, some other heuristics about the relative importance of those builders etc.

Comment 2 by martiniss@chromium.org, Sep 29 2016

Blocking: 630006

Comment 3 by martiniss@chromium.org, Oct 6 2016 

An update on this: landing a change to help with this. Not going to work on this much more soon, although long term it'd be nice to solve this.
Project Member

Comment 4 by bugdroid1@chromium.org, Oct 6 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/a92ff18bac1eea81a563a62004a23656ba7c7959

commit a92ff18bac1eea81a563a62004a23656ba7c7959
Author: Stephen Martinis <martiniss@chromium.org>
Date: Thu Oct 06 21:26:07 2016

Bumping prod version

BUG=651497

Change-Id: I6d0411001a52f4d3b82ae8c316914d8049726f1c
Reviewed-on: https://chromium-review.googlesource.com/394867
Reviewed-by: Sean McCullough <seanmccullough@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>

[modify] https://crrev.com/a92ff18bac1eea81a563a62004a23656ba7c7959/go/src/infra/appengine/sheriff-o-matic/RELNOTES.md

Comment 5 by seanmccullough@google.com, Oct 20 2016

Labels: Milestone-UX

Comment 6 by martiniss@chromium.org, Dec 11 2017

Cc: -sullivan@chromium.org
Labels: -OS-Linux
Owner: ----
Status: Available (was: Assigned)
I'm not working on this anymore. I think the SOM team is kinda working on this?

Comment 7 by zhangtiff@chromium.org, Feb 15 2018

Labels: Milestone-Polish
►

Sign in to add a comment