New issue
Advanced search Search tips

Issue 689284 link

Starred by 2 users
Status: Fixed
Owner:
seanmccullough@chromium.org
Closed: Jun 2017
Cc:
martiniss@chromium.org
zhangtiff@chromium.org
seanmccullough@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: ----


Participants' hotlists:
GAE-cron


Sign in to add a comment

explain diffs btw alerts-dispatcher and cron alerts

Project Member Reported by seanmccullough@google.com, Feb 7 2017 
Here's a metric for the edit distance between the two alerts JSONs per tree:
https://pcon.corp.google.com/p#chrome-infra/queryplayground?duration=86400&heatmapColorScale=viceroy&legendtable=true&names=Requests%20by%20app%20version&oldHeatmap=false&outputPoints=900&showEditor=true&stacked=true&title=Requests%20by%20app%20version&yAxisLabel=QPM&yAxisMin=0&query=mash&mash=Fetch(Raw('monarch.acquisitions.Task',%20'/chrome/infra/analyzer/cron_alert_diffs'),%20%7B'data_center':%20'appengine',%20'service_name':%20'sheriff-o-matic-staging'%7D)%0A%7C%20Window(Rate('20m'))%0A%7C%20Point(VAL%20*%2060)%0A%7C%20GroupBy(%5B'metric:tree'%5D)

Since the alert generating logic is identical (modulo any skew between what's deployed on infra.cron and appengine), I suspect the differences come from either the source of the data (a-d gets it from CBE, cron gets it from Milo) or from the run schedules (a-d and the crons do not run on the same schedule).

Note that there are brief periods where the diffs drop to near zero. This might suggest they are different because of the schedules.

You can manually inspect the diffs for a given tree by viewing /alertdiff/<treename> e.g. https://sheriff-o-matic-staging.appspot.com/alertdiff/chromium

We should resolve this before turning alerts-dispatcher down.

Comment 1 by martiniss@chromium.org, Feb 7 2017 

I manually looked at chromium right now. I see a couple missing things.

1. No findit data in new data.
2. No revision summaries. (do we use/need these)?

I don't see these differences reflected on the alertdiff page...

Also, are you making sure to sort the json dictionaries by keys? Otherwise the diff might be useless.

Comment 2 by seanmccullough@chromium.org, Apr 17 2017

Cc: -seanmccullough@chromium.org
Owner: seanmccullough@chromium.org
Status: Started (was: Available)
Project Member

Comment 5 by bugdroid1@chromium.org, May 30 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/d40d17c903dd630e271146cadb177e40f4d426f3

commit d40d17c903dd630e271146cadb177e40f4d426f3
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Tue May 30 17:19:45 2017

[som] Switch android trees to use GAE cron tasks instead of a-d alerts.

Bug:689284
Change-Id: Icea6e32103eb2cf27c463bdac06e4a61b97b35f3
Reviewed-on: https://chromium-review.googlesource.com/517908
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/d40d17c903dd630e271146cadb177e40f4d426f3/go/src/infra/appengine/sheriff-o-matic/som/alerts.go
Project Member

Comment 6 by bugdroid1@chromium.org, Jun 13 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/67832db5e228d1c641193d951e9a86ddf3e7cd8d

commit 67832db5e228d1c641193d951e9a86ddf3e7cd8d
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Tue Jun 13 22:14:44 2017

[som] Default chromium.perf to use the GAE cron analyzer alerts.

Bug:  689284 
Change-Id: I24f321b8cfe9695750e50146fd84a86f86924b14
Reviewed-on: https://chromium-review.googlesource.com/533639
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/67832db5e228d1c641193d951e9a86ddf3e7cd8d/go/src/infra/appengine/sheriff-o-matic/som/alerts.go
[modify] https://crrev.com/67832db5e228d1c641193d951e9a86ddf3e7cd8d/go/src/infra/appengine/sheriff-o-matic/som/main_test.go

Comment 7 by seanmccullough@chromium.org, Jun 15 2017

Status: Fixed (was: Started)
All trees that were analyzed by alerts-dispatcher are now analyzed by GAE cron jobs. 

We can now turn down A-D. https://bugs.chromium.org/p/chromium/issues/detail?id=733039
►

Sign in to add a comment