New issue
Advanced search Search tips

Issue 675688 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jan 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: ----



Sign in to add a comment

run alerts-dispatcher as a SoM cron task

Project Member Reported by seanmccullough@chromium.org, Dec 19 2016

Issue description

Move all of the alerts-dispatcher logic to an appengine cron task that runs inside SoM. I think we're able to do that now.

The pubsub logic has been slow going (still plenty of work to do!). This would get us into a much simpler single-deployable-app state much more quickly.

SoM on appengine can get all master data from Milo API today (correct?).

test-results, FindIt, build.chromium.org etc I think are all visible from GAE. We can get already gatekeeper config data from gitiles at this point.

And we can probably do without the snapshot/replay feature for the time being, and later re-implement it with a more general purpose {g|p}RPC log replay system.

Re: record/replay: martiniss@ says "Talked to sean friday, we agreed it'd be good. Want to keep replay ability though, so we should figure out where to store the data. Vadim said that we can stream data to cloud storage, and pretend it's a file, so that might work.  https://godoc.org/cloud.google.com/go/storage#ObjectHandle.NewWriter looks like the appropriate function."

 
Project Member

Comment 1 by bugdroid1@chromium.org, Dec 20 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/ae8a2632fef977d4dd6513716c1b264de9610454

commit ae8a2632fef977d4dd6513716c1b264de9610454
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Tue Dec 20 20:01:17 2016

[a-d] Read from milo instead of CBE if milo-host flag is set.

BUG= 675688 

Change-Id: Ic9d736970f6832dbb735e8775b0b1871733b875e
Reviewed-on: https://chromium-review.googlesource.com/422577
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[add] https://crrev.com/ae8a2632fef977d4dd6513716c1b264de9610454/go/src/infra/monitoring/client/milo.go
[modify] https://crrev.com/ae8a2632fef977d4dd6513716c1b264de9610454/go/src/infra/monitoring/dispatcher/dispatcher.go

Project Member

Comment 2 by bugdroid1@chromium.org, Dec 21 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/c121dd502b0398ab6d6165c543cfb9c9c3b4a115

commit c121dd502b0398ab6d6165c543cfb9c9c3b4a115
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Wed Dec 21 00:22:18 2016

[som] /_cron/analyze/:tree handler

This basically runs alerts-dispatcher as an appenengine cron job.

It just spits out the alerts json for now. Subsequent CLs will wire
it up to the datastore.

BUG= 675688 

Change-Id: Ibf8faddf2652699fa819cc2d0247171454e6efbc
Reviewed-on: https://chromium-review.googlesource.com/422494
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>

[add] https://crrev.com/c121dd502b0398ab6d6165c543cfb9c9c3b4a115/go/src/infra/appengine/sheriff-o-matic/som/analyze.go
[add] https://crrev.com/c121dd502b0398ab6d6165c543cfb9c9c3b4a115/go/src/infra/appengine/sheriff-o-matic/som/analyze_test.go
[modify] https://crrev.com/c121dd502b0398ab6d6165c543cfb9c9c3b4a115/go/src/infra/appengine/sheriff-o-matic/som/main.go
[modify] https://crrev.com/c121dd502b0398ab6d6165c543cfb9c9c3b4a115/go/src/infra/monitoring/client/client.go
[modify] https://crrev.com/c121dd502b0398ab6d6165c543cfb9c9c3b4a115/go/src/infra/monitoring/client/milo.go
[modify] https://crrev.com/c121dd502b0398ab6d6165c543cfb9c9c3b4a115/go/src/infra/monitoring/client/test/test.go
[modify] https://crrev.com/c121dd502b0398ab6d6165c543cfb9c9c3b4a115/go/src/infra/monitoring/dispatcher/dispatcher.go

Project Member

Comment 3 by bugdroid1@chromium.org, Dec 21 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/521dc46ba8c12f5eb82facbd37553e8464cd0b63

commit 521dc46ba8c12f5eb82facbd37553e8464cd0b63
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Wed Dec 21 20:06:27 2016

[som] Add cron tasks, actually store alerts from analyzer running in SoM

This prefixes the tree name with "milo." in the datastore key so as not
to overwrite the current alerts from alerts-dispatcher. This will let
us diff the two for some time to make sure the cron job is not causing
regressions.

It also runs a separate cron task for each tree. This allows us some
level of isolation between clients (they can run on different schedules,
not interfere with eachother if one of them crashes etc).

BUG= 675688 

Change-Id: I7c4b258c9ada5e740f0e0327ce850a542441889f
Reviewed-on: https://chromium-review.googlesource.com/422993
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>

[modify] https://crrev.com/521dc46ba8c12f5eb82facbd37553e8464cd0b63/go/src/infra/appengine/sheriff-o-matic/cron.yaml
[modify] https://crrev.com/521dc46ba8c12f5eb82facbd37553e8464cd0b63/go/src/infra/appengine/sheriff-o-matic/som/analyze.go
[modify] https://crrev.com/521dc46ba8c12f5eb82facbd37553e8464cd0b63/go/src/infra/appengine/sheriff-o-matic/som/analyze_test.go

Status: Fixed (was: Started)

Sign in to add a comment