New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 803314 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Last visit > 30 days ago
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug

Blocked on:
issue 806391
issue 804618
issue 810964
issue 813941



Sign in to add a comment

Improve audit app

Project Member Reported by robert...@chromium.org, Jan 18 2018

Issue description

Metabug to make audit app useful for teams outside findit during Q1 '18.

Scope for this objective includes detailed design and implementation of enhancements to support.
- Releasebot usecase (non-constant branch names/refs)
- TBR usecase (time-delayed auditing)

 
Blockedon: 804618
Blockedon: 806391

Comment 3 by st...@chromium.org, Jan 26 2018

TBR usecase could be considered but not the critical case, but no one has a plan to do that because there is no consensus yet.
Blockedon: 810964
Blockedon: 813941
Components: Infra>Audit
Project Member

Comment 8 by bugdroid1@chromium.org, Apr 20 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/e932ace4e465dddb32a5c9685ea1175c47981eb0

commit e932ace4e465dddb32a5c9685ea1175c47981eb0
Author: Roberto Carrillo <robertocn@google.com>
Date: Fri Apr 20 22:29:19 2018

[cr-audit-commits] Use a taskqueue to run audits.

Also, use a single cron job to schedule the audit tasks.

To be clear, this new approach involves two phases (Scheduler Cron Job,
and Auditor Task Queue)

Phase I - Scheduler Cron Job
A new handler called Scheduler is added to iterate over the available
configs and schedule needed audit tasks, creating new RepoState
datastore entries as needed.  Note that the URL to each ref is passed
to the taskqueue task (Auditor handler below), and that the name of the
corresponding config has been added as a field to the RepoState entity.

Phase II - Auditor Task Queue
This change replaces the CommitScanner and CommitAuditor handlers with a
single handler called Auditor, intended to be run as a taskqueue task
when scheduled by the 'Scheduler' cron job above. This task will scan
the git ref, whose url is given as a parameter and using the matching
configuration (from the hardcoded RuleMap) it will create RelevantCommit
entities when appropriate, followed by an audit of such commits and
notifications for those that are found to be in violation of any of the
rules.

The following design docs contain the discussions about this change:
   - go/audit-configuration-for-dynamic-refs
   - go/syncing-audit-app-scheduler-with-audit-task

R=stgao,chanli
BUG=803314

Change-Id: I1b7f52fa68b41db9cc5575f2a42041afde36443c
Reviewed-on: https://chromium-review.googlesource.com/1010976
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Shuotao Gao <stgao@chromium.org>

[modify] https://crrev.com/e932ace4e465dddb32a5c9685ea1175c47981eb0/go/src/infra/appengine/cr-audit-commits/app/monitoring.go
[modify] https://crrev.com/e932ace4e465dddb32a5c9685ea1175c47981eb0/go/src/infra/appengine/cr-audit-commits/app/utils.go
[modify] https://crrev.com/e932ace4e465dddb32a5c9685ea1175c47981eb0/go/src/infra/appengine/cr-audit-commits/app/commit_scanner.go
[modify] https://crrev.com/e932ace4e465dddb32a5c9685ea1175c47981eb0/go/src/infra/appengine/cr-audit-commits/app/cron.yaml
[modify] https://crrev.com/e932ace4e465dddb32a5c9685ea1175c47981eb0/go/src/infra/appengine/cr-audit-commits/app/app.go
[modify] https://crrev.com/e932ace4e465dddb32a5c9685ea1175c47981eb0/go/src/infra/appengine/cr-audit-commits/app/notifier_test.go
[modify] https://crrev.com/e932ace4e465dddb32a5c9685ea1175c47981eb0/go/src/infra/appengine/cr-audit-commits/app/commit_auditor.go
[modify] https://crrev.com/e932ace4e465dddb32a5c9685ea1175c47981eb0/go/src/infra/appengine/cr-audit-commits/app/commit_auditor_test.go
[modify] https://crrev.com/e932ace4e465dddb32a5c9685ea1175c47981eb0/go/src/infra/appengine/cr-audit-commits/app/notifier.go
[modify] https://crrev.com/e932ace4e465dddb32a5c9685ea1175c47981eb0/go/src/infra/appengine/cr-audit-commits/app/commit_scanner_test.go
[modify] https://crrev.com/e932ace4e465dddb32a5c9685ea1175c47981eb0/go/src/infra/appengine/cr-audit-commits/app/rules_config_test.go
[add] https://crrev.com/e932ace4e465dddb32a5c9685ea1175c47981eb0/go/src/infra/appengine/cr-audit-commits/app/queue.yaml
[modify] https://crrev.com/e932ace4e465dddb32a5c9685ea1175c47981eb0/go/src/infra/appengine/cr-audit-commits/app/model.go
[add] https://crrev.com/e932ace4e465dddb32a5c9685ea1175c47981eb0/go/src/infra/appengine/cr-audit-commits/app/scheduler.go

Project Member

Comment 9 by bugdroid1@chromium.org, Apr 20 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/e94ac373204d46f3c4e5757728273d58b3bb09df

commit e94ac373204d46f3c4e5757728273d58b3bb09df
Author: Roberto Carrillo <robertocn@google.com>
Date: Fri Apr 20 22:41:09 2018

[cr-audit-commits] Dynamic Refs Support.

This change adds support for dynamically determining the specific git
ref that a given configuration should apply to. i.e. a configuration for
a stable branch applies to a ref that changes over time.

The following design docs contain the discussions about this change:
  - go/audit-configuration-for-dynamic-refs
  - go/dynamic-ref-functions

R=chanli,stgao,cmasso
BUG=803314

Change-Id: I20b565dfc7fcd637a4ab2c4edfbe7a183076eaa5
Reviewed-on: https://chromium-review.googlesource.com/1011123
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Chan Li <chanli@chromium.org>
Reviewed-by: Shuotao Gao <stgao@chromium.org>

[modify] https://crrev.com/e94ac373204d46f3c4e5757728273d58b3bb09df/go/src/infra/appengine/cr-audit-commits/app/utils.go
[modify] https://crrev.com/e94ac373204d46f3c4e5757728273d58b3bb09df/go/src/infra/appengine/cr-audit-commits/app/scheduler.go
[modify] https://crrev.com/e94ac373204d46f3c4e5757728273d58b3bb09df/go/src/infra/appengine/cr-audit-commits/app/model.go
[modify] https://crrev.com/e94ac373204d46f3c4e5757728273d58b3bb09df/go/src/infra/appengine/cr-audit-commits/app/rules_config.go

Project Member

Comment 10 by bugdroid1@chromium.org, Apr 23 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/0d6f5792079ea8e8d5af8c99056930d668124c0e

commit 0d6f5792079ea8e8d5af8c99056930d668124c0e
Author: Roberto Carrillo <robertocn@google.com>
Date: Mon Apr 23 22:51:39 2018

[cr-audit-commits] Merge files, rename files.

This change is a noop functionally. It only renames some files and
merges together related code:
  - auditor.go: functionality related to scanning, auditing and
      notifying
  - auditor_test.go: all tests related to the auditor handler.
  - audit_worker: functionality that deals with distributing audits to
      multiple goroutines.

R=stgao,chanli
BUG=803314

Change-Id: If93cfa813352e00ee0dc4cc481564534c8d53eb2
Reviewed-on: https://chromium-review.googlesource.com/1011125
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Shuotao Gao <stgao@chromium.org>
Reviewed-by: Chan Li <chanli@chromium.org>

[delete] https://crrev.com/a839180e669ded9bd806421a24ba15ab5b999e53/go/src/infra/appengine/cr-audit-commits/app/commit_scanner.go
[rename] https://crrev.com/0d6f5792079ea8e8d5af8c99056930d668124c0e/go/src/infra/appengine/cr-audit-commits/app/auditor_test.go
[rename] https://crrev.com/0d6f5792079ea8e8d5af8c99056930d668124c0e/go/src/infra/appengine/cr-audit-commits/app/auditor_notification_test.go
[delete] https://crrev.com/a839180e669ded9bd806421a24ba15ab5b999e53/go/src/infra/appengine/cr-audit-commits/app/commit_auditor_test.go
[modify] https://crrev.com/0d6f5792079ea8e8d5af8c99056930d668124c0e/go/src/infra/appengine/cr-audit-commits/app/notification.go
[delete] https://crrev.com/a839180e669ded9bd806421a24ba15ab5b999e53/go/src/infra/appengine/cr-audit-commits/app/notifier.go
[add] https://crrev.com/0d6f5792079ea8e8d5af8c99056930d668124c0e/go/src/infra/appengine/cr-audit-commits/app/auditor.go
[rename] https://crrev.com/0d6f5792079ea8e8d5af8c99056930d668124c0e/go/src/infra/appengine/cr-audit-commits/app/audit_worker.go

Sign in to add a comment