New issue
Advanced search Search tips

Issue 847665 link

Starred by 1 user
Status: Started
Owner:
juliehockett@google.com
Cc:
no...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Feature

Blocked on:
issue 855133
issue 873397


Sign in to add a comment

Support recipe-based analyzers

Project Member Reported by qyears...@chromium.org, May 29 2018 
Related doc: go/tricium-recipe-based-analyzers

This will involve adding some integration with buildbucket to trigger builds and receive results.

Sub-tasks likely include:
 - Determine what a request to buildbucket requires what form it should take
 - Change Recipe message to have a CIPD package and version
 - Remove RecipeCmd-related code
 - Change worker creation to fill input in properties etc
 - Change driver to make requests to buildbucket instead of swarming for recipe-based workers
 - Add a pubsub handler that requests builds (buildbucket "get" request) on build completion

Comment 1 by qyears...@chromium.org, May 30 2018

Description: Show this description
Project Member

Comment 2 by bugdroid1@chromium.org, Jun 4 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/85c74f964d2d2e308c1ccd7aa134c44b6d3abab8

commit 85c74f964d2d2e308c1ccd7aa134c44b6d3abab8
Author: Quinten Yearsley <qyearsley@chromium.org>
Date: Mon Jun 04 21:35:43 2018

[tricium] Change the specification of recipe-based analyzers

This would change the specification of a recipe in the config
schema to just have a cipd package and version; this also
removes the partial implementation of worker setup for
recipe-based analyzers, which would be filled in a later CL.

There are currently no recipe-based analyzers in any configs,
so this should have no effect now.

Bug: 847665
Change-Id: I0b45ff2a9e9144f50b903c8820f950b5b0541d42
Reviewed-on: https://chromium-review.googlesource.com/1080025
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/85c74f964d2d2e308c1ccd7aa134c44b6d3abab8/go/src/infra/tricium/api/v1/tricium.pb.go
[modify] https://crrev.com/85c74f964d2d2e308c1ccd7aa134c44b6d3abab8/go/src/infra/tricium/api/v1/data.pb.go
[modify] https://crrev.com/85c74f964d2d2e308c1ccd7aa134c44b6d3abab8/go/src/infra/tricium/appengine/devcfg/services/tricium-dev/service.cfg
[modify] https://crrev.com/85c74f964d2d2e308c1ccd7aa134c44b6d3abab8/go/src/infra/tricium/api/v1/config.pb.go
[modify] https://crrev.com/85c74f964d2d2e308c1ccd7aa134c44b6d3abab8/go/src/infra/tricium/appengine/common/config/generate_test.go
[modify] https://crrev.com/85c74f964d2d2e308c1ccd7aa134c44b6d3abab8/go/src/infra/tricium/api/v1/pb.discovery.go
[modify] https://crrev.com/85c74f964d2d2e308c1ccd7aa134c44b6d3abab8/go/src/infra/tricium/api/v1/function.proto
[modify] https://crrev.com/85c74f964d2d2e308c1ccd7aa134c44b6d3abab8/go/src/infra/tricium/appengine/common/config/generate.go
[modify] https://crrev.com/85c74f964d2d2e308c1ccd7aa134c44b6d3abab8/go/src/infra/tricium/api/v1/function.pb.go
[modify] https://crrev.com/85c74f964d2d2e308c1ccd7aa134c44b6d3abab8/go/src/infra/tricium/api/v1/platform.pb.go
[modify] https://crrev.com/85c74f964d2d2e308c1ccd7aa134c44b6d3abab8/go/src/infra/tricium/api/v1/config_helpers.go
[modify] https://crrev.com/85c74f964d2d2e308c1ccd7aa134c44b6d3abab8/go/src/infra/tricium/api/v1/config_helpers_test.go

Comment 3 by qyears...@chromium.org, Jun 15 2018

Components: Infra>Platform>Tricium

Comment 4 by qyears...@chromium.org, Jun 15 2018

Components: -Infra>CodeAnalysis
Labels: -Tricium

Comment 5 by qyears...@chromium.org, Jun 21 2018

Blockedon: 855133

Comment 6 by qyears...@chromium.org, Jul 16

Owner: juliehockett@google.com
Reassigning :-)

Question for Nodir:

Last week on Friday Julie was asking whether the swarmbucket API would be relevant, since presumably we'll be wanting to schedule builds that use swarming.

Is there any documentation for swarmbucket besides https://godoc.org/go.chromium.org/luci/common/api/buildbucket/swarmbucket/v1?

From the list of methods in the swarmbucket API, it looks maybe like that's not used for scheduling builds, and we might just want the buildbucket "put" method; is that right?

Comment 7 by qyearsley@google.com, Jul 16 

(Confirmed with Nodir offline, we'd be using the buildbucket API for scheduling builds.)

Comment 8 by no...@chromium.org, Jul 16 

https://godoc.org/go.chromium.org/luci/common/api/buildbucket/swarmbucket/v1?
is irrelevant for this bug.

More relevant is 
https://godoc.org/go.chromium.org/luci/common/api/buildbucket/buildbucket/v1#Service.Put
which is a thin client for https://apis-explorer.appspot.com/apis-explorer/?base=https://cr-buildbucket.appspot.com/_ah/api#p/buildbucket/v1/buildbucket.put
and
go/swarmbucket
covers some of the swarmbucket-specific parts of that API.

overall the new version of this API is ScheduleBuild in go/buildbucket-api-v2. It is planned to be deployed this quarter. Once it is available, it should be used instead of buildbucket.put (which is more difficult to use).

Comment 9 by qyears...@chromium.org, Aug 3

Status: Started (was: Assigned)
Should now be implemented: https://chromium-review.googlesource.com/c/infra/infra/+/1141076

Next step: testing.
Project Member

Comment 10 by bugdroid1@chromium.org, Aug 6 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/891592ede6b36f4a64998d1137074e921b2a9812

commit 891592ede6b36f4a64998d1137074e921b2a9812
Author: Quinten Yearsley <qyearsley@chromium.org>
Date: Mon Aug 06 14:55:20 2018

[tricium] Rename buildbucket_server to buildbucket_server_host

In this CL:
 - Rename buildbucket_server to buildbucket_server_host in config
 - Rename BuildbucketServer to BuildbucketServerHost in workflow
   request API and in datastore schema.
 - Disable validation of this field in configs, so that this change
   can be deployed before updating configs
 - Rename parameters_json to parametersJSON (got presubmit warning)
 - Add schema part in URL for buildbucket v1 request.

Bug: 847665
Change-Id: I424f33ca9d34bf9c7570eec98476f08007f65327
Reviewed-on: https://chromium-review.googlesource.com/1162874
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>

[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/appengine/common/track/track.go
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/api/admin/v1/pb.discovery.go
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/appengine/tracker/rpc_workflow_launched.go
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/appengine/common/buildbucket.go
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/appengine/driver/rpc_trigger.go
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/appengine/driver/rpc_collect.go
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/api/v1/config.proto
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/api/v1/pb.discovery.go
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/appengine/frontend/rpc_progress.go
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/api/admin/v1/workflow.proto
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/api/v1/config.pb.go
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/appengine/common/config/validate.go
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/appengine/launcher/rpc_launch_test.go
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/appengine/common/config/validate_test.go
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/appengine/common/config/generate.go
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/appengine/common/config/generate_test.go
[modify] https://crrev.com/891592ede6b36f4a64998d1137074e921b2a9812/go/src/infra/tricium/api/admin/v1/workflow.pb.go
Project Member

Comment 11 by bugdroid1@chromium.org, Aug 6 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/991bc94e32dfe962f54bb05805fee5a17a6abaad

commit 991bc94e32dfe962f54bb05805fee5a17a6abaad
Author: Quinten Yearsley <qyearsley@chromium.org>
Date: Mon Aug 06 17:04:51 2018
Project Member

Comment 12 by bugdroid1@chromium.org, Aug 6 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/8e5f7385b78611e7391137dd1811cf9a5f5ee250

commit 8e5f7385b78611e7391137dd1811cf9a5f5ee250
Author: Quinten Yearsley <qyearsley@chromium.org>
Date: Mon Aug 06 17:04:53 2018

Comment 13 by qyears...@chromium.org, Aug 6 

Update: the buildbucket API integration has been added, but not yet tested/verified; the a builder has been added (https://ci.chromium.org/p/infra/builders/luci.infra.tricium/tricium), but also not yet tested or verified.
Project Member

Comment 14 by bugdroid1@chromium.org, Aug 6 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/5ed46a82e01e8d94376a87de534a19fcbd122135

commit 5ed46a82e01e8d94376a87de534a19fcbd122135
Author: Quinten Yearsley <qyearsley@chromium.org>
Date: Mon Aug 06 19:00:49 2018

[tricium] Check presence of buildbucket server in service config

Bug: 847665
Change-Id: I23855367952b41c263367314c5179f92ee3d4f46
Reviewed-on: https://chromium-review.googlesource.com/1163837
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>

[modify] https://crrev.com/5ed46a82e01e8d94376a87de534a19fcbd122135/go/src/infra/tricium/appengine/common/config/validate.go
Project Member

Comment 15 by bugdroid1@chromium.org, Aug 8 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/3460ef9058aa8478be96532956c778d1281c4eec

commit 3460ef9058aa8478be96532956c778d1281c4eec
Author: Julie Hockett <juliehockett@google.com>
Date: Wed Aug 08 19:14:23 2018

[tricium] Adding builder and bucket to trigger request

Bug: 847665
Change-Id: I6df6b36e3c850d9d6f4ce58ce5f227b144557ba7
Reviewed-on: https://chromium-review.googlesource.com/1167547
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Commit-Queue: Julie Hockett <juliehockett@google.com>

[modify] https://crrev.com/3460ef9058aa8478be96532956c778d1281c4eec/go/src/infra/tricium/appengine/common/buildbucket.go
[modify] https://crrev.com/3460ef9058aa8478be96532956c778d1281c4eec/go/src/infra/tricium/appengine/common/buildbucket_test.go
Project Member

Comment 16 by bugdroid1@chromium.org, Aug 8 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/recipes-py/+/4f89beb3e0d1c128aa748398fe619ec38a4452b3

commit 4f89beb3e0d1c128aa748398fe619ec38a4452b3
Author: Julie Hockett <juliehockett@google.com>
Date: Wed Aug 08 21:12:10 2018

[tricium] Add repo and ref access properties

Also adding the call to the test

Bug: 847665
Change-Id: Id5a1a9c30f579572dba866e99523d968da309066
Reviewed-on: https://chromium-review.googlesource.com/1167966
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Julie Hockett <juliehockett@google.com>

[modify] https://crrev.com/4f89beb3e0d1c128aa748398fe619ec38a4452b3/recipe_modules/tricium/api.py
[modify] https://crrev.com/4f89beb3e0d1c128aa748398fe619ec38a4452b3/recipe_modules/tricium/examples/full.py
[modify] https://crrev.com/4f89beb3e0d1c128aa748398fe619ec38a4452b3/README.recipes.md

Comment 17 by qyears...@chromium.org, Aug 10

Blockedon: 873397
Project Member

Comment 18 by bugdroid1@chromium.org, Aug 20 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/7d2653711704a873672a21b008ec8d61c135cc55

commit 7d2653711704a873672a21b008ec8d61c135cc55
Author: Julie Hockett <juliehockett@google.com>
Date: Mon Aug 20 22:23:01 2018

[tricium] Check for buildbucket ID in addition to a swarming one

Bug: 847665

Change-Id: I8030266b927484f3c4db6bf781bb67d73de5007e
Reviewed-on: https://chromium-review.googlesource.com/1182193
Commit-Queue: Julie Hockett <juliehockett@google.com>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>

[modify] https://crrev.com/7d2653711704a873672a21b008ec8d61c135cc55/go/src/infra/tricium/appengine/tracker/rpc_worker_launched.go
[modify] https://crrev.com/7d2653711704a873672a21b008ec8d61c135cc55/go/src/infra/tricium/appengine/tracker/rpc_worker_launched_test.go
Project Member

Comment 19 by bugdroid1@chromium.org, Aug 29 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/25bbcceae5f49a408d27b4b8dd30956f2b693cda

commit 25bbcceae5f49a408d27b4b8dd30956f2b693cda
Author: Julie Hockett <juliehockett@google.com>
Date: Wed Aug 29 20:12:36 2018

[tricium] Add buildbucket handling to pubsub handler

Bug: 847665
Change-Id: If7b9c3b735e2a610ef2464a7feec59b7fbd2d8d5
Reviewed-on: https://chromium-review.googlesource.com/1187714
Commit-Queue: Julie Hockett <juliehockett@google.com>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#17285}
[modify] https://crrev.com/25bbcceae5f49a408d27b4b8dd30956f2b693cda/go/src/infra/tricium/appengine/driver/handlers_test.go
[modify] https://crrev.com/25bbcceae5f49a408d27b4b8dd30956f2b693cda/go/src/infra/tricium/appengine/driver/handlers.go
►

Sign in to add a comment