New issue
Advanced search Search tips
Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jan 29
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Feature

Blocked on:
issue 851036
issue 914996
issue 917851

Blocking:
issue 832733
issue 843318
issue 859231
issue 879752



Sign in to add a comment
link

Issue 875056: Deploy ScheduleBuild RPC

Reported by no...@chromium.org, Aug 16 Project Member

Issue description

Implement and deploy ScheduleBuild v2 RPC from go/buildbucket-api-v2
 

Comment 1 by bugdroid1@chromium.org, Aug 21

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/61fc6b56155ff2074f1989a6d5a0628986df1ad0

commit 61fc6b56155ff2074f1989a6d5a0628986df1ad0
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Aug 21 16:01:53 2018

[buildbucket] Add ScheduleBuild RPC

Define ScheduleBuild RPC with its request message.

This API supersedes current
https://apis-explorer.appspot.com/apis-explorer/?base=https://cr-buildbucket-dev.appspot.com/_ah/api#p/buildbucket/v1/buildbucket.put

This CL does not yet add "dynamic builders" support.

Bug:  875056 
Change-Id: Id6f995e9b1cc238780c58931e578d0778650912b
Reviewed-on: https://chromium-review.googlesource.com/1178923
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Reviewed-by: Shuotao Gao <stgao@chromium.org>
Reviewed-by: Roberto Carrillo <robertocn@chromium.org>

[modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/build.pb.go
[modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/build.proto
[modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/common.pb.go
[modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/common.proto
[modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/config/project_config.pb.go
[modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/config/project_config.proto
[modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/config/service_config.pb.go
[add] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/notification.pb.go
[add] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/notification.proto
[modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/pb.discovery.go
[modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/rpc.mock.pb.go
[modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/rpc.pb.go
[modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/rpc.proto

Comment 2 by bugdroid1@chromium.org, Aug 23

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/09d87c18d938627e494fe76835abe22558a1c97a

commit 09d87c18d938627e494fe76835abe22558a1c97a
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Aug 23 17:41:38 2018

[buildbucket] Allow overriding builder of template build

ScheduleBuildRequest currently requires that template_build_id and builder
are mutually exclusive. Remove this requirement.

Bug:  875056 
Change-Id: I674cd69510a65c1446d963ce49f3378216833cb3
Reviewed-on: https://chromium-review.googlesource.com/1186035
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/09d87c18d938627e494fe76835abe22558a1c97a/buildbucket/proto/pb.discovery.go
[modify] https://crrev.com/09d87c18d938627e494fe76835abe22558a1c97a/buildbucket/proto/rpc.pb.go
[modify] https://crrev.com/09d87c18d938627e494fe76835abe22558a1c97a/buildbucket/proto/rpc.proto

Comment 3 by no...@chromium.org, Aug 23

Blockedon: 877161

Comment 4 by bugdroid1@chromium.org, Aug 24

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/5d58f4aebf92273a3ee73a54688b171ece780dc9

commit 5d58f4aebf92273a3ee73a54688b171ece780dc9
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Aug 24 00:02:46 2018

[buildbucket] Add ScheduleBuildRequest validation

Also prohibit properties that are superseded by new typed fields,
e.g. gitiles_commit supersedes repository, branch and revision properties.
See crbug.com/877161

Bug:  875056 , 877161
Change-Id: I1da3730dd1073eca91d5f585ac667d6d3a2b944f
Reviewed-on: https://chromium-review.googlesource.com/1186063
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Erik Staab <estaab@chromium.org>

[modify] https://crrev.com/5d58f4aebf92273a3ee73a54688b171ece780dc9/appengine/cr-buildbucket/v2/test/validation_test.py
[modify] https://crrev.com/5d58f4aebf92273a3ee73a54688b171ece780dc9/appengine/cr-buildbucket/v2/validation.py

Comment 5 by no...@chromium.org, Aug 25

Blockedon: 859231

Comment 6 by bugdroid1@chromium.org, Aug 30

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/23031707dce2731f547b911ad762f76943701f88

commit 23031707dce2731f547b911ad762f76943701f88
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Aug 30 14:51:40 2018

[buildbucket] Store input GitilesCommit

Buildbucket currently synthesizes v2 Build.input.gitiles_commit from tags.
However, new ScheduleBuild API allows specifying ref and position, which are
not present in the tags.

Add new Build datastore property that stores the input
GitilesCommit in binary form. Use it when converting a model.Build to
Build message. Extend creation.BuildRequest to include gitiles_commit.
It will be used in v2 ScheduleBuild implementation.

R=tandrii@chromiium.org

Bug:  875056 
Change-Id: I85e6d129dc2bf8932efe6c67e1162fbf95029662
Reviewed-on: https://chromium-review.googlesource.com/1187651
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#17312}
[modify] https://crrev.com/23031707dce2731f547b911ad762f76943701f88/appengine/cr-buildbucket/v2/builds.py
[modify] https://crrev.com/23031707dce2731f547b911ad762f76943701f88/appengine/cr-buildbucket/model.py
[modify] https://crrev.com/23031707dce2731f547b911ad762f76943701f88/appengine/cr-buildbucket/test/creation_test.py
[modify] https://crrev.com/23031707dce2731f547b911ad762f76943701f88/appengine/cr-buildbucket/creation.py

Comment 7 by no...@chromium.org, Aug 31

Blocking: 879752

Comment 8 by no...@chromium.org, Sep 7

Blocking: 843318

Comment 9 by no...@chromium.org, Sep 24

Blockedon: -877161
unblocking this from issue 877161 (stop using deprecated properties in recipes).
we will implement this before updating recipes. The new API won't be usable for recipes that depend on deprecated properties.

Comment 10 by no...@chromium.org, Sep 25

Blocking: 859231

Comment 11 by no...@chromium.org, Sep 25

Blockedon: -859231

Comment 12 by no...@chromium.org, Sep 25

Blockedon: 851036

Comment 13 by bugdroid1@chromium.org, Dec 11

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/3b6791a2b2b73127adbc9b4ad760b212511a6312

commit 3b6791a2b2b73127adbc9b4ad760b212511a6312
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Dec 11 00:03:51 2018

[buildbucket] simplify BuildRequest.normalize

99% of what BuildRequest.normalize does is validation, and normalization
is trivial. Rename to validate() and do not return a new BuildRequest.

Bug:  875056 
Change-Id: I846f57c4adc8bc6fcad15d41189528cac862c1f4
Reviewed-on: https://chromium-review.googlesource.com/c/1370557
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19469}
[modify] https://crrev.com/3b6791a2b2b73127adbc9b4ad760b212511a6312/appengine/cr-buildbucket/swarming/swarmbucket_api.py
[modify] https://crrev.com/3b6791a2b2b73127adbc9b4ad760b212511a6312/appengine/cr-buildbucket/creation.py

Comment 14 by bugdroid1@chromium.org, Dec 11

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/9e3690566f676908f39f172570faecf78b8d6c25

commit 9e3690566f676908f39f172570faecf78b8d6c25
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Dec 11 19:27:19 2018

[luci_notify] Read email_notify from properties

In the past we've decided to pass email_notify list via parameters, as opposed
to properties. V2 does not have parameters, only properties, and there are no
other use cases to extend go/build-proto.

It is possible that a recipe may want to mutate the email_notify list, e.g.
define who to notify, thus it makes sense to have the email notify list in
build output too.

To avoid polluting go/build-proto for a luci-notify-specific feature, keep
it a "schemaless" property, as opposed to a proto field in go/build-proto.

Parse email_notify from output. If it isn't there, read it from input.
Otherwise fallback to parameters.
ScheduleBuild v2 API does not allow passing parameters, so eventually users
will be forced to switch to properties.

Bug:  875056 
Change-Id: I36da0eb11743f368424132ccb5c8bda37d0ffde6
Reviewed-on: https://chromium-review.googlesource.com/c/1370610
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/9e3690566f676908f39f172570faecf78b8d6c25/luci_notify/notify/pubsub.go
[modify] https://crrev.com/9e3690566f676908f39f172570faecf78b8d6c25/luci_notify/notify/pubsub_test.go

Comment 15 by bugdroid1@chromium.org, Dec 12

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/a6837f49bed316e1a08cecfa76a85eb5f2d7e5d8

commit a6837f49bed316e1a08cecfa76a85eb5f2d7e5d8
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Dec 12 15:33:32 2018

[buildbucket] Store input properties as a struct

Store input properties in proto binary format.
Start writing the entity on build creation.

R=vadimsh@chromium.org

Bug:  875056 
Change-Id: Iacdb53f7c6440c66adbd29fbf669c2634bc8760d
Reviewed-on: https://chromium-review.googlesource.com/c/1372341
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19513}
[modify] https://crrev.com/a6837f49bed316e1a08cecfa76a85eb5f2d7e5d8/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/a6837f49bed316e1a08cecfa76a85eb5f2d7e5d8/appengine/cr-buildbucket/v2/builds.py
[modify] https://crrev.com/a6837f49bed316e1a08cecfa76a85eb5f2d7e5d8/appengine/cr-buildbucket/model.py
[modify] https://crrev.com/a6837f49bed316e1a08cecfa76a85eb5f2d7e5d8/appengine/cr-buildbucket/test/creation_test.py
[modify] https://crrev.com/a6837f49bed316e1a08cecfa76a85eb5f2d7e5d8/appengine/cr-buildbucket/creation.py

Comment 16 by bugdroid1@chromium.org, Dec 12

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/197f6943642e554c4b98d1349a2128d8cb8bcf0f

commit 197f6943642e554c4b98d1349a2128d8cb8bcf0f
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Dec 12 15:47:42 2018

[buildbucket] Store output properties separately

Store output properties in a separate entity, in proto binary format.
Start writing the entity on build update.
Reading part of the code will be updated later.

Unlike input properties, output properties are much larger, and
thus it makes sense to store them separately, as an optimization.

R=vadimsh@chromium.org

Bug:  875056 
Change-Id: If15a8983978c642ef5578aa3a4833e6192f23f93
Reviewed-on: https://chromium-review.googlesource.com/c/1372031
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19514}
[modify] https://crrev.com/197f6943642e554c4b98d1349a2128d8cb8bcf0f/appengine/cr-buildbucket/service.py
[modify] https://crrev.com/197f6943642e554c4b98d1349a2128d8cb8bcf0f/appengine/cr-buildbucket/test/service_test.py
[modify] https://crrev.com/197f6943642e554c4b98d1349a2128d8cb8bcf0f/appengine/cr-buildbucket/v2/api.py
[modify] https://crrev.com/197f6943642e554c4b98d1349a2128d8cb8bcf0f/appengine/cr-buildbucket/model.py
[modify] https://crrev.com/197f6943642e554c4b98d1349a2128d8cb8bcf0f/appengine/cr-buildbucket/v2/test/api_test.py

Comment 17 by bugdroid1@chromium.org, Dec 12

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/896de66b8b44c7d1a13ac1c8f395511ed0a7e246

commit 896de66b8b44c7d1a13ac1c8f395511ed0a7e246
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Dec 12 15:53:22 2018

[buildbucket] Backfill property entities

Backfill BuildInputProperties and BuildOutputProperties entities.
Repurpose put_builds to fix_builds to do arbitrary transformations.

R=vadimsh@chromium.org

Bug:  875056 
Change-Id: Ib018dd2d06f9815f830cace579ff1d42e7d8deaf
Reviewed-on: https://chromium-review.googlesource.com/c/1372739
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19515}
[add] https://crrev.com/896de66b8b44c7d1a13ac1c8f395511ed0a7e246/appengine/cr-buildbucket/fix_builds.py
[delete] https://crrev.com/197f6943642e554c4b98d1349a2128d8cb8bcf0f/appengine/cr-buildbucket/reput_builds.py
[modify] https://crrev.com/896de66b8b44c7d1a13ac1c8f395511ed0a7e246/appengine/cr-buildbucket/api.py

Comment 18 by bugdroid1@chromium.org, Dec 12

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/924cd595d8b6a142178a2877a4e99ac565661129

commit 924cd595d8b6a142178a2877a4e99ac565661129
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Dec 12 16:52:33 2018

Fix fix_builds.py

Forgot to upload a fix to the uploaded patchset.

TBR=vadimsh@chromium.org

Bug:  875056 
Change-Id: Idb72cfb06e74e7f78f73c39dc1a16a91287d6e0c
Reviewed-on: https://chromium-review.googlesource.com/c/1373991
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19517}
[modify] https://crrev.com/924cd595d8b6a142178a2877a4e99ac565661129/appengine/cr-buildbucket/fix_builds.py

Comment 19 by bugdroid1@chromium.org, Dec 13

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/8fe09ae98917c44b29ed8d8441625e43f4615d9e

commit 8fe09ae98917c44b29ed8d8441625e43f4615d9e
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Dec 13 19:56:37 2018

[buildbucket] Log integer properties

We've discovered that google.protobuf.Struct is a lossy format
compared to raw JSON: Struct encodes all numbers are floats, i.e. converts
`0` to `0.0`. This may break recipes.

Log integer properties to estimate the impact.

This CL will be reverted.

Bug:  875056 
Change-Id: I70eafc225c02d7945dfc704aac590ccbc2e440cf
Reviewed-on: https://chromium-review.googlesource.com/c/1376949
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19542}
[modify] https://crrev.com/8fe09ae98917c44b29ed8d8441625e43f4615d9e/appengine/cr-buildbucket/test/creation_test.py
[modify] https://crrev.com/8fe09ae98917c44b29ed8d8441625e43f4615d9e/appengine/cr-buildbucket/creation.py

Comment 20 by bugdroid1@chromium.org, Dec 13

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/6168d72a7f46884768262a5e6929eba8c08df269

commit 6168d72a7f46884768262a5e6929eba8c08df269
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Dec 13 20:28:21 2018

[buildbucket] Split fix_builds RPCs.

fix_builds.py is doing one large RPC and it hits RPC size limits.
Split the RPC into two, runs them in parallel.

Bug:  875056 
Change-Id: I55ec89be5d7511deb26a72386dba2c0d6bd4b11d
Reviewed-on: https://chromium-review.googlesource.com/c/1376950
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19544}
[modify] https://crrev.com/6168d72a7f46884768262a5e6929eba8c08df269/appengine/cr-buildbucket/fix_builds.py

Comment 21 by bugdroid1@chromium.org, Dec 13

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/35dc0c37bc6d2b8af8c4a071ebb3f8b30a38da63

commit 35dc0c37bc6d2b8af8c4a071ebb3f8b30a38da63
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Dec 13 21:16:16 2018

[buildbucket] Check that properties is dict

fix_builds: ignore output properties that are not dict.

Bug:  875056 
Change-Id: I48436a98a2bc4cb7ae0552c157508127e0d4241a
Reviewed-on: https://chromium-review.googlesource.com/c/1376952
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19550}
[modify] https://crrev.com/35dc0c37bc6d2b8af8c4a071ebb3f8b30a38da63/appengine/cr-buildbucket/fix_builds.py

Comment 22 by no...@chromium.org, Dec 13

Blockedon: 914996

Comment 23 by bugdroid1@chromium.org, Dec 14

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c0216c0041ff14aa6961f4d48bb48f356dbd7e68

commit c0216c0041ff14aa6961f4d48bb48f356dbd7e68
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Dec 14 02:32:10 2018

[buildbucket] Check input properties is dict

The non-dict properties also exist in parameters.
Check their type too.

Bug:  875056 
Change-Id: Ia6db0bbdb68ffd65b5c790387e2379e7a9bf1a3d
Reviewed-on: https://chromium-review.googlesource.com/c/1376797
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19563}
[modify] https://crrev.com/c0216c0041ff14aa6961f4d48bb48f356dbd7e68/appengine/cr-buildbucket/fix_builds.py

Comment 24 by bugdroid1@chromium.org, Dec 14

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/98795d43019524c872f6fa2f4d468eccedbbbde8

commit 98795d43019524c872f6fa2f4d468eccedbbbde8
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Dec 14 20:18:52 2018

[buildbucket] Fix build parts sequentially

ndb has put batcher that combines puts. Do puts sequentially.

Bug:  875056 
Change-Id: If7d8e9079c7a8ecd25d503738ed4a4d3626ebfe1
Reviewed-on: https://chromium-review.googlesource.com/c/1378786
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19567}
[modify] https://crrev.com/98795d43019524c872f6fa2f4d468eccedbbbde8/appengine/cr-buildbucket/fix_builds.py

Comment 25 by bugdroid1@chromium.org, Dec 18

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f

commit aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Dec 18 02:28:13 2018

[buildbucket] Remove Build.parameters_actual

parameters_actual was needed because server might have change user-specified
parameters. Now server uses Build.input_properties, which is allowed to have
server-defined properties.

Update all usages to use Build.input_properties instead.

This has a non-obvious side effect. Struct message in the latest version of
protobuf-python does not accept integers. All numbers are floats, and
they are rendered as doubles in JSON (0 -> 0.0). In general, this is a
breaking change.  crbug.com/914996  updates recipes to tolerate floats where
it used to expect integers, but this CL also has a whitelist of widely used,
deprecated properties (like buildnumber) which buildbucket converts back to
int. Updating all recipes to use that is a lot of work, and recipes must
migrate to new recipe API anyway: crbug.com/877161

R=vadimsh@chromium.org

Bug:  875056 
Change-Id: I39979d3541c12542bdf572ebafda5126d56d033b
Reviewed-on: https://chromium-review.googlesource.com/c/1373312
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19627}
[modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/fix_builds.py
[modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/test/api_common_test.py
[modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/api_common.py
[modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/v2/builds.py
[modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/v2/test/api_test.py
[modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/v2/test/builds_test.py
[add] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/bbutil.py
[modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/swarming/test/swarmbucket_api_test.py
[modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/model.py

Comment 26 by bugdroid1@chromium.org, Dec 19

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/e0201aaefff27fe54c21120b052cb19b456087d6

commit e0201aaefff27fe54c21120b052cb19b456087d6
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Dec 19 00:33:37 2018

[buildbucket] Switch to input_properties

Update code that reads input properties from expecting 'properties' parameter
to reading model.Build.input_properties.

This changes the v1 API a bit. API responses now include input properties
defined on the server. This should have no effect though.

R=vadimsh@chromium.org

Bug:  875056 
Change-Id: Id5435893303793cb562485d45f00d103bc993516
Reviewed-on: https://chromium-review.googlesource.com/c/1374000
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19654}
[modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/test/api_common_test.py
[modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/api_common.py
[modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/test/api_test.py
[modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/test/notifications_test.py
[modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/bbutil.py
[modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/model.py

Comment 27 by bugdroid1@chromium.org, Dec 19

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/48881a12617a58e106222f6c675a6afcc6cbec05

commit 48881a12617a58e106222f6c675a6afcc6cbec05
Author: Nodir Turakulov <nodir@chromium.org>
Date: Wed Dec 19 02:43:17 2018

Revert "[buildbucket] Switch to input_properties"

This reverts commit e0201aaefff27fe54c21120b052cb19b456087d6.

Reason for revert:  crbug.com/916359 

Original change's description:
> [buildbucket] Switch to input_properties
> 
> Update code that reads input properties from expecting 'properties' parameter
> to reading model.Build.input_properties.
> 
> This changes the v1 API a bit. API responses now include input properties
> defined on the server. This should have no effect though.
> 
> R=‚Äčvadimsh@chromium.org
> 
> Bug:  875056 
> Change-Id: Id5435893303793cb562485d45f00d103bc993516
> Reviewed-on: https://chromium-review.googlesource.com/c/1374000
> Commit-Queue: Nodir Turakulov <nodir@chromium.org>
> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#19654}

TBR=vadimsh@chromium.org,nodir@chromium.org

Change-Id: I538a9bca666160659e2f13d01934586b50a011fb
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  875056 
Reviewed-on: https://chromium-review.googlesource.com/c/1383651
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19660}
[modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/test/api_common_test.py
[modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/api_common.py
[modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/test/api_test.py
[modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/test/notifications_test.py
[modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/bbutil.py
[modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/model.py

Comment 28 by bugdroid1@chromium.org, Dec 20

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/ed92b5f582a1fcb23c11ab4f3b47ef0c0a8f9016

commit ed92b5f582a1fcb23c11ab4f3b47ef0c0a8f9016
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Dec 20 19:50:03 2018

[buildbucket] Add requested_properties

Add Build.infra.buildbucket.requested_properties, the properties that were
specified in ScheduleBuildRequest. This is needed for CQ to judge about
CQ-suitability of a build.

Note that embedding entire ScheduleBuildRequest into Build is excessive (many
field values match) and hard (proto file cyclic dependency).

Bug:  875056 
Change-Id: I866d671454dc8f6ddaf603ac33b0351a7e76ea80
Reviewed-on: https://chromium-review.googlesource.com/c/1385674
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/ed92b5f582a1fcb23c11ab4f3b47ef0c0a8f9016/buildbucket/proto/build.pb.go
[modify] https://crrev.com/ed92b5f582a1fcb23c11ab4f3b47ef0c0a8f9016/buildbucket/proto/build.proto
[modify] https://crrev.com/ed92b5f582a1fcb23c11ab4f3b47ef0c0a8f9016/buildbucket/proto/pb.discovery.go

Comment 29 by no...@chromium.org, Dec 26

Blocking: 917851

Comment 30 by bugdroid1@chromium.org, Jan 3

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/a535d939ef4d9f15ce38dfc25d31e83cddc732cc

commit a535d939ef4d9f15ce38dfc25d31e83cddc732cc
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Jan 03 18:20:15 2019

Reland "[buildbucket] Switch to input_properties"

Update code that reads input properties from expecting 'properties' parameter
to reading model.Build.input_properties.

Unlike https://chromium-review.googlesource.com/c/infra/infra/+/1374000
does not make breaking changes to API.

Bug:  875056 

Change-Id: I0d023311f7f2490d03bbf5f59c89fb746f9c380f
Reviewed-on: https://chromium-review.googlesource.com/c/1385945
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19754}
[modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/test/api_common_test.py
[modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/api_common.py
[modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/test/api_test.py
[modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/test/notifications_test.py
[modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/bbutil.py
[modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/model.py

Comment 31 by bugdroid1@chromium.org, Jan 4

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/fbc480d33ffc14bb421b53db0146350929750ade

commit fbc480d33ffc14bb421b53db0146350929750ade
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jan 04 20:20:41 2019

[buildbucket] Remove logging of int properties

It served its purpose.

R=vadimsh@chromium.org

Bug:  875056 
Change-Id: I5599fbb000ae78d7798eead9f2a81395499b88d7
Reviewed-on: https://chromium-review.googlesource.com/c/1383342
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19783}
[modify] https://crrev.com/fbc480d33ffc14bb421b53db0146350929750ade/appengine/cr-buildbucket/test/creation_test.py
[modify] https://crrev.com/fbc480d33ffc14bb421b53db0146350929750ade/appengine/cr-buildbucket/creation.py

Comment 32 by bugdroid1@chromium.org, Jan 10

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/767c5a8823fbff83aea5b1e7268943e3b7ece2bb

commit 767c5a8823fbff83aea5b1e7268943e3b7ece2bb
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Jan 10 19:35:03 2019

[buildbucket] Use ScheduleBuildRequest

Update creation.BuildRequest to embed rpc_pb2.ScheduleBuildRequest.
Leave BuildRequest for legacy stuff.

Move BuildRequest validation to api.py because v2/api.py has its own
validation.

Change "retry" API to create a build with all input properties that
were actually in the build, as opposed to properties that were
specified in the request to create the build.
This is the semantics of the future API.
In the past it was done differently because buildbucket rejected such requests,
because they attempted to use reserved properties (created by buildbucket
itself). Now such builds are allowed to use reserved properties, but they
would be overwritten anyway.

Move "retry" implementation to api.py because v2 will use a different impl.

Bug:  875056 

Change-Id: I53c2c892865de52930c6d38a56c88f718fda76d4
Reviewed-on: https://chromium-review.googlesource.com/c/1375133
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19901}
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/test/api_common_test.py
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/v2/builds.py
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/swarming/swarmingcfg.py
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/test/service_test.py
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/api.py
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/test/api_test.py
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/test/creation_test.py
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/v2/test/api_test.py
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/swarming/swarmbucket_api.py
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/creation.py
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/swarming/test/swarmbucket_api_test.py
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/bbutil.py
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/model.py
[modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/v2/validation.py

Comment 33 by bugdroid1@chromium.org, Jan 15

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/a737e4e78db9a7d1782807b7f16779cdd32de943

commit a737e4e78db9a7d1782807b7f16779cdd32de943
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Jan 15 00:58:51 2019

[buildbucket] Move permission check to api.py

V1 put_batch API returns a single HTTP 403 if any of the sub-requests fail
due to insufficient permissions.
It was done so because v1's application-level error codes did not have a code
for access denials. It was too late to add them there.

V2 can have a more uniform semantics.
This leads to two different access check implementations.
Move the current implementation to (v1) api.py.

Also this fixes a deadlock bug: utils.async_apply was used in a tasklet.

Bug:  875056 
Change-Id: Icd50678e2146d0d9e434cc4ac932f5d579450bb6
Reviewed-on: https://chromium-review.googlesource.com/c/1409367
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19973}
[modify] https://crrev.com/a737e4e78db9a7d1782807b7f16779cdd32de943/appengine/cr-buildbucket/test/creation_test.py
[modify] https://crrev.com/a737e4e78db9a7d1782807b7f16779cdd32de943/appengine/cr-buildbucket/creation.py
[modify] https://crrev.com/a737e4e78db9a7d1782807b7f16779cdd32de943/appengine/cr-buildbucket/test/api_test.py
[modify] https://crrev.com/a737e4e78db9a7d1782807b7f16779cdd32de943/appengine/cr-buildbucket/api.py

Comment 34 by bugdroid1@chromium.org, Jan 15

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/a422dd55a71c6bf2a9d74fa0696ab56a562e8ae5

commit a422dd55a71c6bf2a9d74fa0696ab56a562e8ae5
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Jan 15 09:06:01 2019

[buildbucket] Add batched ScheduleBuild requests.

Add schedule build semantics to batch request/response messages.

R=vadimsh@chromium.org

Bug:  875056 
Change-Id: I18a8d161783a979ee1df31979e05c8a17415f711
Reviewed-on: https://chromium-review.googlesource.com/c/1409772
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Auto-Submit: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/a422dd55a71c6bf2a9d74fa0696ab56a562e8ae5/buildbucket/proto/pb.discovery.go
[modify] https://crrev.com/a422dd55a71c6bf2a9d74fa0696ab56a562e8ae5/buildbucket/proto/rpc.pb.go
[modify] https://crrev.com/a422dd55a71c6bf2a9d74fa0696ab56a562e8ae5/buildbucket/proto/rpc.proto

Comment 35 by bugdroid1@chromium.org, Jan 15

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c

commit e7ff36293e7abfbf7dc1d230e881d3ca9021c64c
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Jan 15 18:24:10 2019

[buildbucket] Move parse_bucket_cfg to test_util

parse_bucket_cfg is used often in tests. Move it to test_util.py

Bug:  875056 
Change-Id: I25ab2e144b63d82b4cc488214bd7c72c3af8bfef
Reviewed-on: https://chromium-review.googlesource.com/c/1411832
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19986}
[modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/test/api_common_test.py
[modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/test/config_test.py
[modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/test/test_util.py
[modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/test/api_test.py
[modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/test/creation_test.py
[modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/test/metrics_test.py
[modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/swarming/test/swarmbucket_api_test.py
[modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/access/test/api_test.py
[modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/test/service_test.py

Comment 36 by bugdroid1@chromium.org, Jan 15

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/aba26554e528a6e800870c973129e559373c2e96

commit aba26554e528a6e800870c973129e559373c2e96
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Jan 15 23:36:21 2019

[buildbucket] Add RequestedDimension message

ScheduleBuildRequest.dimensions needs an expiration per dimension.
Add RequestedDimension message (key, value, expiration) and use it in
ScheduleBuildRequest.dimensions.

Bug:  875056 
Change-Id: I4e2729385403809f0878264399fd5ceab7c66254
Reviewed-on: https://chromium-review.googlesource.com/c/1412223
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/aba26554e528a6e800870c973129e559373c2e96/buildbucket/proto/pb.discovery.go
[modify] https://crrev.com/aba26554e528a6e800870c973129e559373c2e96/buildbucket/proto/rpc.pb.go
[modify] https://crrev.com/aba26554e528a6e800870c973129e559373c2e96/buildbucket/proto/rpc.proto

Comment 37 by no...@chromium.org, Jan 16

Blockedon: 917851
i cannot continue until model.Build stores the proto itself ( issue 917851 ) otherwise there is no place to store a list of gerrit changes...

Comment 38 by no...@chromium.org, Jan 16

Blocking: -917851

Comment 39 by bugdroid1@chromium.org, Jan 16

Project Member

Comment 40 by bugdroid, Jan 25

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/fc28e4ae8391918cb0b1b35a1dab24d0e714556e

commit fc28e4ae8391918cb0b1b35a1dab24d0e714556e
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jan 25 20:31:40 2019

[buildbucket] Apply requested_dimensions

Apply dimensions specified in ScheduleBuildRequest.

Bug:  875056 
Change-Id: Iaf82f89104715522b4d77ca72fea8d9707bea233
Reviewed-on: https://chromium-review.googlesource.com/c/1436846
Auto-Submit: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Cr-Commit-Position: refs/heads/master@{#20230}
[modify] https://crrev.com/fc28e4ae8391918cb0b1b35a1dab24d0e714556e/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/fc28e4ae8391918cb0b1b35a1dab24d0e714556e/appengine/cr-buildbucket/swarming/test/swarming_test.py

Comment 41 by no...@chromium.org, Jan 29

Status: Fixed (was: Started)

Sign in to add a comment