Implement and deploy ScheduleBuild v2 RPC from go/buildbucket-api-v2
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/61fc6b56155ff2074f1989a6d5a0628986df1ad0 commit 61fc6b56155ff2074f1989a6d5a0628986df1ad0 Author: Nodir Turakulov <nodir@google.com> Date: Tue Aug 21 16:01:53 2018 [buildbucket] Add ScheduleBuild RPC Define ScheduleBuild RPC with its request message. This API supersedes current https://apis-explorer.appspot.com/apis-explorer/?base=https://cr-buildbucket-dev.appspot.com/_ah/api#p/buildbucket/v1/buildbucket.put This CL does not yet add "dynamic builders" support. Bug: 875056 Change-Id: Id6f995e9b1cc238780c58931e578d0778650912b Reviewed-on: https://chromium-review.googlesource.com/1178923 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Reviewed-by: Jason Clinton <jclinton@chromium.org> Reviewed-by: Shuotao Gao <stgao@chromium.org> Reviewed-by: Roberto Carrillo <robertocn@chromium.org> [modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/build.pb.go [modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/build.proto [modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/common.pb.go [modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/common.proto [modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/config/project_config.pb.go [modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/config/project_config.proto [modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/config/service_config.pb.go [add] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/notification.pb.go [add] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/notification.proto [modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/pb.discovery.go [modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/rpc.mock.pb.go [modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/rpc.pb.go [modify] https://crrev.com/61fc6b56155ff2074f1989a6d5a0628986df1ad0/buildbucket/proto/rpc.proto
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/09d87c18d938627e494fe76835abe22558a1c97a commit 09d87c18d938627e494fe76835abe22558a1c97a Author: Nodir Turakulov <nodir@google.com> Date: Thu Aug 23 17:41:38 2018 [buildbucket] Allow overriding builder of template build ScheduleBuildRequest currently requires that template_build_id and builder are mutually exclusive. Remove this requirement. Bug: 875056 Change-Id: I674cd69510a65c1446d963ce49f3378216833cb3 Reviewed-on: https://chromium-review.googlesource.com/1186035 Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> [modify] https://crrev.com/09d87c18d938627e494fe76835abe22558a1c97a/buildbucket/proto/pb.discovery.go [modify] https://crrev.com/09d87c18d938627e494fe76835abe22558a1c97a/buildbucket/proto/rpc.pb.go [modify] https://crrev.com/09d87c18d938627e494fe76835abe22558a1c97a/buildbucket/proto/rpc.proto
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/5d58f4aebf92273a3ee73a54688b171ece780dc9 commit 5d58f4aebf92273a3ee73a54688b171ece780dc9 Author: Nodir Turakulov <nodir@google.com> Date: Fri Aug 24 00:02:46 2018 [buildbucket] Add ScheduleBuildRequest validation Also prohibit properties that are superseded by new typed fields, e.g. gitiles_commit supersedes repository, branch and revision properties. See crbug.com/877161 Bug: 875056, 877161 Change-Id: I1da3730dd1073eca91d5f585ac667d6d3a2b944f Reviewed-on: https://chromium-review.googlesource.com/1186063 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Reviewed-by: Erik Staab <estaab@chromium.org> [modify] https://crrev.com/5d58f4aebf92273a3ee73a54688b171ece780dc9/appengine/cr-buildbucket/v2/test/validation_test.py [modify] https://crrev.com/5d58f4aebf92273a3ee73a54688b171ece780dc9/appengine/cr-buildbucket/v2/validation.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/23031707dce2731f547b911ad762f76943701f88 commit 23031707dce2731f547b911ad762f76943701f88 Author: Nodir Turakulov <nodir@google.com> Date: Thu Aug 30 14:51:40 2018 [buildbucket] Store input GitilesCommit Buildbucket currently synthesizes v2 Build.input.gitiles_commit from tags. However, new ScheduleBuild API allows specifying ref and position, which are not present in the tags. Add new Build datastore property that stores the input GitilesCommit in binary form. Use it when converting a model.Build to Build message. Extend creation.BuildRequest to include gitiles_commit. It will be used in v2 ScheduleBuild implementation. R=tandrii@chromiium.org Bug: 875056 Change-Id: I85e6d129dc2bf8932efe6c67e1162fbf95029662 Reviewed-on: https://chromium-review.googlesource.com/1187651 Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> Cr-Commit-Position: refs/heads/master@{#17312} [modify] https://crrev.com/23031707dce2731f547b911ad762f76943701f88/appengine/cr-buildbucket/v2/builds.py [modify] https://crrev.com/23031707dce2731f547b911ad762f76943701f88/appengine/cr-buildbucket/model.py [modify] https://crrev.com/23031707dce2731f547b911ad762f76943701f88/appengine/cr-buildbucket/test/creation_test.py [modify] https://crrev.com/23031707dce2731f547b911ad762f76943701f88/appengine/cr-buildbucket/creation.py
unblocking this from issue 877161 (stop using deprecated properties in recipes). we will implement this before updating recipes. The new API won't be usable for recipes that depend on deprecated properties.
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/3b6791a2b2b73127adbc9b4ad760b212511a6312 commit 3b6791a2b2b73127adbc9b4ad760b212511a6312 Author: Nodir Turakulov <nodir@google.com> Date: Tue Dec 11 00:03:51 2018 [buildbucket] simplify BuildRequest.normalize 99% of what BuildRequest.normalize does is validation, and normalization is trivial. Rename to validate() and do not return a new BuildRequest. Bug: 875056 Change-Id: I846f57c4adc8bc6fcad15d41189528cac862c1f4 Reviewed-on: https://chromium-review.googlesource.com/c/1370557 Commit-Queue: Vadim Shtayura <vadimsh@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Auto-Submit: Nodir Turakulov <nodir@chromium.org> Cr-Commit-Position: refs/heads/master@{#19469} [modify] https://crrev.com/3b6791a2b2b73127adbc9b4ad760b212511a6312/appengine/cr-buildbucket/swarming/swarmbucket_api.py [modify] https://crrev.com/3b6791a2b2b73127adbc9b4ad760b212511a6312/appengine/cr-buildbucket/creation.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/9e3690566f676908f39f172570faecf78b8d6c25 commit 9e3690566f676908f39f172570faecf78b8d6c25 Author: Nodir Turakulov <nodir@google.com> Date: Tue Dec 11 19:27:19 2018 [luci_notify] Read email_notify from properties In the past we've decided to pass email_notify list via parameters, as opposed to properties. V2 does not have parameters, only properties, and there are no other use cases to extend go/build-proto. It is possible that a recipe may want to mutate the email_notify list, e.g. define who to notify, thus it makes sense to have the email notify list in build output too. To avoid polluting go/build-proto for a luci-notify-specific feature, keep it a "schemaless" property, as opposed to a proto field in go/build-proto. Parse email_notify from output. If it isn't there, read it from input. Otherwise fallback to parameters. ScheduleBuild v2 API does not allow passing parameters, so eventually users will be forced to switch to properties. Bug: 875056 Change-Id: I36da0eb11743f368424132ccb5c8bda37d0ffde6 Reviewed-on: https://chromium-review.googlesource.com/c/1370610 Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> [modify] https://crrev.com/9e3690566f676908f39f172570faecf78b8d6c25/luci_notify/notify/pubsub.go [modify] https://crrev.com/9e3690566f676908f39f172570faecf78b8d6c25/luci_notify/notify/pubsub_test.go
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/a6837f49bed316e1a08cecfa76a85eb5f2d7e5d8 commit a6837f49bed316e1a08cecfa76a85eb5f2d7e5d8 Author: Nodir Turakulov <nodir@google.com> Date: Wed Dec 12 15:33:32 2018 [buildbucket] Store input properties as a struct Store input properties in proto binary format. Start writing the entity on build creation. R=vadimsh@chromium.org Bug: 875056 Change-Id: Iacdb53f7c6440c66adbd29fbf669c2634bc8760d Reviewed-on: https://chromium-review.googlesource.com/c/1372341 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Auto-Submit: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Cr-Commit-Position: refs/heads/master@{#19513} [modify] https://crrev.com/a6837f49bed316e1a08cecfa76a85eb5f2d7e5d8/appengine/cr-buildbucket/swarming/swarming.py [modify] https://crrev.com/a6837f49bed316e1a08cecfa76a85eb5f2d7e5d8/appengine/cr-buildbucket/v2/builds.py [modify] https://crrev.com/a6837f49bed316e1a08cecfa76a85eb5f2d7e5d8/appengine/cr-buildbucket/model.py [modify] https://crrev.com/a6837f49bed316e1a08cecfa76a85eb5f2d7e5d8/appengine/cr-buildbucket/test/creation_test.py [modify] https://crrev.com/a6837f49bed316e1a08cecfa76a85eb5f2d7e5d8/appengine/cr-buildbucket/creation.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/197f6943642e554c4b98d1349a2128d8cb8bcf0f commit 197f6943642e554c4b98d1349a2128d8cb8bcf0f Author: Nodir Turakulov <nodir@google.com> Date: Wed Dec 12 15:47:42 2018 [buildbucket] Store output properties separately Store output properties in a separate entity, in proto binary format. Start writing the entity on build update. Reading part of the code will be updated later. Unlike input properties, output properties are much larger, and thus it makes sense to store them separately, as an optimization. R=vadimsh@chromium.org Bug: 875056 Change-Id: If15a8983978c642ef5578aa3a4833e6192f23f93 Reviewed-on: https://chromium-review.googlesource.com/c/1372031 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Auto-Submit: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Cr-Commit-Position: refs/heads/master@{#19514} [modify] https://crrev.com/197f6943642e554c4b98d1349a2128d8cb8bcf0f/appengine/cr-buildbucket/service.py [modify] https://crrev.com/197f6943642e554c4b98d1349a2128d8cb8bcf0f/appengine/cr-buildbucket/test/service_test.py [modify] https://crrev.com/197f6943642e554c4b98d1349a2128d8cb8bcf0f/appengine/cr-buildbucket/v2/api.py [modify] https://crrev.com/197f6943642e554c4b98d1349a2128d8cb8bcf0f/appengine/cr-buildbucket/model.py [modify] https://crrev.com/197f6943642e554c4b98d1349a2128d8cb8bcf0f/appengine/cr-buildbucket/v2/test/api_test.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/896de66b8b44c7d1a13ac1c8f395511ed0a7e246 commit 896de66b8b44c7d1a13ac1c8f395511ed0a7e246 Author: Nodir Turakulov <nodir@google.com> Date: Wed Dec 12 15:53:22 2018 [buildbucket] Backfill property entities Backfill BuildInputProperties and BuildOutputProperties entities. Repurpose put_builds to fix_builds to do arbitrary transformations. R=vadimsh@chromium.org Bug: 875056 Change-Id: Ib018dd2d06f9815f830cace579ff1d42e7d8deaf Reviewed-on: https://chromium-review.googlesource.com/c/1372739 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Auto-Submit: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Cr-Commit-Position: refs/heads/master@{#19515} [add] https://crrev.com/896de66b8b44c7d1a13ac1c8f395511ed0a7e246/appengine/cr-buildbucket/fix_builds.py [delete] https://crrev.com/197f6943642e554c4b98d1349a2128d8cb8bcf0f/appengine/cr-buildbucket/reput_builds.py [modify] https://crrev.com/896de66b8b44c7d1a13ac1c8f395511ed0a7e246/appengine/cr-buildbucket/api.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/924cd595d8b6a142178a2877a4e99ac565661129 commit 924cd595d8b6a142178a2877a4e99ac565661129 Author: Nodir Turakulov <nodir@google.com> Date: Wed Dec 12 16:52:33 2018 Fix fix_builds.py Forgot to upload a fix to the uploaded patchset. TBR=vadimsh@chromium.org Bug: 875056 Change-Id: Idb72cfb06e74e7f78f73c39dc1a16a91287d6e0c Reviewed-on: https://chromium-review.googlesource.com/c/1373991 Reviewed-by: Nodir Turakulov <nodir@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> Cr-Commit-Position: refs/heads/master@{#19517} [modify] https://crrev.com/924cd595d8b6a142178a2877a4e99ac565661129/appengine/cr-buildbucket/fix_builds.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/8fe09ae98917c44b29ed8d8441625e43f4615d9e commit 8fe09ae98917c44b29ed8d8441625e43f4615d9e Author: Nodir Turakulov <nodir@google.com> Date: Thu Dec 13 19:56:37 2018 [buildbucket] Log integer properties We've discovered that google.protobuf.Struct is a lossy format compared to raw JSON: Struct encodes all numbers are floats, i.e. converts `0` to `0.0`. This may break recipes. Log integer properties to estimate the impact. This CL will be reverted. Bug: 875056 Change-Id: I70eafc225c02d7945dfc704aac590ccbc2e440cf Reviewed-on: https://chromium-review.googlesource.com/c/1376949 Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> Cr-Commit-Position: refs/heads/master@{#19542} [modify] https://crrev.com/8fe09ae98917c44b29ed8d8441625e43f4615d9e/appengine/cr-buildbucket/test/creation_test.py [modify] https://crrev.com/8fe09ae98917c44b29ed8d8441625e43f4615d9e/appengine/cr-buildbucket/creation.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/6168d72a7f46884768262a5e6929eba8c08df269 commit 6168d72a7f46884768262a5e6929eba8c08df269 Author: Nodir Turakulov <nodir@google.com> Date: Thu Dec 13 20:28:21 2018 [buildbucket] Split fix_builds RPCs. fix_builds.py is doing one large RPC and it hits RPC size limits. Split the RPC into two, runs them in parallel. Bug: 875056 Change-Id: I55ec89be5d7511deb26a72386dba2c0d6bd4b11d Reviewed-on: https://chromium-review.googlesource.com/c/1376950 Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> Cr-Commit-Position: refs/heads/master@{#19544} [modify] https://crrev.com/6168d72a7f46884768262a5e6929eba8c08df269/appengine/cr-buildbucket/fix_builds.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/35dc0c37bc6d2b8af8c4a071ebb3f8b30a38da63 commit 35dc0c37bc6d2b8af8c4a071ebb3f8b30a38da63 Author: Nodir Turakulov <nodir@google.com> Date: Thu Dec 13 21:16:16 2018 [buildbucket] Check that properties is dict fix_builds: ignore output properties that are not dict. Bug: 875056 Change-Id: I48436a98a2bc4cb7ae0552c157508127e0d4241a Reviewed-on: https://chromium-review.googlesource.com/c/1376952 Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> Cr-Commit-Position: refs/heads/master@{#19550} [modify] https://crrev.com/35dc0c37bc6d2b8af8c4a071ebb3f8b30a38da63/appengine/cr-buildbucket/fix_builds.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/c0216c0041ff14aa6961f4d48bb48f356dbd7e68 commit c0216c0041ff14aa6961f4d48bb48f356dbd7e68 Author: Nodir Turakulov <nodir@google.com> Date: Fri Dec 14 02:32:10 2018 [buildbucket] Check input properties is dict The non-dict properties also exist in parameters. Check their type too. Bug: 875056 Change-Id: Ia6db0bbdb68ffd65b5c790387e2379e7a9bf1a3d Reviewed-on: https://chromium-review.googlesource.com/c/1376797 Auto-Submit: Nodir Turakulov <nodir@chromium.org> Commit-Queue: Vadim Shtayura <vadimsh@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Cr-Commit-Position: refs/heads/master@{#19563} [modify] https://crrev.com/c0216c0041ff14aa6961f4d48bb48f356dbd7e68/appengine/cr-buildbucket/fix_builds.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/98795d43019524c872f6fa2f4d468eccedbbbde8 commit 98795d43019524c872f6fa2f4d468eccedbbbde8 Author: Nodir Turakulov <nodir@google.com> Date: Fri Dec 14 20:18:52 2018 [buildbucket] Fix build parts sequentially ndb has put batcher that combines puts. Do puts sequentially. Bug: 875056 Change-Id: If7d8e9079c7a8ecd25d503738ed4a4d3626ebfe1 Reviewed-on: https://chromium-review.googlesource.com/c/1378786 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Commit-Queue: Vadim Shtayura <vadimsh@chromium.org> Auto-Submit: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Cr-Commit-Position: refs/heads/master@{#19567} [modify] https://crrev.com/98795d43019524c872f6fa2f4d468eccedbbbde8/appengine/cr-buildbucket/fix_builds.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f commit aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f Author: Nodir Turakulov <nodir@google.com> Date: Tue Dec 18 02:28:13 2018 [buildbucket] Remove Build.parameters_actual parameters_actual was needed because server might have change user-specified parameters. Now server uses Build.input_properties, which is allowed to have server-defined properties. Update all usages to use Build.input_properties instead. This has a non-obvious side effect. Struct message in the latest version of protobuf-python does not accept integers. All numbers are floats, and they are rendered as doubles in JSON (0 -> 0.0). In general, this is a breaking change. crbug.com/914996 updates recipes to tolerate floats where it used to expect integers, but this CL also has a whitelist of widely used, deprecated properties (like buildnumber) which buildbucket converts back to int. Updating all recipes to use that is a lot of work, and recipes must migrate to new recipe API anyway: crbug.com/877161 R=vadimsh@chromium.org Bug: 875056 Change-Id: I39979d3541c12542bdf572ebafda5126d56d033b Reviewed-on: https://chromium-review.googlesource.com/c/1373312 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Cr-Commit-Position: refs/heads/master@{#19627} [modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/swarming/swarming.py [modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/fix_builds.py [modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/test/api_common_test.py [modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/api_common.py [modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/v2/builds.py [modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/v2/test/api_test.py [modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/v2/test/builds_test.py [add] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/bbutil.py [modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/swarming/test/swarmbucket_api_test.py [modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/swarming/test/swarming_test.py [modify] https://crrev.com/aeb421a02de5fb4bd9f92f2addbce69dce2c6d7f/appengine/cr-buildbucket/model.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/e0201aaefff27fe54c21120b052cb19b456087d6 commit e0201aaefff27fe54c21120b052cb19b456087d6 Author: Nodir Turakulov <nodir@google.com> Date: Wed Dec 19 00:33:37 2018 [buildbucket] Switch to input_properties Update code that reads input properties from expecting 'properties' parameter to reading model.Build.input_properties. This changes the v1 API a bit. API responses now include input properties defined on the server. This should have no effect though. R=vadimsh@chromium.org Bug: 875056 Change-Id: Id5435893303793cb562485d45f00d103bc993516 Reviewed-on: https://chromium-review.googlesource.com/c/1374000 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Cr-Commit-Position: refs/heads/master@{#19654} [modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/swarming/swarming.py [modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/test/api_common_test.py [modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/api_common.py [modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/test/api_test.py [modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/test/notifications_test.py [modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/swarming/test/swarming_test.py [modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/bbutil.py [modify] https://crrev.com/e0201aaefff27fe54c21120b052cb19b456087d6/appengine/cr-buildbucket/model.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/48881a12617a58e106222f6c675a6afcc6cbec05 commit 48881a12617a58e106222f6c675a6afcc6cbec05 Author: Nodir Turakulov <nodir@chromium.org> Date: Wed Dec 19 02:43:17 2018 Revert "[buildbucket] Switch to input_properties" This reverts commit e0201aaefff27fe54c21120b052cb19b456087d6. Reason for revert: crbug.com/916359 Original change's description: > [buildbucket] Switch to input_properties > > Update code that reads input properties from expecting 'properties' parameter > to reading model.Build.input_properties. > > This changes the v1 API a bit. API responses now include input properties > defined on the server. This should have no effect though. > > R=vadimsh@chromium.org > > Bug: 875056 > Change-Id: Id5435893303793cb562485d45f00d103bc993516 > Reviewed-on: https://chromium-review.googlesource.com/c/1374000 > Commit-Queue: Nodir Turakulov <nodir@chromium.org> > Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> > Cr-Commit-Position: refs/heads/master@{#19654} TBR=vadimsh@chromium.org,nodir@chromium.org Change-Id: I538a9bca666160659e2f13d01934586b50a011fb No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 875056 Reviewed-on: https://chromium-review.googlesource.com/c/1383651 Reviewed-by: Nodir Turakulov <nodir@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> Cr-Commit-Position: refs/heads/master@{#19660} [modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/swarming/swarming.py [modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/test/api_common_test.py [modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/api_common.py [modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/test/api_test.py [modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/test/notifications_test.py [modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/swarming/test/swarming_test.py [modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/bbutil.py [modify] https://crrev.com/48881a12617a58e106222f6c675a6afcc6cbec05/appengine/cr-buildbucket/model.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/ed92b5f582a1fcb23c11ab4f3b47ef0c0a8f9016 commit ed92b5f582a1fcb23c11ab4f3b47ef0c0a8f9016 Author: Nodir Turakulov <nodir@google.com> Date: Thu Dec 20 19:50:03 2018 [buildbucket] Add requested_properties Add Build.infra.buildbucket.requested_properties, the properties that were specified in ScheduleBuildRequest. This is needed for CQ to judge about CQ-suitability of a build. Note that embedding entire ScheduleBuildRequest into Build is excessive (many field values match) and hard (proto file cyclic dependency). Bug: 875056 Change-Id: I866d671454dc8f6ddaf603ac33b0351a7e76ea80 Reviewed-on: https://chromium-review.googlesource.com/c/1385674 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> [modify] https://crrev.com/ed92b5f582a1fcb23c11ab4f3b47ef0c0a8f9016/buildbucket/proto/build.pb.go [modify] https://crrev.com/ed92b5f582a1fcb23c11ab4f3b47ef0c0a8f9016/buildbucket/proto/build.proto [modify] https://crrev.com/ed92b5f582a1fcb23c11ab4f3b47ef0c0a8f9016/buildbucket/proto/pb.discovery.go
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/a535d939ef4d9f15ce38dfc25d31e83cddc732cc commit a535d939ef4d9f15ce38dfc25d31e83cddc732cc Author: Nodir Turakulov <nodir@google.com> Date: Thu Jan 03 18:20:15 2019 Reland "[buildbucket] Switch to input_properties" Update code that reads input properties from expecting 'properties' parameter to reading model.Build.input_properties. Unlike https://chromium-review.googlesource.com/c/infra/infra/+/1374000 does not make breaking changes to API. Bug: 875056 Change-Id: I0d023311f7f2490d03bbf5f59c89fb746f9c380f Reviewed-on: https://chromium-review.googlesource.com/c/1385945 Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> Cr-Commit-Position: refs/heads/master@{#19754} [modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/swarming/swarming.py [modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/test/api_common_test.py [modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/api_common.py [modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/test/api_test.py [modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/test/notifications_test.py [modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/swarming/test/swarming_test.py [modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/bbutil.py [modify] https://crrev.com/a535d939ef4d9f15ce38dfc25d31e83cddc732cc/appengine/cr-buildbucket/model.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/fbc480d33ffc14bb421b53db0146350929750ade commit fbc480d33ffc14bb421b53db0146350929750ade Author: Nodir Turakulov <nodir@google.com> Date: Fri Jan 04 20:20:41 2019 [buildbucket] Remove logging of int properties It served its purpose. R=vadimsh@chromium.org Bug: 875056 Change-Id: I5599fbb000ae78d7798eead9f2a81395499b88d7 Reviewed-on: https://chromium-review.googlesource.com/c/1383342 Auto-Submit: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> Cr-Commit-Position: refs/heads/master@{#19783} [modify] https://crrev.com/fbc480d33ffc14bb421b53db0146350929750ade/appengine/cr-buildbucket/test/creation_test.py [modify] https://crrev.com/fbc480d33ffc14bb421b53db0146350929750ade/appengine/cr-buildbucket/creation.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/767c5a8823fbff83aea5b1e7268943e3b7ece2bb commit 767c5a8823fbff83aea5b1e7268943e3b7ece2bb Author: Nodir Turakulov <nodir@google.com> Date: Thu Jan 10 19:35:03 2019 [buildbucket] Use ScheduleBuildRequest Update creation.BuildRequest to embed rpc_pb2.ScheduleBuildRequest. Leave BuildRequest for legacy stuff. Move BuildRequest validation to api.py because v2/api.py has its own validation. Change "retry" API to create a build with all input properties that were actually in the build, as opposed to properties that were specified in the request to create the build. This is the semantics of the future API. In the past it was done differently because buildbucket rejected such requests, because they attempted to use reserved properties (created by buildbucket itself). Now such builds are allowed to use reserved properties, but they would be overwritten anyway. Move "retry" implementation to api.py because v2 will use a different impl. Bug: 875056 Change-Id: I53c2c892865de52930c6d38a56c88f718fda76d4 Reviewed-on: https://chromium-review.googlesource.com/c/1375133 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Cr-Commit-Position: refs/heads/master@{#19901} [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/swarming/swarming.py [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/test/api_common_test.py [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/v2/builds.py [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/swarming/swarmingcfg.py [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/test/service_test.py [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/api.py [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/test/api_test.py [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/test/creation_test.py [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/v2/test/api_test.py [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/swarming/swarmbucket_api.py [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/creation.py [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/swarming/test/swarmbucket_api_test.py [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/bbutil.py [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/model.py [modify] https://crrev.com/767c5a8823fbff83aea5b1e7268943e3b7ece2bb/appengine/cr-buildbucket/v2/validation.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/a737e4e78db9a7d1782807b7f16779cdd32de943 commit a737e4e78db9a7d1782807b7f16779cdd32de943 Author: Nodir Turakulov <nodir@google.com> Date: Tue Jan 15 00:58:51 2019 [buildbucket] Move permission check to api.py V1 put_batch API returns a single HTTP 403 if any of the sub-requests fail due to insufficient permissions. It was done so because v1's application-level error codes did not have a code for access denials. It was too late to add them there. V2 can have a more uniform semantics. This leads to two different access check implementations. Move the current implementation to (v1) api.py. Also this fixes a deadlock bug: utils.async_apply was used in a tasklet. Bug: 875056 Change-Id: Icd50678e2146d0d9e434cc4ac932f5d579450bb6 Reviewed-on: https://chromium-review.googlesource.com/c/1409367 Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> Cr-Commit-Position: refs/heads/master@{#19973} [modify] https://crrev.com/a737e4e78db9a7d1782807b7f16779cdd32de943/appengine/cr-buildbucket/test/creation_test.py [modify] https://crrev.com/a737e4e78db9a7d1782807b7f16779cdd32de943/appengine/cr-buildbucket/creation.py [modify] https://crrev.com/a737e4e78db9a7d1782807b7f16779cdd32de943/appengine/cr-buildbucket/test/api_test.py [modify] https://crrev.com/a737e4e78db9a7d1782807b7f16779cdd32de943/appengine/cr-buildbucket/api.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/a422dd55a71c6bf2a9d74fa0696ab56a562e8ae5 commit a422dd55a71c6bf2a9d74fa0696ab56a562e8ae5 Author: Nodir Turakulov <nodir@google.com> Date: Tue Jan 15 09:06:01 2019 [buildbucket] Add batched ScheduleBuild requests. Add schedule build semantics to batch request/response messages. R=vadimsh@chromium.org Bug: 875056 Change-Id: I18a8d161783a979ee1df31979e05c8a17415f711 Reviewed-on: https://chromium-review.googlesource.com/c/1409772 Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> Auto-Submit: Nodir Turakulov <nodir@chromium.org> [modify] https://crrev.com/a422dd55a71c6bf2a9d74fa0696ab56a562e8ae5/buildbucket/proto/pb.discovery.go [modify] https://crrev.com/a422dd55a71c6bf2a9d74fa0696ab56a562e8ae5/buildbucket/proto/rpc.pb.go [modify] https://crrev.com/a422dd55a71c6bf2a9d74fa0696ab56a562e8ae5/buildbucket/proto/rpc.proto
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c commit e7ff36293e7abfbf7dc1d230e881d3ca9021c64c Author: Nodir Turakulov <nodir@google.com> Date: Tue Jan 15 18:24:10 2019 [buildbucket] Move parse_bucket_cfg to test_util parse_bucket_cfg is used often in tests. Move it to test_util.py Bug: 875056 Change-Id: I25ab2e144b63d82b4cc488214bd7c72c3af8bfef Reviewed-on: https://chromium-review.googlesource.com/c/1411832 Auto-Submit: Nodir Turakulov <nodir@chromium.org> Commit-Queue: Vadim Shtayura <vadimsh@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Cr-Commit-Position: refs/heads/master@{#19986} [modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/test/api_common_test.py [modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/test/config_test.py [modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/test/test_util.py [modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/test/api_test.py [modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/test/creation_test.py [modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/test/metrics_test.py [modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/swarming/test/swarmbucket_api_test.py [modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/access/test/api_test.py [modify] https://crrev.com/e7ff36293e7abfbf7dc1d230e881d3ca9021c64c/appengine/cr-buildbucket/test/service_test.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/aba26554e528a6e800870c973129e559373c2e96 commit aba26554e528a6e800870c973129e559373c2e96 Author: Nodir Turakulov <nodir@google.com> Date: Tue Jan 15 23:36:21 2019 [buildbucket] Add RequestedDimension message ScheduleBuildRequest.dimensions needs an expiration per dimension. Add RequestedDimension message (key, value, expiration) and use it in ScheduleBuildRequest.dimensions. Bug: 875056 Change-Id: I4e2729385403809f0878264399fd5ceab7c66254 Reviewed-on: https://chromium-review.googlesource.com/c/1412223 Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> [modify] https://crrev.com/aba26554e528a6e800870c973129e559373c2e96/buildbucket/proto/pb.discovery.go [modify] https://crrev.com/aba26554e528a6e800870c973129e559373c2e96/buildbucket/proto/rpc.pb.go [modify] https://crrev.com/aba26554e528a6e800870c973129e559373c2e96/buildbucket/proto/rpc.proto
i cannot continue until model.Build stores the proto itself (issue 917851) otherwise there is no place to store a list of gerrit changes...
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/ab0afc2622ae80256beae93a525839548c73411c commit ab0afc2622ae80256beae93a525839548c73411c Author: Nodir Turakulov <nodir@google.com> Date: Wed Jan 16 23:42:27 2019 [buildbucket] Add ScheduleBuild RPC handlers Add singular and batch variants of ScheduleBuild RPC. R=vadimsh@chromium.org Bug: 875056 Change-Id: I7ae42d7438700634228dffd7d662cd26faa66936 Reviewed-on: https://chromium-review.googlesource.com/c/1412043 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Cr-Commit-Position: refs/heads/master@{#20040} [modify] https://crrev.com/ab0afc2622ae80256beae93a525839548c73411c/infra/libs/buildbucket/proto/rpc_prpc_pb2.py [modify] https://crrev.com/ab0afc2622ae80256beae93a525839548c73411c/appengine/cr-buildbucket/v2/api.py [modify] https://crrev.com/ab0afc2622ae80256beae93a525839548c73411c/infra/libs/buildbucket/proto/rpc_pb2.py [modify] https://crrev.com/ab0afc2622ae80256beae93a525839548c73411c/appengine/cr-buildbucket/v2/test/api_test.py
Comment 1 by bugdroid1@chromium.org
, Aug 21