New issue
Advanced search Search tips

Issue 853450 link

Starred by 0 users

Issue metadata

Status: Duplicate
Merged: issue 908482
Owner:
Closed: Dec 4
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Feature

Blocked on:
issue 854792

Blocking:
issue 851585
issue 903372
issue 904271
issue 811395
issue 812021
issue 843714



Sign in to add a comment

Refine buildbucket-kitchen communication

Project Member Reported by no...@chromium.org, Jun 16 2018

Issue description

make the following happen:
- buildbucket sends input to kitchen in a form of buildbucket.v2.Build message
- kitchen sends output back to buildbucket in a form of buildbucket.v2.Build message, a subset thereof. Not annotation.proto.
- kitchen sends output via direct RPCs to buildbucket (not logdog, not pubsub, not isolate) each time there is something new (e.g. new step)

this moves some responsibilities from buidbucket to kitchen, including (annotation.proto -> Build) conversion. This removes annotation.proto from buildbucket<->kitchen protocols. kitchen will encapsulate annotations. This also removes recipe-specific, kitchen-specific stuff from the protocol.

This establishes a generic protocol for potential alternative "build executors", e.g. like an appengine app that executes a build in a special way (FindIt's "Virtual tester", DM), or something on a bot, but not a recipe.

This is prerequisite for real-time step info ( issue 843714 ) and indexing by commit position (issue 851585)

Design TBD.
 

Comment 1 by no...@chromium.org, Jun 16 2018

Description: Show this description

Comment 2 by no...@chromium.org, Jun 16 2018

Blocking: 851585

Comment 3 by no...@chromium.org, Jun 16 2018

Blocking: 812021

Comment 4 by no...@chromium.org, Jun 20 2018

Blockedon: 854792
Project Member

Comment 5 by bugdroid1@chromium.org, Jun 26 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/24d76effaebc11983b2f29abc3f4a83d3c819c6d

commit 24d76effaebc11983b2f29abc3f4a83d3c819c6d
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Jun 26 22:20:21 2018

[buildbucket] Add annotations.py

Move all annotation-specific to code annotations.py
This is pure code refactoring.

Bug:  853450 
Change-Id: I18b405be56a54aa9a2159b1de2850bc6e4fb2bdc
Reviewed-on: https://chromium-review.googlesource.com/1115726
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/24d76effaebc11983b2f29abc3f4a83d3c819c6d/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/24d76effaebc11983b2f29abc3f4a83d3c819c6d/appengine/cr-buildbucket/v2/api.py
[modify] https://crrev.com/24d76effaebc11983b2f29abc3f4a83d3c819c6d/appengine/cr-buildbucket/bq.py
[modify] https://crrev.com/24d76effaebc11983b2f29abc3f4a83d3c819c6d/appengine/cr-buildbucket/v2/__init__.py
[rename] https://crrev.com/24d76effaebc11983b2f29abc3f4a83d3c819c6d/appengine/cr-buildbucket/annotations.py
[rename] https://crrev.com/24d76effaebc11983b2f29abc3f4a83d3c819c6d/appengine/cr-buildbucket/test/annotations.pb.txt
[modify] https://crrev.com/24d76effaebc11983b2f29abc3f4a83d3c819c6d/appengine/cr-buildbucket/test/bq_test.py
[modify] https://crrev.com/24d76effaebc11983b2f29abc3f4a83d3c819c6d/appengine/cr-buildbucket/v2/test/api_test.py
[modify] https://crrev.com/24d76effaebc11983b2f29abc3f4a83d3c819c6d/appengine/cr-buildbucket/swarming/test/swarming_test.py
[rename] https://crrev.com/24d76effaebc11983b2f29abc3f4a83d3c819c6d/appengine/cr-buildbucket/test/annotations_test.py
[modify] https://crrev.com/24d76effaebc11983b2f29abc3f4a83d3c819c6d/appengine/cr-buildbucket/model.py
[rename] https://crrev.com/24d76effaebc11983b2f29abc3f4a83d3c819c6d/appengine/cr-buildbucket/test/expected_steps.pb.txt

Project Member

Comment 6 by bugdroid1@chromium.org, Jun 27 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/6f75dc3fa65ec7026758286b8b8a5ca4758427bb

commit 6f75dc3fa65ec7026758286b8b8a5ca4758427bb
Author: Nodir Turakulov <nodir@google.com>
Date: Wed Jun 27 21:10:35 2018

[buildbucket] Store steps in v2 format

Start storing build steps in v2 format, as a binary protobuf messages contained
in a buildbucket.v2.Build, in new datastore entity "BuildSteps".

Leave BuildAnnotations entity until we migrate to the new schema.

Bug:  853450 ,  856724 
Change-Id: I4e48f71707d59d635faac544e5f6ead64bde384a
Reviewed-on: https://chromium-review.googlesource.com/1115766
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/6f75dc3fa65ec7026758286b8b8a5ca4758427bb/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/6f75dc3fa65ec7026758286b8b8a5ca4758427bb/appengine/cr-buildbucket/bq.py
[modify] https://crrev.com/6f75dc3fa65ec7026758286b8b8a5ca4758427bb/appengine/cr-buildbucket/annotations.py
[modify] https://crrev.com/6f75dc3fa65ec7026758286b8b8a5ca4758427bb/appengine/cr-buildbucket/v2/api.py
[modify] https://crrev.com/6f75dc3fa65ec7026758286b8b8a5ca4758427bb/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/6f75dc3fa65ec7026758286b8b8a5ca4758427bb/appengine/cr-buildbucket/model.py

Comment 7 by no...@chromium.org, Jun 28 2018

Status: Started (was: Assigned)
Project Member

Comment 8 by bugdroid1@chromium.org, Jun 29 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/799ecab64b59a2430ed9e6b215173c52e20fad79

commit 799ecab64b59a2430ed9e6b215173c52e20fad79
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jun 29 14:41:43 2018

[protoutil] Add support for update mask parsing

The advanced field mask symantics does not apply to update masks.
Such masks are simple: a repeated field may be only the last field in a paths
string.

R=smut@chromium.org

Bug:  853450 
Change-Id: I85e07b496989801a2d8b2997d05eb8ef294551b9
Reviewed-on: https://chromium-review.googlesource.com/1119366
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: smut <smut@google.com>

[modify] https://crrev.com/799ecab64b59a2430ed9e6b215173c52e20fad79/appengine/components/components/protoutil/field_masks.py
[modify] https://crrev.com/799ecab64b59a2430ed9e6b215173c52e20fad79/appengine/components/components/protoutil/field_masks_test.py

Project Member

Comment 9 by bugdroid1@chromium.org, Jun 29 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/42a5d4d44d04d1caaaef819ddaa460df9d711fae

commit 42a5d4d44d04d1caaaef819ddaa460df9d711fae
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jun 29 14:45:44 2018

[buildbucket] Generalize bulkproc for any Build descendant

Will be used to convert all BuildAnnotations to BuildSteps

Bug:  853450 
Change-Id: Iab2ed79e9a7a8d4a05756d9f737ff7b2593a2d4e
Reviewed-on: https://chromium-review.googlesource.com/1118711
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/42a5d4d44d04d1caaaef819ddaa460df9d711fae/appengine/cr-buildbucket/bulkproc.py
[modify] https://crrev.com/42a5d4d44d04d1caaaef819ddaa460df9d711fae/appengine/cr-buildbucket/test/bulkproc_test.py

Project Member

Comment 10 by bugdroid1@chromium.org, Jun 29 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/7b24b6f5e0857c227f32336e9aa6c3b62cf023a5

commit 7b24b6f5e0857c227f32336e9aa6c3b62cf023a5
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jun 29 22:15:11 2018

[buildbucket] Populate BuildSteps from BuildAnnotations

For each BuildAnnotations entity, put BuildSteps entity.
Leave BuildAnnotations for the time being. We will delete them all at once
using Datastore tools.

This CL doesn't have tests because it is one off and was tested on dev,
does not delete data and doesn't write data that is currently being used.

Bug:  853450 
Change-Id: I8b5c7aa981da9845874ac75f427fbff95ce47815
Reviewed-on: https://chromium-review.googlesource.com/1119092
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/7b24b6f5e0857c227f32336e9aa6c3b62cf023a5/appengine/cr-buildbucket/annotations.py
[modify] https://crrev.com/7b24b6f5e0857c227f32336e9aa6c3b62cf023a5/appengine/cr-buildbucket/bulkproc.py
[add] https://crrev.com/7b24b6f5e0857c227f32336e9aa6c3b62cf023a5/appengine/cr-buildbucket/backfill_build_steps.py
[modify] https://crrev.com/7b24b6f5e0857c227f32336e9aa6c3b62cf023a5/appengine/cr-buildbucket/test/bulkproc_test.py
[modify] https://crrev.com/7b24b6f5e0857c227f32336e9aa6c3b62cf023a5/appengine/cr-buildbucket/api.py

Project Member

Comment 11 by bugdroid1@chromium.org, Jun 29 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/9bd9c0538bd3ebad02eb61d671885ade48fd5aa0

commit 9bd9c0538bd3ebad02eb61d671885ade48fd5aa0
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jun 29 23:07:21 2018

[buildbucket] Skip large BuildSteps

BuildAnnotations are stored compressed. BuildSteps are not.
In the past buildbucket did not limit size of annotations, so BuildAnnotations
are large.
https://chromium-review.googlesource.com/c/infra/infra/+/1044283
changed that. Since then such build are marked as infra failure.

Skip builds with large steps.

Bug:  853450 
Change-Id: I7c988bac221ccd7d67c59a0a05168df1ad5ef132
Reviewed-on: https://chromium-review.googlesource.com/1121341
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/9bd9c0538bd3ebad02eb61d671885ade48fd5aa0/appengine/cr-buildbucket/backfill_build_steps.py

Project Member

Comment 13 by bugdroid1@chromium.org, Jun 29 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/b249bd2d1f201d53551662b24b7a784589e13ef5

commit b249bd2d1f201d53551662b24b7a784589e13ef5
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jun 29 23:45:21 2018

[buildbucket] Fix reference to RequestTooLargeError

RequestTooLargeError is defined in google.appengine.runtime.apiproxy_errors,
not ndb

TBR=vadimsh@chromium.org

Bug:  853450 
Change-Id: Id6d056b02b311729afc66b33d826d3fd3770ce18
Reviewed-on: https://chromium-review.googlesource.com/1121417
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/b249bd2d1f201d53551662b24b7a784589e13ef5/appengine/cr-buildbucket/backfill_build_steps.py

Project Member

Comment 14 by bugdroid1@chromium.org, Jun 30 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/1211a855851875cc1daf4dd20fa4787014c7320b

commit 1211a855851875cc1daf4dd20fa4787014c7320b
Author: Nodir Turakulov <nodir@google.com>
Date: Sat Jun 30 00:14:11 2018

[buildbucket] Handle TombstonedTaskError

If a task T0 enqueued T1 which finished, and then T0 gets scheduled again,
its attempt to schedule T1 again will fail with TombstonedTaskError
error. Swallow such error, like we already swallow TaskAlreadyExistsError.

TBR=vadimsh@chromium.org

Bug:  853450 
Change-Id: Ib48f49193450cab3e1e74cebe73d13110aceec50
Reviewed-on: https://chromium-review.googlesource.com/1121536
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/1211a855851875cc1daf4dd20fa4787014c7320b/appengine/cr-buildbucket/bulkproc.py

Project Member

Comment 15 by bugdroid1@chromium.org, Jul 10

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/1c42880c5dfa10f5fde645bf08742351b2aeccb3

commit 1c42880c5dfa10f5fde645bf08742351b2aeccb3
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Jul 10 03:30:30 2018

[buildbucket] Define model.BuildSteps.MAX_STEPS_LEN

Define a new limit at the BuildSteps level.

Bug:  853450 
Change-Id: Ic402805b24b4f8eae46c2eaa0e11f0706cc78257
Reviewed-on: https://chromium-review.googlesource.com/1121620
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>

[modify] https://crrev.com/1c42880c5dfa10f5fde645bf08742351b2aeccb3/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/1c42880c5dfa10f5fde645bf08742351b2aeccb3/appengine/cr-buildbucket/backfill_build_steps.py
[modify] https://crrev.com/1c42880c5dfa10f5fde645bf08742351b2aeccb3/appengine/cr-buildbucket/model.py

Project Member

Comment 16 by bugdroid1@chromium.org, Jul 26

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/54ac0334e644bdf5c39bc2d25d93f789a5437fab

commit 54ac0334e644bdf5c39bc2d25d93f789a5437fab
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Jul 26 22:22:58 2018

[buildbucket] Add UpdateBuild RPC

See  crbug.com/853450  for motivation.
Initially this will be updated by kitchen to provide real-time step information.
Later we will expand update mask to support other fields and completely
eliminate build_run_result.json file that is currently dumped by kitchen
to an isolate dir and then consumed by buildbucket on completion.

When scheduling a task, buildbucket will pass a build-specific secret to the
swarming task and then expect that secret in UpdateBuild RPC.
The secret will be passed via HTTP headers / RPC metadata, to be consistent
with how we do OAuth 2.0 and delegation token auth.

Bug:  853450 
Change-Id: I73bbe3fe7b915bb9b40362b0a860c87ee6599868
Reviewed-on: https://chromium-review.googlesource.com/1119349
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/54ac0334e644bdf5c39bc2d25d93f789a5437fab/buildbucket/proto/pb.discovery.go
[modify] https://crrev.com/54ac0334e644bdf5c39bc2d25d93f789a5437fab/buildbucket/proto/rpc.pb.go
[modify] https://crrev.com/54ac0334e644bdf5c39bc2d25d93f789a5437fab/buildbucket/proto/rpc.proto

Project Member

Comment 17 by bugdroid1@chromium.org, Jul 26

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4

commit ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Jul 26 23:11:38 2018

[buildbucket] Remove annotations.BuildAnnotations

Build steps are not stored in model.BuildSteps.
model.BuildAnnotations are not longer needed. Remove it.

Bug:  853450 

Change-Id: I11a0f5c347a86959101c0f516d16d3af2a39ddc3
Reviewed-on: https://chromium-review.googlesource.com/1119163
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4/appengine/cr-buildbucket/test/bq_test.py
[modify] https://crrev.com/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4/appengine/cr-buildbucket/test/search_test.py
[modify] https://crrev.com/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4/appengine/cr-buildbucket/bq.py
[modify] https://crrev.com/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4/appengine/cr-buildbucket/v2/builds.py
[modify] https://crrev.com/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4/appengine/cr-buildbucket/annotations.py
[modify] https://crrev.com/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4/appengine/cr-buildbucket/test/user_test.py
[modify] https://crrev.com/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4/appengine/cr-buildbucket/test/annotations_test.py
[modify] https://crrev.com/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4/appengine/cr-buildbucket/v2/test/api_test.py
[modify] https://crrev.com/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4/appengine/cr-buildbucket/v2/api.py
[modify] https://crrev.com/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4/appengine/cr-buildbucket/v2/test/builds_test.py
[modify] https://crrev.com/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4/appengine/cr-buildbucket/test/bulkproc_test.py
[modify] https://crrev.com/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4/appengine/cr-buildbucket/api.py
[delete] https://crrev.com/b1db84920264d657daf9d8a173a6b0404369f9af/appengine/cr-buildbucket/backfill_build_steps.py
[modify] https://crrev.com/ac7a86522fa1ddfba52ef48f54a0e9e3e1f069f4/appengine/cr-buildbucket/model.py

Cc: no...@chromium.org
Owner: jchin...@chromium.org
Cc: iannu...@google.com
Cc: -iannucci@chromium.org
Blocking: 903372
Blocking: 904271
Owner: no...@chromium.org
Status: Assigned (was: Started)
most of this is done, but the legacy pipeline is still used. The remaining work is to switch to UpdateBuild completely.
Mergedinto: 908482
Status: Duplicate (was: Assigned)
i believe bug 908482 represents the rest of the work in this bug.

Sign in to add a comment