Milo: Swarmbucket pubsub pipeline |
|||||
Issue descriptionIn milo's buildbucket builder view, much of the information can be filled by buildbucket. However, there are three pieces of information that needs to be filled out from the actual task: * got revision * final status * step text (free-form status text for the overall build) To get these information, Milo needs to get this information from the source of truth for the build (swarming, at the moment). That means buildbucket needs to emit events for Milo to consume, when: * A build is scheduled (Non-existant -> Pending) * A build has started (Pending -> Running) * A build has exited (Pending -> Completed/Exception/etc) Milo will consume these events (from pubsub) and use it as a signal to fetch the build information from the source of truth (swarming), and fill in the builder struct to make sure this information is cached.
,
May 15 2017
I started working on the buildbuket->pubsub part
,
May 15 2017
besides build replication in Milo, this is needed for luci-migration app
,
May 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/e8508ccc54dcc4273fbadd890ebec7acce24fc82 commit e8508ccc54dcc4273fbadd890ebec7acce24fc82 Author: Nodir Turakulov <nodir@google.com> Date: Mon May 15 19:32:02 2017 buildbucket: log who requests for pubsub callback Milo and Luci-Migration apps need Buildbucket to send PubSub events for all builds. Currently Buildbucket optionally sends pubsub messages only to those who created the build. Such a pubsub message contains only build id and user-supplied data. Build id is not enough for clients subscribed to the pubsub topics receieving messages for *ALL* build. Such clients would have unconditionally load build by build id. This is a waste. We'd like to change pubsub message: instead of sending only build id, send other fields too. Clients can determine the order of messages by looking at the update_time field. To make such a breaking change, first we'd like to know who uses this feature R=tandrii@chromium.org BUG= 624960 Change-Id: Ic75b264613e3f09f799d4400e565db607202115f Reviewed-on: https://chromium-review.googlesource.com/506108 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> [modify] https://crrev.com/e8508ccc54dcc4273fbadd890ebec7acce24fc82/appengine/cr-buildbucket/service.py
,
May 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/ba036c0eda01e6322b6fec87bf1b50c31719c890 commit ba036c0eda01e6322b6fec87bf1b50c31719c890 Author: Nodir Turakulov <nodir@google.com> Date: Fri May 26 23:15:29 2017 buildbucket: replace build_id with build in pubsub messages Milo and Luci-Migration apps need Buildbucket to send PubSub events for all builds. Currently Buildbucket optionally sends pubsub messages only to those who created the build. Such a pubsub message contains only build id and user-supplied data. Build id is not enough for clients subscribed to the pubsub topics receieving messages for *ALL* build. Such clients would have unconditionally load build by build id. This is a waste. Instead of sending only build id, send other fields too. Clients can we determine the order of messages by looking at update_time field. R=iannucci@chromium.org BUG= 624960 Change-Id: Ie92ebf36e34b464249113c732c0a135a5ebbd716 Reviewed-on: https://chromium-review.googlesource.com/506036 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Roberto Carrillo <robertocn@chromium.org> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> [add] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/cr-buildbucket/test/api_common_test.py [modify] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/cr-buildbucket/events.py [modify] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/cr-buildbucket/test/events_test.py [modify] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/findit/handlers/test/try_job_push_test.py [add] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/cr-buildbucket/api_common.py [modify] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/findit/handlers/try_job_push.py [modify] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/cr-buildbucket/test/api_test.py [modify] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/cr-buildbucket/api.py [modify] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/cr-buildbucket/test/service_test.py
,
May 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/ba036c0eda01e6322b6fec87bf1b50c31719c890 commit ba036c0eda01e6322b6fec87bf1b50c31719c890 Author: Nodir Turakulov <nodir@google.com> Date: Fri May 26 23:15:29 2017 buildbucket: replace build_id with build in pubsub messages Milo and Luci-Migration apps need Buildbucket to send PubSub events for all builds. Currently Buildbucket optionally sends pubsub messages only to those who created the build. Such a pubsub message contains only build id and user-supplied data. Build id is not enough for clients subscribed to the pubsub topics receieving messages for *ALL* build. Such clients would have unconditionally load build by build id. This is a waste. Instead of sending only build id, send other fields too. Clients can we determine the order of messages by looking at update_time field. R=iannucci@chromium.org BUG= 624960 Change-Id: Ie92ebf36e34b464249113c732c0a135a5ebbd716 Reviewed-on: https://chromium-review.googlesource.com/506036 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Roberto Carrillo <robertocn@chromium.org> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> [add] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/cr-buildbucket/test/api_common_test.py [modify] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/cr-buildbucket/events.py [modify] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/cr-buildbucket/test/events_test.py [modify] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/findit/handlers/test/try_job_push_test.py [add] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/cr-buildbucket/api_common.py [modify] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/findit/handlers/try_job_push.py [modify] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/cr-buildbucket/test/api_test.py [modify] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/cr-buildbucket/api.py [modify] https://crrev.com/ba036c0eda01e6322b6fec87bf1b50c31719c890/appengine/cr-buildbucket/test/service_test.py
,
May 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/61997500662ea1aa01e7cc3a165ae20c953d7f94 commit 61997500662ea1aa01e7cc3a165ae20c953d7f94 Author: Nodir Turakulov <nodir@google.com> Date: Fri May 26 23:29:39 2017 buildbucket: global pubsub topic Publish messages for all builds in "builds" topic. The topic will be used by Milo for build replication and Luci-Migration for retries. Bug: 624960 Change-Id: Idf08f75b01fc256b82ea584ecff077f64dee3b77 Reviewed-on: https://chromium-review.googlesource.com/510033 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> [modify] https://crrev.com/61997500662ea1aa01e7cc3a165ae20c953d7f94/appengine/cr-buildbucket/test/service_test.py [modify] https://crrev.com/61997500662ea1aa01e7cc3a165ae20c953d7f94/appengine/cr-buildbucket/events.py [modify] https://crrev.com/61997500662ea1aa01e7cc3a165ae20c953d7f94/appengine/cr-buildbucket/test/events_test.py [modify] https://crrev.com/61997500662ea1aa01e7cc3a165ae20c953d7f94/appengine/cr-buildbucket/swarming/test/swarming_test.py [modify] https://crrev.com/61997500662ea1aa01e7cc3a165ae20c953d7f94/appengine/cr-buildbucket/test/api_test.py
,
May 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/8e616e381c2bc72ff74608d35c5dc5de043c87a2 commit 8e616e381c2bc72ff74608d35c5dc5de043c87a2 Author: Nodir Turakulov <nodir@chromium.org> Date: Fri May 26 23:51:09 2017 Revert "buildbucket: log who requests for pubsub callback" This reverts commit e8508ccc54dcc4273fbadd890ebec7acce24fc82. Reason for revert: served its purpose, not needed anymore Original change's description: > buildbucket: log who requests for pubsub callback > > Milo and Luci-Migration apps need Buildbucket to send PubSub events for all > builds. Currently Buildbucket optionally sends pubsub messages only to those > who created the build. Such a pubsub message contains only build id and > user-supplied data. Build id is not enough for clients subscribed to the pubsub > topics receieving messages for *ALL* build. Such clients would have > unconditionally load build by build id. This is a waste. > > We'd like to change pubsub message: instead of sending only build id, send other > fields too. Clients can determine the order of messages by looking at the > update_time field. > > To make such a breaking change, first we'd like to know who uses this feature > > R=tandrii@chromium.org > BUG= 624960 > > Change-Id: Ic75b264613e3f09f799d4400e565db607202115f > Reviewed-on: https://chromium-review.googlesource.com/506108 > Commit-Queue: Nodir Turakulov <nodir@chromium.org> > Reviewed-by: Robbie Iannucci <iannucci@chromium.org> > Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> > TBR=iannucci@chromium.org,nodir@chromium.org,tandrii@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. BUG= 624960 Change-Id: I28afe61569a8d017e9a33935a2be45993a882c8c Reviewed-on: https://chromium-review.googlesource.com/517663 Reviewed-by: Nodir Turakulov <nodir@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> [modify] https://crrev.com/8e616e381c2bc72ff74608d35c5dc5de043c87a2/appengine/cr-buildbucket/service.py
,
May 30 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/e764c9a5dbb368072be79ae779a37de75202ffd2 commit e764c9a5dbb368072be79ae779a37de75202ffd2 Author: Nodir Turakulov <nodir@google.com> Date: Tue May 30 16:57:33 2017 buildbucket: get build in notification task https://chromium-review.googlesource.com/c/506036 changed notifications to contain entire build. However, it puts an entire build to a push task which may hit the task size limit. Put only build id and mode to task body. Get the build in the task. Bug: 624960 Change-Id: If4c5f0c9e68e8a1b8dc7d014cf1cf48c851f8bb5 Reviewed-on: https://chromium-review.googlesource.com/518264 Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> [modify] https://crrev.com/e764c9a5dbb368072be79ae779a37de75202ffd2/appengine/cr-buildbucket/events.py [modify] https://crrev.com/e764c9a5dbb368072be79ae779a37de75202ffd2/appengine/cr-buildbucket/test/events_test.py [modify] https://crrev.com/e764c9a5dbb368072be79ae779a37de75202ffd2/appengine/cr-buildbucket/test/service_test.py
,
Jun 2 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/e0466393117fa7c42af55ba4902ff3a80b2a4735 commit e0466393117fa7c42af55ba4902ff3a80b2a4735 Author: Nodir Turakulov <nodir@google.com> Date: Fri Jun 02 21:49:54 2017 buildbucket: fix build_to_dict build_to_dict incorrectly formats int64 fields as numbers instead of strings, except for id. In particularly, timestamps are returned as numbers. They must be strings to be compatible with Cloud Endpoints. R=hinoka@chromium.org Bug: 624960 Change-Id: Ie931baa22e748ca81d9ead0caeff76bdd07347d5 Reviewed-on: https://chromium-review.googlesource.com/522850 Reviewed-by: Ryan Tseng <hinoka@chromium.org> Commit-Queue: Nodir Turakulov <nodir@chromium.org> [modify] https://crrev.com/e0466393117fa7c42af55ba4902ff3a80b2a4735/appengine/cr-buildbucket/api_common.py [modify] https://crrev.com/e0466393117fa7c42af55ba4902ff3a80b2a4735/appengine/cr-buildbucket/test/api_common_test.py
,
Jun 23 2017
,
Jun 23 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/bc418f11d657247d22069aa3dd0074516fc218ff commit bc418f11d657247d22069aa3dd0074516fc218ff Author: Ryan Tseng <hinoka@google.com> Date: Fri Jun 23 23:39:44 2017
,
Jul 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/20a1165ab0ef98cce14e4eac859879a4630abc92 commit 20a1165ab0ef98cce14e4eac859879a4630abc92 Author: hinoka <hinoka@google.com> Date: Mon Jul 10 22:01:25 2017 Milo: Buildbucket PubSub ingestion outline This is a basic buildbucket ingestion outline which: 1. Sets up the buildbucket pubsub subscription, given the correct permissions 2. Receives pubsub messages, acks them, and logs them. BUG= 624960 Review-Url: https://codereview.chromium.org/2955223002 [modify] https://crrev.com/20a1165ab0ef98cce14e4eac859879a4630abc92/milo/api/config/settings.pb.go [modify] https://crrev.com/20a1165ab0ef98cce14e4eac859879a4630abc92/milo/api/config/settings.proto [modify] https://crrev.com/20a1165ab0ef98cce14e4eac859879a4630abc92/milo/buildsource/buildbot/pubsub.go [modify] https://crrev.com/20a1165ab0ef98cce14e4eac859879a4630abc92/milo/buildsource/buildbot/pubsub_test.go [modify] https://crrev.com/20a1165ab0ef98cce14e4eac859879a4630abc92/milo/buildsource/buildbucket/builder_test.go [add] https://crrev.com/20a1165ab0ef98cce14e4eac859879a4630abc92/milo/buildsource/buildbucket/pubsub.go [modify] https://crrev.com/20a1165ab0ef98cce14e4eac859879a4630abc92/milo/common/config.go [modify] https://crrev.com/20a1165ab0ef98cce14e4eac859879a4630abc92/milo/common/config_test.go [add] https://crrev.com/20a1165ab0ef98cce14e4eac859879a4630abc92/milo/common/pubsub.go [add] https://crrev.com/20a1165ab0ef98cce14e4eac859879a4630abc92/milo/common/pubsub_test.go [modify] https://crrev.com/20a1165ab0ef98cce14e4eac859879a4630abc92/milo/frontend/config.go [modify] https://crrev.com/20a1165ab0ef98cce14e4eac859879a4630abc92/milo/frontend/main.go
,
Jul 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/3f5ae6c4a1c17663c8113708a912fea6d8b98781 commit 3f5ae6c4a1c17663c8113708a912fea6d8b98781 Author: hinoka <hinoka@google.com> Date: Tue Jul 11 23:15:33 2017 Buildbucket: Save buildbucket build info and summary on pubsub push BUG= 624960 Review-Url: https://codereview.chromium.org/2964143002 [modify] https://crrev.com/3f5ae6c4a1c17663c8113708a912fea6d8b98781/milo/api/resp/build.go [modify] https://crrev.com/3f5ae6c4a1c17663c8113708a912fea6d8b98781/milo/buildsource/buildbucket/pubsub.go [add] https://crrev.com/3f5ae6c4a1c17663c8113708a912fea6d8b98781/milo/buildsource/buildbucket/struct.go [modify] https://crrev.com/3f5ae6c4a1c17663c8113708a912fea6d8b98781/milo/buildsource/swarming/build.go [modify] https://crrev.com/3f5ae6c4a1c17663c8113708a912fea6d8b98781/milo/buildsource/swarming/build_test.go [modify] https://crrev.com/3f5ae6c4a1c17663c8113708a912fea6d8b98781/milo/buildsource/swarming/buildinfo.go [modify] https://crrev.com/3f5ae6c4a1c17663c8113708a912fea6d8b98781/milo/buildsource/swarming/buildinfo_test.go [modify] https://crrev.com/3f5ae6c4a1c17663c8113708a912fea6d8b98781/milo/buildsource/swarming/html.go [modify] https://crrev.com/3f5ae6c4a1c17663c8113708a912fea6d8b98781/milo/buildsource/swarming/html_data.go
,
Jul 12 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/config/+/4109538b8b49b4a8fe9d7ef3209d8eaab182dddb commit 4109538b8b49b4a8fe9d7ef3209d8eaab182dddb Author: Ryan Tseng <hinoka@google.com> Date: Wed Jul 12 00:06:56 2017
,
Jul 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/4bfe11ebfbb203db86104241a7a748df16f507ef commit 4bfe11ebfbb203db86104241a7a748df16f507ef Author: hinoka <hinoka@google.com> Date: Fri Jul 14 17:48:11 2017 Milo: Move buildbucket pubsub sub from buildbucket project to milo project This moves (for instance) the subscription from: projects/cr-buildbucket/subscriptions/luci-milo to: projects/luci-milo/subscriptions/buildbucket This removes a series of complex steps when setting up the pubsub pipeline. Nothing else changes. BUG= 624960 Review-Url: https://codereview.chromium.org/2981683002 [modify] https://crrev.com/4bfe11ebfbb203db86104241a7a748df16f507ef/milo/common/pubsub.go [modify] https://crrev.com/4bfe11ebfbb203db86104241a7a748df16f507ef/milo/common/pubsub_test.go
,
Jul 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/33f9af6122fbc8938f1208ad0cbd35a2ffcc929b commit 33f9af6122fbc8938f1208ad0cbd35a2ffcc929b Author: hinoka <hinoka@google.com> Date: Mon Jul 17 21:48:48 2017 Milo pubsub: Don't decode properties out of parameters This isn't used and also isn't always a string (could be an object) BUG= 624960 Review-Url: https://codereview.chromium.org/2980253002 [modify] https://crrev.com/33f9af6122fbc8938f1208ad0cbd35a2ffcc929b/milo/buildsource/buildbucket/pubsub.go
,
Oct 5 2017
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by hinoka@chromium.org
, Jul 18 2016