Be able to return LUCI builds as buildbot builds in Milo's buildstore
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/40d54f12b2cf43fb6b9c785a59e48b6275fe85a9 commit 40d54f12b2cf43fb6b9c785a59e48b6275fe85a9 Author: Nodir Turakulov <nodir@google.com> Date: Mon Oct 09 22:26:38 2017 [buildbucket] add set_next_build_number API to migrate a build from buildbot, we may need to bump build number generation so that new LUCI builds have numbers higher than Buildbot builds. Add set_next_build_number to Swarmbucket API. Bug: 772696 Change-Id: I25321ade58b5fc9bcdb3375e9161eaafe3fc1d1b Reviewed-on: https://chromium-review.googlesource.com/707360 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> [modify] https://crrev.com/40d54f12b2cf43fb6b9c785a59e48b6275fe85a9/appengine/cr-buildbucket/swarming/swarming.py [modify] https://crrev.com/40d54f12b2cf43fb6b9c785a59e48b6275fe85a9/appengine/cr-buildbucket/acl.py [modify] https://crrev.com/40d54f12b2cf43fb6b9c785a59e48b6275fe85a9/appengine/cr-buildbucket/sequence.py [modify] https://crrev.com/40d54f12b2cf43fb6b9c785a59e48b6275fe85a9/appengine/cr-buildbucket/swarming/swarmbucket_api.py [modify] https://crrev.com/40d54f12b2cf43fb6b9c785a59e48b6275fe85a9/appengine/cr-buildbucket/swarming/test/swarmbucket_api_test.py [modify] https://crrev.com/40d54f12b2cf43fb6b9c785a59e48b6275fe85a9/appengine/cr-buildbucket/test/sequence_test.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/47274eb9450e97cb316abc6a2fefe0e759414615 commit 47274eb9450e97cb316abc6a2fefe0e759414615 Author: Nodir Turakulov <nodir@google.com> Date: Mon Oct 09 23:38:19 2017 [buildbucket] add SearchQuery move search parameters to new SearchQuery class. Use it in both search implementations. Bug: 772696 Change-Id: I671751d82e31f8c40055babecf97b66b78578491 Reviewed-on: https://chromium-review.googlesource.com/706133 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Michael Knyszek <mknyszek@google.com> [modify] https://crrev.com/47274eb9450e97cb316abc6a2fefe0e759414615/appengine/cr-buildbucket/service.py [modify] https://crrev.com/47274eb9450e97cb316abc6a2fefe0e759414615/appengine/cr-buildbucket/test/service_test.py [modify] https://crrev.com/47274eb9450e97cb316abc6a2fefe0e759414615/appengine/cr-buildbucket/test/api_test.py [modify] https://crrev.com/47274eb9450e97cb316abc6a2fefe0e759414615/appengine/cr-buildbucket/api.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/b3ea727699d6b6d181203ed10e3212ec348f233b commit b3ea727699d6b6d181203ed10e3212ec348f233b Author: Nodir Turakulov <nodir@google.com> Date: Tue Oct 10 19:06:03 2017 [milo] add Query.NumbersOnly Add Query.NumbersOnly boolean field. If true, performs a keys-only datastore query. Use Query.NumbersOnly in GetMaster instead of direct datastore query. R=iannucci@chromium.org, hinoka@chromium.org Bug: 772696 Change-Id: I642f2f491e058bacd8e5b163098d0fc02b9012c7 Reviewed-on: https://chromium-review.googlesource.com/706704 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Reviewed-by: Ryan Tseng <hinoka@chromium.org> [modify] https://crrev.com/b3ea727699d6b6d181203ed10e3212ec348f233b/milo/buildsource/buildbot/buildstore/master.go [modify] https://crrev.com/b3ea727699d6b6d181203ed10e3212ec348f233b/milo/buildsource/buildbot/buildstore/query.go
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/1e2b3df3d3e64172c2368debce03d75717ec0da2 commit 1e2b3df3d3e64172c2368debce03d75717ec0da2 Author: Nodir Turakulov <nodir@google.com> Date: Tue Oct 10 21:57:47 2017 [milo] remove datastore cursors from buildbot The new numeric cursors were added last week. Presumably, nobody needs datastore cursors, so delete them. Bug: 772696 Change-Id: I49fd4c75c145bf7edfdf18419a74efa653bd19a0 Reviewed-on: https://chromium-review.googlesource.com/710515 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> [modify] https://crrev.com/1e2b3df3d3e64172c2368debce03d75717ec0da2/milo/buildsource/buildbot/buildstore/query.go
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/913680c69e2318af2eac8f3aa6608850280c0ca9 commit 913680c69e2318af2eac8f3aa6608850280c0ca9 Author: Nodir Turakulov <nodir@google.com> Date: Tue Oct 10 22:02:13 2017 [buildbucket] implement incomplete build search If a user needs to find all scheduled and running builds, they have to issue two queries and then merge-sort results. Make it easier for users to do this by implementing a search by incomplete status. Add computed "incomplete" boolean property. We search only by incomplete=true. A build lives in incomplete state up to 48h => 48h after this change is deployed, the implication statement "build is incomplete => its incomplete property is True" will apply to all builds in the datastore, Bug: 772696 Change-Id: I3de8c6078b2a832c5cd8bb13d03484fef1ffe863 Reviewed-on: https://chromium-review.googlesource.com/706188 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Michael Knyszek <mknyszek@google.com> [modify] https://crrev.com/913680c69e2318af2eac8f3aa6608850280c0ca9/appengine/cr-buildbucket/service.py [modify] https://crrev.com/913680c69e2318af2eac8f3aa6608850280c0ca9/appengine/cr-buildbucket/test/service_test.py [modify] https://crrev.com/913680c69e2318af2eac8f3aa6608850280c0ca9/appengine/cr-buildbucket/model.py [modify] https://crrev.com/913680c69e2318af2eac8f3aa6608850280c0ca9/appengine/cr-buildbucket/test/api_test.py [modify] https://crrev.com/913680c69e2318af2eac8f3aa6608850280c0ca9/appengine/cr-buildbucket/api.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/3eca679595dda0691832f9087861a390ba0227e8 commit 3eca679595dda0691832f9087861a390ba0227e8 Author: Nodir Turakulov <nodir@google.com> Date: Tue Oct 17 18:14:25 2017 [gitiles] allow anonymous requests gitiles.Client currently assumes that its HTTP client sends authenticated requests. If the latter sends anonymous requests, they fail because of the forced /a/ URL path prefix. Remove the assumption and make authentication opt-in. R=tandrii@chromium.org Bug: 772696 Change-Id: I70eabb8636303cefeda7325448105684429c67c6 Reviewed-on: https://chromium-review.googlesource.com/723785 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> [modify] https://crrev.com/3eca679595dda0691832f9087861a390ba0227e8/common/api/gitiles/gitiles.go [modify] https://crrev.com/3eca679595dda0691832f9087861a390ba0227e8/common/api/gitiles/gitiles_test.go [modify] https://crrev.com/3eca679595dda0691832f9087861a390ba0227e8/milo/git/history.go [modify] https://crrev.com/3eca679595dda0691832f9087861a390ba0227e8/milo/srcman/blamelist.go [modify] https://crrev.com/3eca679595dda0691832f9087861a390ba0227e8/scheduler/appengine/task/buildbucket/buildbucket.go [modify] https://crrev.com/3eca679595dda0691832f9087861a390ba0227e8/scheduler/appengine/task/gitiles/gitiles.go
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/358e86a960216e581b2af71267f30cc92e21d6c2 commit 358e86a960216e581b2af71267f30cc92e21d6c2 Author: Nodir Turakulov <nodir@google.com> Date: Thu Oct 19 15:40:39 2017 [milo] emulate LUCI in buildstore Emulate Buildbot in API and GUI at the storage level. Add EmulationOptions proto message to gRPC endpoints and add equivalent query string parameters to the page handlers. The emulation options per builder are inserted into context and used in buildstore to decide what to return. Users will not pass emulation options in requests, but Milo developers can pass them to preview Milo behavior when a Buildbot builder is migrated to LUCI. Next CL will add an API to set default emulation options per builder, stored in the datastore. Bug: 772696 Change-Id: I38249b228797585bcc19db17fba95ca1736e0b25 Reviewed-on: https://chromium-review.googlesource.com/717186 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Ryan Tseng <hinoka@chromium.org> [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/common/api/buildbucket/buildbucket/v1/const.go [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/api/buildbot/result.go [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/api/buildbot/structs.go [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/api/proto/buildbot.pb.go [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/api/proto/buildbot.proto [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/api/proto/pb.discovery.go [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/buildsource/buildbot/build.go [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/buildsource/buildbot/builder.go [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/buildsource/buildbot/buildinfo.go [add] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/buildsource/buildbot/buildstore/annotations.go [add] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/buildsource/buildbot/buildstore/annotations_test.go [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/buildsource/buildbot/buildstore/build.go [add] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/buildsource/buildbot/buildstore/buildbucket.go [add] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/buildsource/buildbot/buildstore/emulation.go [add] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/buildsource/buildbot/buildstore/expectations/steps.json [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/buildsource/buildbot/buildstore/master.go [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/buildsource/buildbot/buildstore/query.go [add] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/buildsource/buildbot/buildstore/testdata/annotation.pb.txt [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/buildsource/buildbot/grpc.go [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/frontend/middleware.go [modify] https://crrev.com/358e86a960216e581b2af71267f30cc92e21d6c2/milo/frontend/routes.go
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/f97244436260ef461e10d5cd66108430493382f5 commit f97244436260ef461e10d5cd66108430493382f5 Author: Nodir Turakulov <nodir@google.com> Date: Thu Oct 19 16:10:49 2017 [milo] compute blamelist in buildstore emulation Compute static blamelist for LUCI-emulated builds. Load changes from Gitiles in serving code, but memcaches them. Also memcache revision per build to avoid loading previous builds. Bug: 772696 Change-Id: I5c404e1315e916ff9628d3c60ca46ecf95ce12e3 Reviewed-on: https://chromium-review.googlesource.com/723860 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Ryan Tseng <hinoka@chromium.org> [modify] https://crrev.com/f97244436260ef461e10d5cd66108430493382f5/common/api/gitiles/gitiles.go [add] https://crrev.com/f97244436260ef461e10d5cd66108430493382f5/milo/api/buildbot/result_string.go [modify] https://crrev.com/f97244436260ef461e10d5cd66108430493382f5/milo/api/buildbot/structs.go [add] https://crrev.com/f97244436260ef461e10d5cd66108430493382f5/milo/buildsource/buildbot/buildstore/blame.go [modify] https://crrev.com/f97244436260ef461e10d5cd66108430493382f5/milo/buildsource/buildbot/buildstore/build.go [modify] https://crrev.com/f97244436260ef461e10d5cd66108430493382f5/milo/buildsource/buildbot/buildstore/buildbucket.go [modify] https://crrev.com/f97244436260ef461e10d5cd66108430493382f5/milo/buildsource/buildbot/buildstore/query.go
https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/721698 was the final piece
Comment 1 by no...@chromium.org
, Oct 8 2017