Add configuration and appropriate calls within swarming to implement the scheduler plugin API as described here: https://docs.google.com/document/d/1cm1IsTGGistGqkRXV82X5jPCv3SmRdhUtMo4d9N-1bg/edit?disco=AAAACOZTUwE&ts=5bc49b48 Most likely, begin rolling this out in the chromeos swarming instance first, against a dev instance of the qscheduler-swarming app being developed in Issue 898322
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/87ad04df2431eb063557c1229b60b94623ec7bd2 commit 87ad04df2431eb063557c1229b60b94623ec7bd2 Author: Nodir Turakulov <nodir@google.com> Date: Tue Nov 06 23:18:41 2018 [compile_proto] Pass --prpc-python_out Pass --prpc-python_out in compile_proto.py It is harmless if the proto does not have services. It is useful if it does. Also prepend //appengine/components/tools to $PATH before calling protoc so that protoc finds //appengine/components/tools/protoc-gen-prpc-python R=akeshet@chromium.org, vadimsh@chromium.org Bug: 898323 Change-Id: Ic788c7b1920723d4910e3e89a4be07b3c2fa2ae4 Reviewed-on: https://chromium-review.googlesource.com/c/1320629 Commit-Queue: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> [modify] https://crrev.com/87ad04df2431eb063557c1229b60b94623ec7bd2/appengine/components/tools/compile_proto.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/124e68f036efd32bec08833dec5bdf9c12c6ecf5 commit 124e68f036efd32bec08833dec5bdf9c12c6ecf5 Author: Aviv Keshet <akeshet@chromium.org> Date: Wed Nov 07 00:09:08 2018 [swarming] create a representation of external scheduler in PoolConfig BUG= chromium:898323 TEST=unit test updated Change-Id: I0718c313012a956473c8f62a2963854e01f2c328 Reviewed-on: https://chromium-review.googlesource.com/c/1316288 Commit-Queue: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> [modify] https://crrev.com/124e68f036efd32bec08833dec5bdf9c12c6ecf5/appengine/swarming/server/pools_config.py [modify] https://crrev.com/124e68f036efd32bec08833dec5bdf9c12c6ecf5/appengine/swarming/server/pools_config_test.py [modify] https://crrev.com/124e68f036efd32bec08833dec5bdf9c12c6ecf5/appengine/swarming/server/task_request_test.py [modify] https://crrev.com/124e68f036efd32bec08833dec5bdf9c12c6ecf5/appengine/swarming/server/task_scheduler_test.py [modify] https://crrev.com/124e68f036efd32bec08833dec5bdf9c12c6ecf5/appengine/swarming/test_env_handlers.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/82b479af6353f845c06abd18d5f414ec85854546 commit 82b479af6353f845c06abd18d5f414ec85854546 Author: Aviv Keshet <akeshet@chromium.org> Date: Wed Nov 07 23:54:04 2018 [swarming] regenerate protos to include prpc bindings We want to have python client bindings for the external scheduler plugin service. BUG= chromium:898323 TEST=None Change-Id: I6f0b434cf26d073798aff7098c1379de74d6a937 Reviewed-on: https://chromium-review.googlesource.com/c/1316887 Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> Reviewed-by: Nodir Turakulov <nodir@chromium.org> Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org> [add] https://crrev.com/82b479af6353f845c06abd18d5f414ec85854546/appengine/swarming/proto/plugin_prpc_pb2.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/1f03654b27cc3178aec4d67213dce23448b2fe3e commit 1f03654b27cc3178aec4d67213dce23448b2fe3e Author: Aviv Keshet <akeshet@chromium.org> Date: Sat Nov 10 00:28:47 2018 [swarming] add an enabled field to ExternalSchedulerConfig BUG= chromium:898323 TEST=None Change-Id: I80597822cb8cc99301293ad8ebe1d3fbc4c58773 Reviewed-on: https://chromium-review.googlesource.com/c/1330305 Reviewed-by: Vadim Shtayura <vadimsh@chromium.org> Commit-Queue: Vadim Shtayura <vadimsh@chromium.org> Auto-Submit: Aviv Keshet <akeshet@chromium.org> [modify] https://crrev.com/1f03654b27cc3178aec4d67213dce23448b2fe3e/appengine/swarming/proto/pools.proto [modify] https://crrev.com/1f03654b27cc3178aec4d67213dce23448b2fe3e/appengine/swarming/proto/pools_pb2.py [modify] https://crrev.com/1f03654b27cc3178aec4d67213dce23448b2fe3e/appengine/swarming/server/pools_config.py [modify] https://crrev.com/1f03654b27cc3178aec4d67213dce23448b2fe3e/appengine/swarming/server/pools_config_test.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/92bfa8d709c269cdcafb329152748166d0631640 commit 92bfa8d709c269cdcafb329152748166d0631640 Author: Aviv Keshet <akeshet@chromium.org> Date: Thu Nov 15 18:17:59 2018 [swarming] rename _maybe_pubsub_notify_via_tq No behavioral change. This method will be conditionally enqueueing NotifyRequest calls to external scheduler, in a follow-up CL. Rename the method here to reduce review diff in future. BUG= chromium:898323 TEST=None Change-Id: Ice477594d36bafce45b0d3386899df9500682bf6 Reviewed-on: https://chromium-review.googlesource.com/c/1336452 Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org> [modify] https://crrev.com/92bfa8d709c269cdcafb329152748166d0631640/appengine/swarming/server/task_scheduler.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/7a9e4b59e68e5db8f05425c2cfea031ba83a533b commit 7a9e4b59e68e5db8f05425c2cfea031ba83a533b Author: Aviv Keshet <akeshet@chromium.org> Date: Thu Nov 15 18:24:00 2018 Revert "[swarming] rename _maybe_pubsub_notify_via_tq" This reverts commit 92bfa8d709c269cdcafb329152748166d0631640. Reason for revert: insufficiently confident in CL Original change's description: > [swarming] rename _maybe_pubsub_notify_via_tq > > No behavioral change. > > This method will be conditionally enqueueing NotifyRequest calls to > external scheduler, in a follow-up CL. Rename the method here to reduce > review diff in future. > > BUG= chromium:898323 > TEST=None > > Change-Id: Ice477594d36bafce45b0d3386899df9500682bf6 > Reviewed-on: https://chromium-review.googlesource.com/c/1336452 > Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> > Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org> TBR=maruel@chromium.org,akeshet@chromium.org Change-Id: I91b0df5f08a6493cfdad19e747fa7d47f0dffadd No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: chromium:898323 Reviewed-on: https://chromium-review.googlesource.com/c/1337518 Reviewed-by: Aviv Keshet <akeshet@chromium.org> Commit-Queue: Aviv Keshet <akeshet@chromium.org> [modify] https://crrev.com/7a9e4b59e68e5db8f05425c2cfea031ba83a533b/appengine/swarming/server/task_scheduler.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/c6084353ec05f09b28bd00b86c255caa83d62b14 commit c6084353ec05f09b28bd00b86c255caa83d62b14 Author: Aviv Keshet <akeshet@chromium.org> Date: Mon Nov 19 19:17:13 2018 [swarming] rename _maybe_pubsub_notify_via_tq No behavioral change. This method will be making NotifyRequest calls to external scheduler, in a follow-up CL. Rename the method here to reduce review diff in future. BUG= chromium:898323 TEST=None Change-Id: I1a2a8ba347507d4ed6df493732fc40afe3cd0a3d Reviewed-on: https://chromium-review.googlesource.com/c/1338361 Commit-Queue: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> [modify] https://crrev.com/c6084353ec05f09b28bd00b86c255caa83d62b14/appengine/swarming/server/task_scheduler.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/f10f5aacf87f8a9ba2d96e1975b0e5b0e19c51e2 commit f10f5aacf87f8a9ba2d96e1975b0e5b0e19c51e2 Author: Aviv Keshet <akeshet@chromium.org> Date: Wed Nov 21 23:34:05 2018 [swarming] add external_scheduler assign and notify calls This CL implements the basic calls necessary for swarming tasks to be sent to an external scheduler, and for bots to use an external scheduler. Not yet implemented: Usage of the GetCancellations external_scheduler endpoint. BUG= chromium:898323 TEST=local testing with swarming instance and local qscheduler Change-Id: I7ad016eb2bc2547da87fa86f68fa35e07e142f33 Reviewed-on: https://chromium-review.googlesource.com/c/1303286 Commit-Queue: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> [modify] https://crrev.com/f10f5aacf87f8a9ba2d96e1975b0e5b0e19c51e2/appengine/swarming/proto/pools.proto [add] https://crrev.com/f10f5aacf87f8a9ba2d96e1975b0e5b0e19c51e2/appengine/swarming/server/external_scheduler.py [modify] https://crrev.com/f10f5aacf87f8a9ba2d96e1975b0e5b0e19c51e2/appengine/swarming/server/task_scheduler.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/21a18922a5c0f7b2e8f9496f41dc099f67e327cd commit 21a18922a5c0f7b2e8f9496f41dc099f67e327cd Author: Aviv Keshet <akeshet@chromium.org> Date: Fri Nov 30 00:12:45 2018 swarming: add bot_id argument to cancel_task Add a a bot_id argument to cancel_task, which causes the task to be cancelled only if it is running on the specified bot. BUG= chromium:898323 TEST=new unit test added Change-Id: Ib20a5c2dcd9012f8652ae4757576f15f07e2b8b1 Reviewed-on: https://chromium-review.googlesource.com/c/1354523 Commit-Queue: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> [modify] https://crrev.com/21a18922a5c0f7b2e8f9496f41dc099f67e327cd/appengine/swarming/handlers_backend.py [modify] https://crrev.com/21a18922a5c0f7b2e8f9496f41dc099f67e327cd/appengine/swarming/handlers_endpoints.py [modify] https://crrev.com/21a18922a5c0f7b2e8f9496f41dc099f67e327cd/appengine/swarming/server/task_scheduler.py [modify] https://crrev.com/21a18922a5c0f7b2e8f9496f41dc099f67e327cd/appengine/swarming/server/task_scheduler_test.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/3ae195d6a5ab4c12d43bc6cd2fb126d359dd6d4a commit 3ae195d6a5ab4c12d43bc6cd2fb126d359dd6d4a Author: Aviv Keshet <akeshet@chromium.org> Date: Fri Nov 30 20:31:06 2018 swarming: extract find-then-cancel logic into cancel_task_with_id No behavioral change. In a forthcoming CL, a new task queue handler will be added that needs to reuse the same find-task-then-cancel logic, so move it to a place where it can be shared. BUG= chromium:898323 TEST=unit test added Change-Id: I2956661e4fb5eb1c27446340414ea0eb99a62420 Reviewed-on: https://chromium-review.googlesource.com/c/1355721 Commit-Queue: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> [modify] https://crrev.com/3ae195d6a5ab4c12d43bc6cd2fb126d359dd6d4a/appengine/swarming/handlers_backend.py [modify] https://crrev.com/3ae195d6a5ab4c12d43bc6cd2fb126d359dd6d4a/appengine/swarming/server/task_scheduler.py [modify] https://crrev.com/3ae195d6a5ab4c12d43bc6cd2fb126d359dd6d4a/appengine/swarming/server/task_scheduler_test.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/19a7b88cb4d8b73fd8e80a67b5098473f79fb895 commit 19a7b88cb4d8b73fd8e80a67b5098473f79fb895 Author: Aviv Keshet <akeshet@chromium.org> Date: Mon Dec 03 21:53:28 2018 swarming: add a partial stub for external scheduler cancellations This CL creates a partial stub to implement task cancellation from an external scheduler. A cron handler calls the external scheduler to fetch cancellations; actual cancellation of the task will be implemented in a follow-up. The handler needs to exist before cron.yaml is modified, for rollout reasons. BUG= chromium:898323 Change-Id: I6baf985c0c55424593a4864d05085efef3b7bb45 Reviewed-on: https://chromium-review.googlesource.com/c/1354092 Commit-Queue: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> [modify] https://crrev.com/19a7b88cb4d8b73fd8e80a67b5098473f79fb895/appengine/swarming/handlers_backend.py [modify] https://crrev.com/19a7b88cb4d8b73fd8e80a67b5098473f79fb895/appengine/swarming/server/external_scheduler.py [modify] https://crrev.com/19a7b88cb4d8b73fd8e80a67b5098473f79fb895/appengine/swarming/server/task_scheduler.py [modify] https://crrev.com/19a7b88cb4d8b73fd8e80a67b5098473f79fb895/appengine/swarming/server/task_scheduler_test.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/5e26d3a6797138f1ab92a4b2fa037b1e9d021eea commit 5e26d3a6797138f1ab92a4b2fa037b1e9d021eea Author: Aviv Keshet <akeshet@chromium.org> Date: Mon Dec 03 23:06:44 2018 swarming: add a taskqueue handler for cancelling task on bot This taskqueue will be used in a forthcoming CL. BUG= chromium:898323 TEST=None Change-Id: Idd208a97b11eacdc5bcb78fc5fecc9f5773c6c37 Reviewed-on: https://chromium-review.googlesource.com/c/1355443 Commit-Queue: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> [modify] https://crrev.com/5e26d3a6797138f1ab92a4b2fa037b1e9d021eea/appengine/components/test_support/test_case.py [modify] https://crrev.com/5e26d3a6797138f1ab92a4b2fa037b1e9d021eea/appengine/swarming/handlers_backend.py [modify] https://crrev.com/5e26d3a6797138f1ab92a4b2fa037b1e9d021eea/appengine/swarming/handlers_test.py [modify] https://crrev.com/5e26d3a6797138f1ab92a4b2fa037b1e9d021eea/appengine/swarming/queue.yaml [modify] https://crrev.com/5e26d3a6797138f1ab92a4b2fa037b1e9d021eea/appengine/swarming/server/task_scheduler.py [modify] https://crrev.com/5e26d3a6797138f1ab92a4b2fa037b1e9d021eea/appengine/swarming/server/task_scheduler_test.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/0c0267ce37a1fc9e76700d30c84779b2ada713e2 commit 0c0267ce37a1fc9e76700d30c84779b2ada713e2 Author: Aviv Keshet <akeshet@chromium.org> Date: Mon Dec 10 19:32:48 2018 swarming: add cron entry for external scheduler cancellations BUG= chromium:898323 Change-Id: Ib51a351c45f926586a1ee96ef4c631c1b95af0fe Reviewed-on: https://chromium-review.googlesource.com/c/1354520 Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> Commit-Queue: Aviv Keshet <akeshet@chromium.org> [modify] https://crrev.com/0c0267ce37a1fc9e76700d30c84779b2ada713e2/appengine/swarming/cron.yaml
Comment 1 by akes...@chromium.org
, Oct 23