I'll start with adding the stats retriever with it, and if it pans out, we can start redoing everything, including the bot API.
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/64a8782f95b1028ed70707e4fc93eb37de31ee9f commit 64a8782f95b1028ed70707e4fc93eb37de31ee9f Author: Marc-Antoine Ruel <maruel@chromium.org> Date: Wed Dec 12 01:05:06 2018 [swarming] Add base for pRPC server This will also be useful for BigQuery schema definition. Move tasks.proto into swarming.proto, which will contain everything related to the API. Bug: 913953 Change-Id: I88b43898a16a2852818b0d3eb75f8750dc3937fd Reviewed-on: https://chromium-review.googlesource.com/c/1372364 Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org> [modify] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/app.yaml [add] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/handlers_prpc.py [modify] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/main_frontend.py [modify] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/proto/plugin.proto [modify] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/proto/plugin_pb2.py [modify] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/proto/plugin_prpc_pb2.py [rename] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/proto/swarming.proto [rename] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/proto/swarming_pb2.py [add] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/proto/swarming_prpc_pb2.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/a902e4762f46172e98a970e593bbeb1aef8006da commit a902e4762f46172e98a970e593bbeb1aef8006da Author: Marc-Antoine Ruel <maruel@chromium.org> Date: Fri Dec 14 15:34:22 2018 [swarming] Add BotEvent message and first pRPC RPC It forces me to define the protos, which will then be leveraged for BigQuery table. This also expands the number of visible bot states. Bug: 913953 Bug: 870723 Change-Id: I1ffa1cd3bd47238105da26208d47ee4b693d4e7a Reviewed-on: https://chromium-review.googlesource.com/c/1372368 Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Reviewed-by: Nodir Turakulov <nodir@chromium.org> Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> [modify] https://crrev.com/a902e4762f46172e98a970e593bbeb1aef8006da/appengine/swarming/handlers_prpc.py [add] https://crrev.com/a902e4762f46172e98a970e593bbeb1aef8006da/appengine/swarming/handlers_prpc_test.py [modify] https://crrev.com/a902e4762f46172e98a970e593bbeb1aef8006da/appengine/swarming/proto/plugin_prpc_pb2.py [modify] https://crrev.com/a902e4762f46172e98a970e593bbeb1aef8006da/appengine/swarming/proto/swarming.proto [modify] https://crrev.com/a902e4762f46172e98a970e593bbeb1aef8006da/appengine/swarming/proto/swarming_pb2.py [modify] https://crrev.com/a902e4762f46172e98a970e593bbeb1aef8006da/appengine/swarming/proto/swarming_prpc_pb2.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/09a7b8d245b5c6129e79f77df1490bb1eaf1dfea commit 09a7b8d245b5c6129e79f77df1490bb1eaf1dfea Author: Marc-Antoine Ruel <maruel@chromium.org> Date: Fri Dec 14 15:47:22 2018 [swarming] Implement BotAPI.Events Add corresponding unit test. Can be used live with: echo '{"bot_id":"<id>","page_size":10,"end_time":"2018-12-10T00:00:00Z"}' | \ prpc call <instance> swarming.BotAPI.Events Bug: 913953 Change-Id: If059aa1bfccd65779e84ae5f0b39cae4972c3af5 Reviewed-on: https://chromium-review.googlesource.com/c/1372377 Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org> Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> [modify] https://crrev.com/09a7b8d245b5c6129e79f77df1490bb1eaf1dfea/appengine/swarming/handlers_prpc.py [modify] https://crrev.com/09a7b8d245b5c6129e79f77df1490bb1eaf1dfea/appengine/swarming/handlers_prpc_test.py [modify] https://crrev.com/09a7b8d245b5c6129e79f77df1490bb1eaf1dfea/appengine/swarming/main_frontend.py [modify] https://crrev.com/09a7b8d245b5c6129e79f77df1490bb1eaf1dfea/appengine/swarming/server/bot_management.py [modify] https://crrev.com/09a7b8d245b5c6129e79f77df1490bb1eaf1dfea/appengine/swarming/server/bot_management_test.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/18fa1568a69db2c4363cf8517d175c89c7cc1016 commit 18fa1568a69db2c4363cf8517d175c89c7cc1016 Author: Marc-Antoine Ruel <maruel@chromium.org> Date: Mon Dec 17 18:32:22 2018 [swarming] Split config and api protos; rename api proto package to swarming.v1 - Moving the protos into different subdirectory makes it clearer what is what. - Add READMEs to help with this. - Rename the api package to swarming.v1. This futureproofs the API for an eventual v2, if every needed. This is a breaking change for the pRPC client, but there's no users yet, so now is a good time to do this change. Bug: 913953 Change-Id: I72140a65a8ef891c037ba3091271415401151eb3 Reviewed-on: https://chromium-review.googlesource.com/c/1378572 Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org> [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/handlers_prpc.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/handlers_prpc_test.py [add] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/README.md [rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/Makefile [add] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/README.md [add] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/__init__.py [rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/plugin.proto [rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/plugin_pb2.py [add] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/plugin_prpc_pb2.py [rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/swarming.proto [rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/swarming_pb2.py [add] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/swarming_prpc_pb2.py [copy] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/Makefile [add] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/README.md [add] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/__init__.py [rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/bots.proto [rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/bots_pb2.py [rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/config.proto [rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/config_pb2.py [rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/pools.proto [rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/pools_pb2.py [delete] https://crrev.com/dbea18c46588bb4c0345c05dd33a889e74671a6d/appengine/swarming/proto/plugin_prpc_pb2.py [delete] https://crrev.com/dbea18c46588bb4c0345c05dd33a889e74671a6d/appengine/swarming/proto/swarming_prpc_pb2.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/acl_test.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/bot_archive_test.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/bot_auth_test.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/bot_groups_config.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/bot_groups_config_test.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/bot_management.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/bot_management_test.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/config.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/config_test.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/external_scheduler.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/lease_management_test.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/pools_config.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/pools_config_test.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/task_scheduler_test.py [modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/test_env_handlers.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-py.git/+/3c2b868130fd61c04d7902c493a96297a9db50f9 commit 3c2b868130fd61c04d7902c493a96297a9db50f9 Author: Marc-Antoine Ruel <maruel@chromium.org> Date: Mon Dec 17 19:28:42 2018 [swarming] Rewrite TaskState enum - Make TaskState extremely detailed. - Use clear categories and even create a type for these: TaskStateCategory. - Use a much better bit mask now that enables great queries yet has an efficient use of the bitspace and efficient varint encoding. This leaves 16 types of state per task state category. - The bitmask is not useful for BigQuery, since we store the enums as strings. That's why TaskStateCategory is useful, we can store it too in BigQuery so this makes the BQ queries much simpler when someone doesn't care about the subtle sub-state. Change-Id: I2e06618509910dcb097fdf63ca63e087d44e6635 Bug: 913953 Reviewed-on: https://chromium-review.googlesource.com/c/1378573 Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org> Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> [modify] https://crrev.com/3c2b868130fd61c04d7902c493a96297a9db50f9/appengine/swarming/proto/api/plugin_prpc_pb2.py [modify] https://crrev.com/3c2b868130fd61c04d7902c493a96297a9db50f9/appengine/swarming/proto/api/swarming.proto [modify] https://crrev.com/3c2b868130fd61c04d7902c493a96297a9db50f9/appengine/swarming/proto/api/swarming_pb2.py [modify] https://crrev.com/3c2b868130fd61c04d7902c493a96297a9db50f9/appengine/swarming/proto/api/swarming_prpc_pb2.py [modify] https://crrev.com/3c2b868130fd61c04d7902c493a96297a9db50f9/appengine/swarming/server/task_result.py [modify] https://crrev.com/3c2b868130fd61c04d7902c493a96297a9db50f9/appengine/swarming/server/task_result_test.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/fe836559b510b72b487dfcd5eb06beafe97b1dc9 commit fe836559b510b72b487dfcd5eb06beafe97b1dc9 Author: Aviv Keshet <akeshet@chromium.org> Date: Wed Dec 19 19:14:42 2018 qscheduler: use new swarming task state categories in state conversion BUG=chromium:913953 Change-Id: Id9b1173d9b9e669a7652f9c0f0edc62f08ac2626 Reviewed-on: https://chromium-review.googlesource.com/c/1384804 Reviewed-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> Commit-Queue: Aviv Keshet <akeshet@chromium.org> Auto-Submit: Aviv Keshet <akeshet@chromium.org> Cr-Commit-Position: refs/heads/master@{#19673} [modify] https://crrev.com/fe836559b510b72b487dfcd5eb06beafe97b1dc9/go/src/infra/appengine/qscheduler-swarming/app/frontend/qscheduler.go
Let's block this on migrating out of python to the Go client, since it's going to be slightly more annoying to support it in python.
Comment 1 by bugdroid1@chromium.org
, Dec 12