New issue
Advanced search Search tips

Issue 913953 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Feature

Blocked on:
issue 894045

Blocking:
issue 916548



Sign in to add a comment

swarming: migrate to pRPC

Project Member Reported by mar...@chromium.org, Dec 11

Issue description

I'll start with adding the stats retriever with it, and if it pans out, we can start redoing everything, including the bot API.
 
Project Member

Comment 1 by bugdroid1@chromium.org, Dec 12

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/64a8782f95b1028ed70707e4fc93eb37de31ee9f

commit 64a8782f95b1028ed70707e4fc93eb37de31ee9f
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Wed Dec 12 01:05:06 2018

[swarming] Add base for pRPC server

This will also be useful for BigQuery schema definition.

Move tasks.proto into swarming.proto, which will contain everything
related to the API.

Bug: 913953
Change-Id: I88b43898a16a2852818b0d3eb75f8750dc3937fd
Reviewed-on: https://chromium-review.googlesource.com/c/1372364
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/app.yaml
[add] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/handlers_prpc.py
[modify] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/main_frontend.py
[modify] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/proto/plugin.proto
[modify] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/proto/plugin_pb2.py
[modify] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/proto/plugin_prpc_pb2.py
[rename] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/proto/swarming.proto
[rename] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/proto/swarming_pb2.py
[add] https://crrev.com/64a8782f95b1028ed70707e4fc93eb37de31ee9f/appengine/swarming/proto/swarming_prpc_pb2.py

Project Member

Comment 2 by bugdroid1@chromium.org, Dec 14

Project Member

Comment 3 by bugdroid1@chromium.org, Dec 14

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/09a7b8d245b5c6129e79f77df1490bb1eaf1dfea

commit 09a7b8d245b5c6129e79f77df1490bb1eaf1dfea
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Fri Dec 14 15:47:22 2018

[swarming] Implement BotAPI.Events

Add corresponding unit test.

Can be used live with:

  echo '{"bot_id":"<id>","page_size":10,"end_time":"2018-12-10T00:00:00Z"}' | \
      prpc call <instance> swarming.BotAPI.Events

Bug: 913953
Change-Id: If059aa1bfccd65779e84ae5f0b39cae4972c3af5
Reviewed-on: https://chromium-review.googlesource.com/c/1372377
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>

[modify] https://crrev.com/09a7b8d245b5c6129e79f77df1490bb1eaf1dfea/appengine/swarming/handlers_prpc.py
[modify] https://crrev.com/09a7b8d245b5c6129e79f77df1490bb1eaf1dfea/appengine/swarming/handlers_prpc_test.py
[modify] https://crrev.com/09a7b8d245b5c6129e79f77df1490bb1eaf1dfea/appengine/swarming/main_frontend.py
[modify] https://crrev.com/09a7b8d245b5c6129e79f77df1490bb1eaf1dfea/appengine/swarming/server/bot_management.py
[modify] https://crrev.com/09a7b8d245b5c6129e79f77df1490bb1eaf1dfea/appengine/swarming/server/bot_management_test.py

Project Member

Comment 4 by bugdroid1@chromium.org, Dec 17

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/18fa1568a69db2c4363cf8517d175c89c7cc1016

commit 18fa1568a69db2c4363cf8517d175c89c7cc1016
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Mon Dec 17 18:32:22 2018

[swarming] Split config and api protos; rename api proto package to swarming.v1

- Moving the protos into different subdirectory makes it clearer what is what.
- Add READMEs to help with this.
- Rename the api package to swarming.v1. This futureproofs the API for an
  eventual v2, if every needed. This is a breaking change for the pRPC client,
  but there's no users yet, so now is a good time to do this change.

Bug: 913953
Change-Id: I72140a65a8ef891c037ba3091271415401151eb3
Reviewed-on: https://chromium-review.googlesource.com/c/1378572
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/handlers_prpc.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/handlers_prpc_test.py
[add] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/README.md
[rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/Makefile
[add] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/README.md
[add] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/__init__.py
[rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/plugin.proto
[rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/plugin_pb2.py
[add] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/plugin_prpc_pb2.py
[rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/swarming.proto
[rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/swarming_pb2.py
[add] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/api/swarming_prpc_pb2.py
[copy] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/Makefile
[add] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/README.md
[add] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/__init__.py
[rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/bots.proto
[rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/bots_pb2.py
[rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/config.proto
[rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/config_pb2.py
[rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/pools.proto
[rename] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/proto/config/pools_pb2.py
[delete] https://crrev.com/dbea18c46588bb4c0345c05dd33a889e74671a6d/appengine/swarming/proto/plugin_prpc_pb2.py
[delete] https://crrev.com/dbea18c46588bb4c0345c05dd33a889e74671a6d/appengine/swarming/proto/swarming_prpc_pb2.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/acl_test.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/bot_archive_test.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/bot_auth_test.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/bot_groups_config.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/bot_groups_config_test.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/bot_management.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/bot_management_test.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/config.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/config_test.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/external_scheduler.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/lease_management_test.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/pools_config.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/pools_config_test.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/server/task_scheduler_test.py
[modify] https://crrev.com/18fa1568a69db2c4363cf8517d175c89c7cc1016/appengine/swarming/test_env_handlers.py

Project Member

Comment 5 by bugdroid1@chromium.org, Dec 17

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/3c2b868130fd61c04d7902c493a96297a9db50f9

commit 3c2b868130fd61c04d7902c493a96297a9db50f9
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Mon Dec 17 19:28:42 2018

[swarming] Rewrite TaskState enum

- Make TaskState extremely detailed.
- Use clear categories and even create a type for these: TaskStateCategory.
- Use a much better bit mask now that enables great queries yet has an efficient
  use of the bitspace and efficient varint encoding. This leaves 16 types of
  state per task state category.
- The bitmask is not useful for BigQuery, since we store the enums as strings.
  That's why TaskStateCategory is useful, we can store it too in BigQuery so
  this makes the BQ queries much simpler when someone doesn't care about the
  subtle sub-state.

Change-Id: I2e06618509910dcb097fdf63ca63e087d44e6635
Bug: 913953
Reviewed-on: https://chromium-review.googlesource.com/c/1378573
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>

[modify] https://crrev.com/3c2b868130fd61c04d7902c493a96297a9db50f9/appengine/swarming/proto/api/plugin_prpc_pb2.py
[modify] https://crrev.com/3c2b868130fd61c04d7902c493a96297a9db50f9/appengine/swarming/proto/api/swarming.proto
[modify] https://crrev.com/3c2b868130fd61c04d7902c493a96297a9db50f9/appengine/swarming/proto/api/swarming_pb2.py
[modify] https://crrev.com/3c2b868130fd61c04d7902c493a96297a9db50f9/appengine/swarming/proto/api/swarming_prpc_pb2.py
[modify] https://crrev.com/3c2b868130fd61c04d7902c493a96297a9db50f9/appengine/swarming/server/task_result.py
[modify] https://crrev.com/3c2b868130fd61c04d7902c493a96297a9db50f9/appengine/swarming/server/task_result_test.py

Blocking: 916548
Project Member

Comment 7 by bugdroid1@chromium.org, Dec 19

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/fe836559b510b72b487dfcd5eb06beafe97b1dc9

commit fe836559b510b72b487dfcd5eb06beafe97b1dc9
Author: Aviv Keshet <akeshet@chromium.org>
Date: Wed Dec 19 19:14:42 2018

qscheduler: use new swarming task state categories in state conversion

BUG=chromium:913953

Change-Id: Id9b1173d9b9e669a7652f9c0f0edc62f08ac2626
Reviewed-on: https://chromium-review.googlesource.com/c/1384804
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
Auto-Submit: Aviv Keshet <akeshet@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19673}
[modify] https://crrev.com/fe836559b510b72b487dfcd5eb06beafe97b1dc9/go/src/infra/appengine/qscheduler-swarming/app/frontend/qscheduler.go

Comment 8 by mar...@chromium.org, Today (11 hours ago)

Blockedon: 894045
Let's block this on migrating out of python to the Go client, since it's going to be slightly more annoying to support it in python.

Sign in to add a comment