Consider batching service RPC calls. |
|||
Issue descriptionA given log stream will result from anywhere between 2 to 4 RPC calls to the backend service. Each RPC is fairly small, and it is wasteful to pay for and accommodate scaling to handle each individual RPC. Consider sending them in batch. We can use the cloud library's "bundler" to facilitate: https://godoc.org/google.golang.org/api/support/bundler#Bundler.Add Plan is: 1) Introduce a new service RPC called "Batch" which can embed other service RPC types. 2) Implement a dispatcher in LogDog to handle this RPC. 3) Update Collector and Archivist to use a "bundler" and send Batch RPCs instead of individual ones.
,
Oct 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/eb44c1c990894cd4ba27ed1218519d85cfddfa7f commit eb44c1c990894cd4ba27ed1218519d85cfddfa7f Author: Dan Jacques <dnj@chromium.org> Date: Fri Oct 06 00:55:03 2017 [logdog] Implement Batch service endpoint. Implement the Batch service endpoint. The endpoint is a parallel mux that dispatches requests in parallel to their underlying handlers. BUG= chromium:771845 TEST=None Change-Id: I64d897a85427d4e576988649c7d6922612dce274 Reviewed-on: https://chromium-review.googlesource.com/702158 Commit-Queue: Daniel Jacques <dnj@chromium.org> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> [modify] https://crrev.com/eb44c1c990894cd4ba27ed1218519d85cfddfa7f/logdog/appengine/coordinator/endpoints/services/batch.go [modify] https://crrev.com/eb44c1c990894cd4ba27ed1218519d85cfddfa7f/logdog/appengine/coordinator/endpoints/services/loadStream.go [modify] https://crrev.com/eb44c1c990894cd4ba27ed1218519d85cfddfa7f/logdog/appengine/coordinator/endpoints/services/service.go
,
Oct 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/41e2621d63a20b918ecdd240c6bbdd5f61f8a93e commit 41e2621d63a20b918ecdd240c6bbdd5f61f8a93e Author: Dan Jacques <dnj@chromium.org> Date: Fri Oct 06 00:59:36 2017 [logdog] Enable Collector RPC batching. Update the Collector to enable RPC batching. We do this by implementing a "bundleServicesClient", a client to the Coordinator's Services endpoint that automatically bundles RPCs using a Bundler. This will significantly reduce the number of distinct RPCs that are sent to the Coordinator backend. BUG= chromium:771845 TEST=None Change-Id: I3942490d7379bf9543b946c8eb4a5cd9d8966e0c Reviewed-on: https://chromium-review.googlesource.com/702159 Commit-Queue: Daniel Jacques <dnj@chromium.org> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> [add] https://crrev.com/41e2621d63a20b918ecdd240c6bbdd5f61f8a93e/logdog/server/bundleServicesClient/client.go [modify] https://crrev.com/41e2621d63a20b918ecdd240c6bbdd5f61f8a93e/logdog/server/cmd/logdog_collector/main.go [modify] https://crrev.com/41e2621d63a20b918ecdd240c6bbdd5f61f8a93e/logdog/server/service/service.go
,
Oct 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/7f0b9c6545aeda33fe0069f3dc139c70e1c5338a commit 7f0b9c6545aeda33fe0069f3dc139c70e1c5338a Author: Dan Jacques <dnj@chromium.org> Date: Fri Oct 06 12:05:36 2017 [logdog] Batch accommodates size issues. Update batch protocol to accommodate potential size issues. An AppEngine response is limited in size, and may not be able to populate the full set of repsonses for a given request. Update the protocol to be able to accommodate this. BUG= chromium:771845 TEST=None Change-Id: I65594883fabe23f6fa17d3f94d2ab4428ef4219c Reviewed-on: https://chromium-review.googlesource.com/704157 Commit-Queue: Daniel Jacques <dnj@chromium.org> Reviewed-by: Nodir Turakulov <nodir@chromium.org> [add] https://crrev.com/7f0b9c6545aeda33fe0069f3dc139c70e1c5338a/common/gcloud/gae/quota.go [modify] https://crrev.com/7f0b9c6545aeda33fe0069f3dc139c70e1c5338a/logdog/api/endpoints/coordinator/services/v1/pb.discovery.go [modify] https://crrev.com/7f0b9c6545aeda33fe0069f3dc139c70e1c5338a/logdog/api/endpoints/coordinator/services/v1/service.pb.go [modify] https://crrev.com/7f0b9c6545aeda33fe0069f3dc139c70e1c5338a/logdog/api/endpoints/coordinator/services/v1/service.proto [modify] https://crrev.com/7f0b9c6545aeda33fe0069f3dc139c70e1c5338a/logdog/appengine/coordinator/endpoints/services/batch.go [modify] https://crrev.com/7f0b9c6545aeda33fe0069f3dc139c70e1c5338a/logdog/server/bundleServicesClient/client.go
,
Oct 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/6af02cfe74956cb370cc47bcd6728e8aa401eb13 commit 6af02cfe74956cb370cc47bcd6728e8aa401eb13 Author: Dan Jacques <dnj@chromium.org> Date: Fri Oct 06 14:27:56 2017 [logdog] Add RPC batching to Archivist. Update Archivist to use RPC batching. BUG= chromium:771845 TEST=None Change-Id: If2b111517683d6f7286aae3d204c22d91f7983fa Reviewed-on: https://chromium-review.googlesource.com/704167 Reviewed-by: Nodir Turakulov <nodir@chromium.org> Commit-Queue: Daniel Jacques <dnj@chromium.org> [modify] https://crrev.com/6af02cfe74956cb370cc47bcd6728e8aa401eb13/logdog/server/cmd/logdog_archivist/main.go
,
Oct 10 2017
(This work seems started so I'm just moving it out of the triage queue.)
,
Oct 10 2017
This is fixed.
,
Oct 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/a90d36d07f65098da34b600f6cfb8a5f05199ac5 commit a90d36d07f65098da34b600f6cfb8a5f05199ac5 Author: Dan Jacques <dnj@chromium.org> Date: Tue Oct 17 04:06:05 2017 [logdog] Dispatch bundle request in parallel. Currently, bundle requests are dispatched in series, with the bundler waiting for the request to complete prior to dispatching further requests. Have the bundler client dispatch in parallel so a single RPC, even bundled, doesn't bottleneck the service. This will increase concurrency, but it will still be bounded by the maximum consumer rate. This should increase services' throughput to the rate they were processing prior to bundling. BUG= chromium:771845 TEST=None R=iannucci@chromium.org, nodir@chromium.org Change-Id: I573610da7f212e75b3e4971d4a84a24457197b6e Reviewed-on: https://chromium-review.googlesource.com/722559 Commit-Queue: Daniel Jacques <dnj@chromium.org> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> [modify] https://crrev.com/a90d36d07f65098da34b600f6cfb8a5f05199ac5/logdog/server/bundleServicesClient/client.go |
|||
►
Sign in to add a comment |
|||
Comment 1 by bugdroid1@chromium.org
, Oct 6 2017