I filed issue 786735 as I think the way I approached this up to now with just making all RPCs asynchronous is the wrong way.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/9c3558eccfdc0018a72b5659e945d0726b071f75

commit 9c3558eccfdc0018a72b5659e945d0726b071f75
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Wed Nov 22 20:26:38 2017

swarming: make task_queues.assert_bot an async function

There's no immediate benefit, the sole caller calls it synchronously, but this
enables more parallelism at the call site in a follow up, while keeping this CL
smaller.

This removes the cap of on-going RPCs in _rebuild_bot_cache_async(). If this
starts to throw I'll figure a way to add it back.

Increase batch sizes to 64 to reduce the RTT effect.

Bug: 786027
Change-Id: Ie6399466810a229eb9a24f10073dcf7633a1e95c
Reviewed-on: https://chromium-review.googlesource.com/774759
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/9c3558eccfdc0018a72b5659e945d0726b071f75/appengine/swarming/handlers_bot.py
[modify] https://crrev.com/9c3558eccfdc0018a72b5659e945d0726b071f75/appengine/swarming/server/task_queues.py
[modify] https://crrev.com/9c3558eccfdc0018a72b5659e945d0726b071f75/appengine/swarming/server/task_queues_test.py
[modify] https://crrev.com/9c3558eccfdc0018a72b5659e945d0726b071f75/appengine/swarming/server/task_scheduler_test.py
[modify] https://crrev.com/9c3558eccfdc0018a72b5659e945d0726b071f75/appengine/swarming/server/task_to_run_test.py

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot