I rolled back the version rather than reverting the CL FYI.

the CL affects only get_builders endpoint in default service, while the OOM happens in backend. I double CL is the culprit, but there may be other CLs (e.g. in components) that affect this.

it is also possible that there is a huge swarming task that buildbucket cannot process

Here's the graph of 500's in pcon:  http://shortn/_ga3EL2HtAW

Unfortunately ts_mon can't log OOMs, there's no data there.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/1d74025afa51224395469d6c03e36c72292a0f36

commit 1d74025afa51224395469d6c03e36c72292a0f36
Author: Nodir Turakulov <nodir@google.com>
Date: Tue Apr 10 20:30:52 2018

[buildbucket] limit concurrent counting queries

cr-buildbucket has 2K+ Builder entities, which results in 8K+ Datastore
counting queries. Currently they are executed at the same time and it causes
OOMs.

Limit the number of concurrent queries to 100.

R=jchinlee@chromium.org, tandrii@chromium.org

Bug:  831210 
Change-Id: Id4071af15dfbb76503e4e552dc21fa11505842fa
Reviewed-on: https://chromium-review.googlesource.com/1005727
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/1d74025afa51224395469d6c03e36c72292a0f36/appengine/cr-buildbucket/metrics.py
[modify] https://crrev.com/1d74025afa51224395469d6c03e36c72292a0f36/appengine/cr-buildbucket/test/metrics_test.py