Has anything been done to fix this yet? Or will it run out-of-quota again today? E.g. during emea time?

This is P0 due to severity, even if a fix by waiting is applicable.

Re 1: I'm worried since we seemed to have reached the quota at 8pm PST on a low-traffic weekend. In Monday this might differ.

Issue 777239 has been merged into this issue.

I can see CQ has landed one patch so far, so CQ seems to be up now. There should
be a long backlog, though.

I'll be watching the tree closely (as a sheriff).

Issue 777281 has been merged into this issue.

CQ is steadily landing patches, and I can see no problem that requires immediate
attention (from the sheriff's point of view). Reducing the priority.

Do we have any idea why we ran out of quota?

 Issue 777237  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/f07f4a1446eadefed3d8e8315810b078bc9350ca

commit f07f4a1446eadefed3d8e8315810b078bc9350ca
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Oct 23 16:11:25 2017

Revert "[buildbucket] include annotations in Build"

This reverts commit 3fcdd4db20d660afd545d7809d678e0db7276203.

Reason for revert: suspecting that it caused 18K times higher OOM rate

Original change's description:
> [buildbucket] include annotations in Build
> 
> Milo builder page that sends a query to buildbucket, needs to render "Info"
> column which is annotation's root step Text. (that's what build page does).
> For that it needs an annotation in the buildbucket repsonse.
> 
> The annotation were excluded because we might hit the datastore entity size
> limit. iannucci@ and nodir@ made estimations based on one of long builds,
> compressed it and we were still way below limit. Even if we hit the limit,
> pubsub handler will start failing for that particular build and the build
> won't be updated. We will be able to apply a fix and eventually the build will
> be processed.
> 
> Stop excluding annotation from build_run_result and compress result_details
> property.
> 
> R=​iannucci@chromium.org
> Bug:  769338 
> Change-Id: I8261e0544a36e3c62f59da6fd7fdd50002d8f8ef
> Reviewed-on: https://chromium-review.googlesource.com/691203
> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
> Commit-Queue: Nodir Turakulov <nodir@chromium.org>

TBR=iannucci@chromium.org,vadimsh@chromium.org,nodir@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  769338 ,  777271 
Change-Id: I71a35d5fd7f548b2ab3defbf21ebbbcfb49135d3
Reviewed-on: https://chromium-review.googlesource.com/731796
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/f07f4a1446eadefed3d8e8315810b078bc9350ca/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/f07f4a1446eadefed3d8e8315810b078bc9350ca/appengine/cr-buildbucket/model.py

update: the issue is handled, as of 9:03AM.

postmortem https://docs.google.com/document/d/1HSZN6UJYFGD7FeRoe26g-GyBoniRuMWIloruhdiklZo/edit#

I will keep this open until all permanent fixes land

also, as of 11:12AM, 60% of the quota is already exhausted and the next reset is 13h. There is a chance that we will have to manually reset quota once again today (cannot be done now, we are not close enough to 100%)

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/4056ce32cc6599c654a14b9641820b57d8da2856

commit 4056ce32cc6599c654a14b9641820b57d8da2856
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Oct 23 20:25:07 2017

[luci-migration] reduce buildbucket egress

luci-migration is the biggest contributer to buildbucket egress.

Make buildbucket responses partial.

Bug:  777271 
TBR=tandrii@chromium.org
Change-Id: I9ca48bb1f15cda37f7623a1e64fcc84010c792a3
Reviewed-on: https://chromium-review.googlesource.com/731970
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/4056ce32cc6599c654a14b9641820b57d8da2856/go/src/infra/appengine/luci-migration/analysis/group.go
[modify] https://crrev.com/4056ce32cc6599c654a14b9641820b57d8da2856/go/src/infra/appengine/luci-migration/app/analysis.go
[modify] https://crrev.com/4056ce32cc6599c654a14b9641820b57d8da2856/go/src/infra/appengine/luci-migration/analysis/group_test.go
[modify] https://crrev.com/4056ce32cc6599c654a14b9641820b57d8da2856/go/src/infra/appengine/luci-migration/analysis/tryjobs.go
[modify] https://crrev.com/4056ce32cc6599c654a14b9641820b57d8da2856/go/src/infra/appengine/luci-migration/analysis/compare.go

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/7e775a69cf6a4a6e595b63dca91dc1dc4148529b

commit 7e775a69cf6a4a6e595b63dca91dc1dc4148529b
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Oct 23 21:47:00 2017

Issue 777178 has been merged into this issue.
Issue 777252 has been merged into this issue.
Issue 777282 has been merged into this issue.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/839a7dbf4951f151e62b16260dabe912f525d333

commit 839a7dbf4951f151e62b16260dabe912f525d333
Author: Andrii Shyshkalov <tandrii@google.com>
Date: Mon Oct 23 23:14:01 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/df163e4373c343c379feff42b8d4e1710dd4b44f

commit df163e4373c343c379feff42b8d4e1710dd4b44f
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Oct 23 23:26:26 2017

[buildbucket] tolerate empty status

A partial buildbucket response may not have status. In that case return zero
which happens to be Success.

Bug:  777271 
Change-Id: I34b5e5a8e817fdf69b55104e4cebc8357e4f58e6
Reviewed-on: https://chromium-review.googlesource.com/734316
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/df163e4373c343c379feff42b8d4e1710dd4b44f/buildbucket/status.go
[modify] https://crrev.com/df163e4373c343c379feff42b8d4e1710dd4b44f/buildbucket/status_test.go

buildbucket exhausted 90% of its quota for today. The "Reset Quota" button should appear when "App is close to its budget", according to https://sites.google.com/a/google.com/cloud-admin/get-started/emergency-reset-of-daily-spending-limit

never mind, i was looking in a wrong place. The quota was reset. We should not hit the quota tomorrow.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/9ab65c8eee98998d6900215f5ee1e053878b63d1

commit 9ab65c8eee98998d6900215f5ee1e053878b63d1
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Oct 26 21:27:33 2017

[luci-migration] fetch failure_reason and cancelation_reason

Build status is parsed incorrectly without these fields

Bug:  777271 
Change-Id: Iefa96fb6d4382148c215a372e2c41cb3e8784662
Reviewed-on: https://chromium-review.googlesource.com/734482
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/9ab65c8eee98998d6900215f5ee1e053878b63d1/go/src/infra/appengine/luci-migration/analysis/tryjobs.go

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/c2fd5dcde86546d3b7727b8c56056c2536bfee09

commit c2fd5dcde86546d3b7727b8c56056c2536bfee09
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Oct 27 19:51:35 2017