Error on the server side:

2017-12-21 13:16:42.958 CET goroutine 782512:
#0 go.chromium.org/luci/common/api/gitiles/gitiles.go:438 - gitiles.(*Client).get()
  reason: failed to fetch https://chromium.googlesource.com/a/chromium/src/+log/refs%2Fheads%2Fmaster?format=JSON, status code 429
  tag["HTTP Status code"]: 429
#1 go.chromium.org/luci/common/api/gitiles/gitiles.go:279 - gitiles.(*Client).rawLog()
#2 go.chromium.org/luci/common/api/gitiles/gitiles.go:249 - gitiles.(*Client).Log()
From frame 0 to 3, the following wrappers were found:
  internal reason: MultiError 1/1: following first non-nil error.
#3 go.chromium.org/luci/milo/git/history.go:187 - git.GetHistory.func1()
  reason: GetHistory
#4 go.chromium.org/luci/milo/git/history.go:129 - git.protoCache()
#5 go.chromium.org/luci/milo/git/history.go:209 - git.GetHistory()
#6 go.chromium.org/luci/milo/frontend/view_console.go:178 - frontend.consoleRowCommits()
#7 go.chromium.org/luci/milo/frontend/view_console.go:259 - frontend.console.func1.3()
#8 go.chromium.org/luci/common/sync/parallel/runner.go:51 - parallel.(*WorkItem).execute()
#9 go.chromium.org/luci/common/sync/parallel/runner.go:145 - parallel.(*Runner).dispatchLoopBody.func1()
#10 runtime/asm_amd64.s:2211 - runtime.goexit()

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/021f916c747ab47264f5489580531a4cb7c0b132

commit 021f916c747ab47264f5489580531a4cb7c0b132
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Dec 21 15:58:15 2017

Reland "[milo] disable API emulation"

This is a reland of c9ec9cb9f8f60b749d3ecdb468fc05ae7f491fbd
Original change's description:
> [milo] disable API emulation
>
> It causes Gitiles quota exhaustion.
>
> Bug: 793514
> Change-Id: Idc72356dfde7d6fce4f04040e2b005f174032ee3
> Reviewed-on: https://chromium-review.googlesource.com/834791
> Commit-Queue: Nodir Turakulov <nodir@chromium.org>
> Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

Bug:  796879 
Change-Id: Iaa8293cebcad0819b253d59e1b3078bd57cbd307
TBR=iannucci@chromium.org
Reviewed-on: https://chromium-review.googlesource.com/840040
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/021f916c747ab47264f5489580531a4cb7c0b132/milo/frontend/routes.go

request to reset quota: b/70926682

a lot of Gitiles cache misses starting from 1AM http://shortn/_jcYVQ7jQDs

Quota was reset, https://ci.chromium.org/p/chromium/g/main/console is available again

still happening

Issue 796854 has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/56ed1aecbe064b51aa71d5437b40d0307d3e6e8d

commit 56ed1aecbe064b51aa71d5437b40d0307d3e6e8d
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Dec 21 21:41:21 2017

[gitiles] log response on HTTP 429

Bug:  796879 
Change-Id: I13a86f532b1d6249293ba9e61eaa972d534e2c12
Reviewed-on: https://chromium-review.googlesource.com/840722
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/56ed1aecbe064b51aa71d5437b40d0307d3e6e8d/common/api/gitiles/gitiles.go

Nodir, can you describe what the issue is here? Is it related to the gitiles out of quota issues from API emulation?

The issue is that we exhaust Gitiles quota. It might or might not be triggered by Emulation. I am working on refactoring gitiles and git packages and then going to use git package in emulation so that we have better visibility into what causes quota exhaustion and to improve the way we cache things.

Added tracking labels. Marked as task force blocker since this is blocked on API emulation.

(this is not blocked on emulation. It is the opposite)

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/11b5800174a8fc8e3dc600e23f69fc5a8d05fa18

commit 11b5800174a8fc8e3dc600e23f69fc5a8d05fa18
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jan 05 17:18:59 2018

[milo] improve git package

- fix incorrect git commit hash in history.go
- delete console_git_info.proto. It was used in package git that
  is not specific to consoe. It was an abstraction leak.
- delete code that mirrors gitiles commits to console protos and reuse
  one in gitiles package.
- cache by all commits in a log
- exclude limit from the cache key. It might cause cache misses.
- use new gitiles.NewRESTClient API.

Bug:  796879 
Change-Id: I469da1cb3fbc86a52162eb1d1efe2dca64c46231
Reviewed-on: https://chromium-review.googlesource.com/850634
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>

[delete] https://crrev.com/2c110abe941d6732369c7655b82fa65bc97d7087/milo/api/proto/console_git_info.pb.go
[delete] https://crrev.com/2c110abe941d6732369c7655b82fa65bc97d7087/milo/api/proto/console_git_info.proto
[modify] https://crrev.com/11b5800174a8fc8e3dc600e23f69fc5a8d05fa18/milo/buildsource/buildbot/buildstore/blame.go
[modify] https://crrev.com/11b5800174a8fc8e3dc600e23f69fc5a8d05fa18/milo/buildsource/buildbucket/build.go
[modify] https://crrev.com/11b5800174a8fc8e3dc600e23f69fc5a8d05fa18/milo/common/model/build_summary.go
[modify] https://crrev.com/11b5800174a8fc8e3dc600e23f69fc5a8d05fa18/milo/frontend/middleware.go
[modify] https://crrev.com/11b5800174a8fc8e3dc600e23f69fc5a8d05fa18/milo/frontend/routes.go
[modify] https://crrev.com/11b5800174a8fc8e3dc600e23f69fc5a8d05fa18/milo/frontend/view_console.go
[add] https://crrev.com/11b5800174a8fc8e3dc600e23f69fc5a8d05fa18/milo/git/client.go
[delete] https://crrev.com/2c110abe941d6732369c7655b82fa65bc97d7087/milo/git/history.go
[add] https://crrev.com/11b5800174a8fc8e3dc600e23f69fc5a8d05fa18/milo/git/log.go
[add] https://crrev.com/11b5800174a8fc8e3dc600e23f69fc5a8d05fa18/milo/git/log_test.go
[modify] https://crrev.com/11b5800174a8fc8e3dc600e23f69fc5a8d05fa18/milo/srcman/blamelist.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/5ab119371d9f389c3a47fbc7a06e685ccfdf8177

commit 5ab119371d9f389c3a47fbc7a06e685ccfdf8177
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Jan 05 18:07:53 2018

[milo] use git package in buildstore/blame.go

Use git package in buildstore/blame.go for caching

R=iannucci@chromium.org

Bug:  796879 
Change-Id: I15f9865ad45fdebdf357464fb7c08c12ebbaeab1
Reviewed-on: https://chromium-review.googlesource.com/850994
Commit-Queue: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/5ab119371d9f389c3a47fbc7a06e685ccfdf8177/milo/buildsource/buildbot/buildstore/blame.go
[modify] https://crrev.com/5ab119371d9f389c3a47fbc7a06e685ccfdf8177/milo/buildsource/buildbot/buildstore/query.go

git cache utilization graph before API emulation is enabled: http://shortn/_iKxDcgOV9r

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/db1e023a6c616d8112eb21e3d8d8664afcc17b11

commit db1e023a6c616d8112eb21e3d8d8664afcc17b11
Author: Nodir Turakulov <nodir@chromium.org>
Date: Fri Jan 05 18:21:43 2018

Revert "Reland "[milo] disable API emulation""

This reverts commit 021f916c747ab47264f5489580531a4cb7c0b132.

Reason for revert: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/850994
i.e. the reason of prev revert is addressed, which was gitiles quota exhaustion

Original change's description:
> Reland "[milo] disable API emulation"
> 
> This is a reland of c9ec9cb9f8f60b749d3ecdb468fc05ae7f491fbd
> Original change's description:
> > [milo] disable API emulation
> >
> > It causes Gitiles quota exhaustion.
> >
> > Bug: 793514
> > Change-Id: Idc72356dfde7d6fce4f04040e2b005f174032ee3
> > Reviewed-on: https://chromium-review.googlesource.com/834791
> > Commit-Queue: Nodir Turakulov <nodir@chromium.org>
> > Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
> 
> Bug:  796879 
> Change-Id: Iaa8293cebcad0819b253d59e1b3078bd57cbd307
> TBR=iannucci@chromium.org
> Reviewed-on: https://chromium-review.googlesource.com/840040
> Reviewed-by: Nodir Turakulov <nodir@chromium.org>

TBR=vadimsh@chromium.org,nodir@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  796879 
Change-Id: I8f8101acdf6502ad8854dec8c5dc6639dc47fcf9
Reviewed-on: https://chromium-review.googlesource.com/852416
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/db1e023a6c616d8112eb21e3d8d8664afcc17b11/milo/frontend/routes.go