After discussion with Nodir:

Currently tricium is using cfgclient ("go.chromium.org/luci/config/server/cfgclient") to make requests to luci-config. This library uses caching and doesn't always actually make requests to luci-config.

Actually making requests to luci-config is slow; fetching configs from luci-config is not really meant to be fast. Other services such as luci-notify handle this by using cron-jobs that regularly fetch new configs and store configs in datastore.

To do this for tricium, we would want to:

 - Add a cron job to fetch and store configs.
 - Instead of using cfgclient to fetch clients in other places: fetch configs from datastore.

See:
https://cs.chromium.org/chromium/infra/go/src/go.chromium.org/luci/luci_notify/config/config.go
https://cs.chromium.org/chromium/infra/go/src/go.chromium.org/luci/luci_notify/frontend/cron.yaml

Short-term fix to clear caches: Upload new version; clear memcache; delete old versions.

Update; tried uploading new version / deleting old versions, and clearing memcache, but it still seems to be using stale version of project config for now.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d

commit c4700e49db12f969ad57422cfc5af3fc3cc7fb0d
Author: Quinten Yearsley <qyearsley@chromium.org>
Date: Thu Jul 19 21:20:38 2018

[tricium] Manage configs explicitly with a cron job and datastore

Background: Currently Tricium uses the
go.chromium.org/luci/config/server/cfgclient library
for fetching configs when they're needed. This appears to
have two disadvantages:

1. Ocasionally when a config is actually fetched, it takes
   about 5 seconds, occasionally adding delay to other
   operations (polling, starting analysis, etc.)
2. Sometimes a new config is not fetched when it should be,
   and a stale cached version is used ( crbug.com/852001 ).

This CL changes it so that configs are explicitly stored and
managed in the Tricium datastore, and updated periodically
by a cron job.

This CL keeps the interface and mocks the same, so that
other code that uses configs doesn't have to care about
how the configs are fetched or managed.

Bug:  852001 

Change-Id: I6a8740ce2edfb5691a6b98729a1611b0b4985d28
Reviewed-on: https://chromium-review.googlesource.com/1132547
Commit-Queue: Quinten Yearsley <qyearsley@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/api/v1/gen.go
[add] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/appengine/common/config/update_test.go
[modify] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/appengine/frontend/rpc_report_test.go
[modify] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/api/v1/tricium.pb.go
[add] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/api/v1/proto_gae.gen.go
[modify] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/appengine/common/config/provider.go
[modify] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/appengine/frontend/cron.yaml
[add] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/appengine/common/config/storage.go
[modify] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/appengine/config/init.go
[add] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/appengine/common/config/storage_test.go
[modify] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/api/v1/pb.discovery.go
[add] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/appengine/common/config/update.go
[add] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/appengine/config/update.go
[modify] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/appengine/common/config/config.infra_testing
[add] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/appengine/common/config/workflow.go
[add] https://crrev.com/c4700e49db12f969ad57422cfc5af3fc3cc7fb0d/go/src/infra/tricium/api/v1/README.md

Config management is now changed.

Example of cron job logs after deploying:

First run:

 2018-07-19 15:11:40.400 PDT
 Found new service config. :: {"revision":"07c3044001c319a6a2782526891e6c25e1e6a26c"}
 2018-07-19 15:11:40.896 PDT
 Got config revisions for 7 projects.
 2018-07-19 15:11:41.119 PDT
 Found new project config. :: {"project":"luci-go", "revision":"0f4483886942b6ca660b2e4c1c8a6c400daffc1b"}
 2018-07-19 15:11:41.120 PDT
 Found new project config. :: {"project":"gerrit-tricium-integration", 
 "revision":"54bf8ddf9bcb4ffc748dd6fead98ececa95c98e5"}
 2018-07-19 15:11:41.142 PDT
 Found new project config. :: {"project":"chromiumos", "revision":"fa2b1a19c53b67a84acd5414f7024a632962414b"}
 2018-07-19 15:11:41.299 PDT
 Found new project config. :: {"project":"luci-py", "revision":"b8dc03b381227d127c6124fa5a97977577e912c2"}
 2018-07-19 15:11:41.317 PDT
 Found new project config. :: {"project":"playground-gerrit-tricium-demo", 
 "revision":"e189db0e778247b18cc9bf4a68985a8134c78dfb"}
 2018-07-19 15:11:41.324 PDT
 Found new project config. :: {"project":"pdfium", "revision":"d6699d70624d82aff65c1464c8800c291fa29148"}
 2018-07-19 15:11:41.350 PDT
 Found new project config. :: {"project":"fuchsia", "revision":"f19c18af2fe27fdb682e8f6c3e696928825a0660"}

Second run:

 2018-07-19 15:12:09.706 PDT
 Got config revisions for 7 projects.
 [nothing more]

Configs can be inspected in datastore and can be explicitly refreshed from the cron job page; it should now be when configs are actually refreshed.