New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 682465 link

Starred by 1 user
Status: Fixed
Owner:
d...@chromium.org
Closed: Mar 2017
Cc:
█ iannucci@chromium.org
vadimsh@chromium.org
hinoka@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Update LogDog services to use new packages, API fixes.

Project Member Reported by d...@chromium.org, Jan 18 2017 
LogDog's Collector and Archivist services have been running without update for a while now. Upstream package changes to the way gRPC methods are called from their wrapping libraries mean that, were these services to be rebuilt from HEAD and deployed, they probably would not work. Similar issues were encountered for AppEngine apps.

Additionally, several libraries have been released and/or matured, including:
- The new luci_config cfgclient package.
- The luci/gae cloud datastore implementation.
- The datastore config cache layer.
- The service runtime profiling helper.

These should be integrated into the LogDog services for unity and superior reliability.

Comment 1 by d...@chromium.org, Jan 18 2017 

Additionally, IAM works with BigTable now, so the need to have a separate service account for the dev instance to talk to prod's BigTable is gone.
Project Member

Comment 2 by bugdroid1@chromium.org, Jan 20 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/4590aea3339388a2946d2398c536d71789a0ca0a

commit 4590aea3339388a2946d2398c536d71789a0ca0a
Author: dnj <dnj@chromium.org>
Date: Fri Jan 20 00:03:04 2017

datastorecache: Add Locker field, public settings.

Add the Locker field to datastorecache. Previously, the Locker was not
configurable, and defaulted to memlock. However, locking is optional for
fetching configuration data. This decouples datastorecache from the
requirement that it has a working memcache, allowing it to be used by
other datastore implementations such as cloud.

This also changes the names of the gaeconfig datastore cache settings to
be public so that they can be used by external packages that want to
process the gaeconfig settings.

BUG= chromium:682465 
TEST=unit

Review-Url: https://codereview.chromium.org/2645763002

[modify] https://crrev.com/4590aea3339388a2946d2398c536d71789a0ca0a/appengine/datastorecache/cache.go
[modify] https://crrev.com/4590aea3339388a2946d2398c536d71789a0ca0a/appengine/datastorecache/cache_test.go
[modify] https://crrev.com/4590aea3339388a2946d2398c536d71789a0ca0a/appengine/datastorecache/handler.go
[add] https://crrev.com/4590aea3339388a2946d2398c536d71789a0ca0a/appengine/datastorecache/lock.go
[modify] https://crrev.com/4590aea3339388a2946d2398c536d71789a0ca0a/appengine/datastorecache/model.go
[modify] https://crrev.com/4590aea3339388a2946d2398c536d71789a0ca0a/appengine/datastorecache/util_test.go
[modify] https://crrev.com/4590aea3339388a2946d2398c536d71789a0ca0a/luci_config/appengine/backend/datastore/ds.go
[modify] https://crrev.com/4590aea3339388a2946d2398c536d71789a0ca0a/luci_config/appengine/gaeconfig/default.go
[modify] https://crrev.com/4590aea3339388a2946d2398c536d71789a0ca0a/luci_config/appengine/gaeconfig/default_test.go
[modify] https://crrev.com/4590aea3339388a2946d2398c536d71789a0ca0a/luci_config/appengine/gaeconfig/settings.go
Project Member

Comment 3 by bugdroid1@chromium.org, Jan 20 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/50864029fbaf03ff86650dd64fdd7cfacc7d9a72

commit 50864029fbaf03ff86650dd64fdd7cfacc7d9a72
Author: dnj <dnj@chromium.org>
Date: Fri Jan 20 00:19:42 2017

server/auth: Add TokenSource call.

Add a call to return an oauth2 AsSelf TokenSource from the server/auth
package.

This allows the "server/auth" package to be used with Google API
clients. Specifying manual per-RPC credentials hits a problem with the
automagic setup detailed here:
https://github.com/google/google-api-go-client/issues/186

Consqeuently, gRPC service libraries (Cloud Datastore, BigTable, Pub/Sub,
etc.) require a TokenSource to be provided explicitly.

Also make all auth tests run in parallel :)

BUG= chromium:682465 
TEST=None

Review-Url: https://codereview.chromium.org/2646543003

[modify] https://crrev.com/50864029fbaf03ff86650dd64fdd7cfacc7d9a72/common/auth/auth.go
[modify] https://crrev.com/50864029fbaf03ff86650dd64fdd7cfacc7d9a72/server/auth/addr_test.go
[modify] https://crrev.com/50864029fbaf03ff86650dd64fdd7cfacc7d9a72/server/auth/auth_test.go
[modify] https://crrev.com/50864029fbaf03ff86650dd64fdd7cfacc7d9a72/server/auth/cache_test.go
[modify] https://crrev.com/50864029fbaf03ff86650dd64fdd7cfacc7d9a72/server/auth/client.go
[modify] https://crrev.com/50864029fbaf03ff86650dd64fdd7cfacc7d9a72/server/auth/client_test.go
[modify] https://crrev.com/50864029fbaf03ff86650dd64fdd7cfacc7d9a72/server/auth/context_test.go
[modify] https://crrev.com/50864029fbaf03ff86650dd64fdd7cfacc7d9a72/server/auth/delegation_test.go
[modify] https://crrev.com/50864029fbaf03ff86650dd64fdd7cfacc7d9a72/server/auth/handlers_test.go
[modify] https://crrev.com/50864029fbaf03ff86650dd64fdd7cfacc7d9a72/server/auth/state_test.go
Project Member

Comment 4 by bugdroid1@chromium.org, Jan 20 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/b6f14156e798268e632ddbd6376842144c9c4ae6

commit b6f14156e798268e632ddbd6376842144c9c4ae6
Author: dnj <dnj@chromium.org>
Date: Fri Jan 20 00:42:42 2017

config: Update remote URL handling.

Remove ServiceURL from "common/config".Interface. The remote inteface
now takes in a "host" (e.g., "example.com") and a boolean, "insecure",
which determines if HTTPS (false) or HTTP (true) should be used.

Update "luci_config/server/cfgclient" to use the "host" and "insecure"
properties.

Update "gaeconfig" to use a host field. For backwards-compatibility
purposes, it will extract the host field from a URL if it has one.

Update the LogDog services API to include the configuration service
host in services.GetConfig.

BUG= chromium:682465 
TEST=unit

Review-Url: https://codereview.chromium.org/2643803003

[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/common/config/impl/filesystem/fs.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/common/config/impl/memory/memory.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/common/config/impl/remote/remote.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/common/config/impl/remote/remote_test.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/common/config/interface.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/logdog/api/endpoints/coordinator/services/v1/pb.discovery.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/logdog/api/endpoints/coordinator/services/v1/service.pb.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/logdog/api/endpoints/coordinator/services/v1/service.proto
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/logdog/appengine/coordinator/endpoints/services/getConfig.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/logdog/appengine/coordinator/endpoints/services/getConfig_test.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/logdog/server/service/config/flag.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/logdog/server/service/config/flag_test.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/luci_config/appengine/gaeconfig/default.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/luci_config/appengine/gaeconfig/settings.go
[add] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/luci_config/appengine/gaeconfig/settings_test.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/luci_config/server/cfgclient/backend/client/client.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/luci_config/server/cfgclient/backend/client/client_test.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/luci_config/server/cfgclient/backend/testconfig/local_service.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/luci_config/server/cfgclient/backend/testconfig/local_service_test.go
Project Member

Comment 5 by bugdroid1@chromium.org, Jan 20 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/b6f14156e798268e632ddbd6376842144c9c4ae6

commit b6f14156e798268e632ddbd6376842144c9c4ae6
Author: dnj <dnj@chromium.org>
Date: Fri Jan 20 00:42:42 2017

config: Update remote URL handling.

Remove ServiceURL from "common/config".Interface. The remote inteface
now takes in a "host" (e.g., "example.com") and a boolean, "insecure",
which determines if HTTPS (false) or HTTP (true) should be used.

Update "luci_config/server/cfgclient" to use the "host" and "insecure"
properties.

Update "gaeconfig" to use a host field. For backwards-compatibility
purposes, it will extract the host field from a URL if it has one.

Update the LogDog services API to include the configuration service
host in services.GetConfig.

BUG= chromium:682465 
TEST=unit

Review-Url: https://codereview.chromium.org/2643803003

[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/common/config/impl/filesystem/fs.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/common/config/impl/memory/memory.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/common/config/impl/remote/remote.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/common/config/impl/remote/remote_test.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/common/config/interface.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/logdog/api/endpoints/coordinator/services/v1/pb.discovery.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/logdog/api/endpoints/coordinator/services/v1/service.pb.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/logdog/api/endpoints/coordinator/services/v1/service.proto
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/logdog/appengine/coordinator/endpoints/services/getConfig.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/logdog/appengine/coordinator/endpoints/services/getConfig_test.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/logdog/server/service/config/flag.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/logdog/server/service/config/flag_test.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/luci_config/appengine/gaeconfig/default.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/luci_config/appengine/gaeconfig/settings.go
[add] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/luci_config/appengine/gaeconfig/settings_test.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/luci_config/server/cfgclient/backend/client/client.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/luci_config/server/cfgclient/backend/client/client_test.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/luci_config/server/cfgclient/backend/testconfig/local_service.go
[modify] https://crrev.com/b6f14156e798268e632ddbd6376842144c9c4ae6/luci_config/server/cfgclient/backend/testconfig/local_service_test.go
Project Member

Comment 9 by bugdroid1@chromium.org, Jan 21 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/b4bee637a6593dd2b8d67e96cbc31a515c61f38e

commit b4bee637a6593dd2b8d67e96cbc31a515c61f38e
Author: dnj <dnj@chromium.org>
Date: Sat Jan 21 01:26:14 2017

Add independent config poller service.

Add a configuration poller service which triggers a callback when the
configuration's hash changes.

This will be used in follow-up CLs to replace other configuration
operations.

BUG= chromium:682465 
TEST=None

Review-Url: https://codereview.chromium.org/2644413002

[add] https://crrev.com/b4bee637a6593dd2b8d67e96cbc31a515c61f38e/logdog/server/service/config/poller.go
[add] https://crrev.com/b4bee637a6593dd2b8d67e96cbc31a515c61f38e/logdog/server/service/config/poller_test.go
Project Member

Comment 10 by bugdroid1@chromium.org, Jan 21 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/95c4ae33d6197d7098ab2b895ccc3a14b1b82602

commit 95c4ae33d6197d7098ab2b895ccc3a14b1b82602
Author: dnj <dnj@chromium.org>
Date: Sat Jan 21 01:43:43 2017

LogDog: Use luci_config/server packages for config

Use the luci_config/server configuration packages for configuration.
This includes:
- Using server/cfgclient to pull configurations.
- Using the server/cfgclient proccache backend for caching.

BUG= chromium:682465 
TEST=None

Review-Url: https://codereview.chromium.org/2647083003

[modify] https://crrev.com/95c4ae33d6197d7098ab2b895ccc3a14b1b82602/logdog/server/cmd/logdog_archivist/main.go
[modify] https://crrev.com/95c4ae33d6197d7098ab2b895ccc3a14b1b82602/logdog/server/cmd/logdog_collector/main.go
[delete] https://crrev.com/b4bee637a6593dd2b8d67e96cbc31a515c61f38e/logdog/server/service/config/config.go
[delete] https://crrev.com/b4bee637a6593dd2b8d67e96cbc31a515c61f38e/logdog/server/service/config/config_test.go
[delete] https://crrev.com/b4bee637a6593dd2b8d67e96cbc31a515c61f38e/logdog/server/service/config/flag.go
[delete] https://crrev.com/b4bee637a6593dd2b8d67e96cbc31a515c61f38e/logdog/server/service/config/flag_test.go
[add] https://crrev.com/95c4ae33d6197d7098ab2b895ccc3a14b1b82602/logdog/server/service/config/opts.go
[modify] https://crrev.com/95c4ae33d6197d7098ab2b895ccc3a14b1b82602/logdog/server/service/service.go
Project Member

Comment 11 by bugdroid1@chromium.org, Jan 21 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/d3d6702a41f2ae2d2256d7e541917dbe20eccec8

commit d3d6702a41f2ae2d2256d7e541917dbe20eccec8
Author: dnj <dnj@chromium.org>
Date: Sat Jan 21 01:59:29 2017

LogDog: Use live object cache for configs.

Rather than unmarshalling configurations each time, use a live-object
cache via proccache for configurations.

BUG= chromium:682465 
TEST=None

Review-Url: https://codereview.chromium.org/2648033002

[add] https://crrev.com/d3d6702a41f2ae2d2256d7e541917dbe20eccec8/logdog/server/service/config/cache.go
[modify] https://crrev.com/d3d6702a41f2ae2d2256d7e541917dbe20eccec8/logdog/server/service/service.go
Project Member

Comment 12 by bugdroid1@chromium.org, Jan 21 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/9b5c2ac8e891eedde109b1c9ec77735d77a567ae

commit 9b5c2ac8e891eedde109b1c9ec77735d77a567ae
Author: dnj <dnj@chromium.org>
Date: Sat Jan 21 02:07:03 2017

LogDog: Use datastore config cache when available.

Check if the datastore config cache is available. If it is, incorporate
it into the caching layer.

This will allow the microservices to leverage the project-local caching
for higher fault tolerance and less external reliance on hitting the
configuration service.

BUG= chromium:682465 
TEST=None

Review-Url: https://codereview.chromium.org/2648433007

[modify] https://crrev.com/9b5c2ac8e891eedde109b1c9ec77735d77a567ae/logdog/server/service/config/opts.go
[modify] https://crrev.com/9b5c2ac8e891eedde109b1c9ec77735d77a567ae/logdog/server/service/service.go
Project Member

Comment 13 by bugdroid1@chromium.org, Jan 21 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/9b5c2ac8e891eedde109b1c9ec77735d77a567ae

commit 9b5c2ac8e891eedde109b1c9ec77735d77a567ae
Author: dnj <dnj@chromium.org>
Date: Sat Jan 21 02:07:03 2017

LogDog: Use datastore config cache when available.

Check if the datastore config cache is available. If it is, incorporate
it into the caching layer.

This will allow the microservices to leverage the project-local caching
for higher fault tolerance and less external reliance on hitting the
configuration service.

BUG= chromium:682465 
TEST=None

Review-Url: https://codereview.chromium.org/2648433007

[modify] https://crrev.com/9b5c2ac8e891eedde109b1c9ec77735d77a567ae/logdog/server/service/config/opts.go
[modify] https://crrev.com/9b5c2ac8e891eedde109b1c9ec77735d77a567ae/logdog/server/service/service.go
Project Member

Comment 15 by bugdroid1@chromium.org, Jan 21 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/ee266e37c9d7e787fe9d9d72348136ada6287340

commit ee266e37c9d7e787fe9d9d72348136ada6287340
Author: dnj <dnj@chromium.org>
Date: Sat Jan 21 03:51:44 2017

luci_config: Request content only when desired.

Don't ask the config backend to load content when the user doesn't
actually ask for content.

BUG= chromium:682465 
TEST=unit

Review-Url: https://codereview.chromium.org/2642343003

[modify] https://crrev.com/ee266e37c9d7e787fe9d9d72348136ada6287340/luci_config/server/cfgclient/config.go
[modify] https://crrev.com/ee266e37c9d7e787fe9d9d72348136ada6287340/luci_config/server/cfgclient/config_test.go
Project Member

Comment 16 by bugdroid1@chromium.org, Jan 21 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/2ba0c2640117fc2b6b72724d4c2324101559c9f6

commit 2ba0c2640117fc2b6b72724d4c2324101559c9f6
Author: dnj <dnj@chromium.org>
Date: Sat Jan 21 04:48:09 2017

LogDog: Use server/auth for authentication.

Use server/auth instead of common/auth for authentication. The
server/auth package is oriented to long-running services that operate on
multiple authentication credentials throughout their lifetime. This
incorporates caching and parity with other services.

BUG= chromium:682465 
TEST=unit,development
  - Deployed to development server, seems to work.

Review-Url: https://codereview.chromium.org/2643363002

[modify] https://crrev.com/2ba0c2640117fc2b6b72724d4c2324101559c9f6/logdog/server/cmd/logdog_archivist/main.go
[modify] https://crrev.com/2ba0c2640117fc2b6b72724d4c2324101559c9f6/logdog/server/cmd/logdog_collector/main.go
[add] https://crrev.com/2ba0c2640117fc2b6b72724d4c2324101559c9f6/logdog/server/service/rt.go
[modify] https://crrev.com/2ba0c2640117fc2b6b72724d4c2324101559c9f6/logdog/server/service/service.go

Comment 17 by estaab@chromium.org, Mar 7 2017

Status: Assigned (was: Untriaged)

Comment 18 by d...@chromium.org, Mar 8 2017

Status: Fixed (was: Assigned)
►

Sign in to add a comment