Just a note that telemetry is not a good example here, it should not use gsutil, tracked in issue 549140. The rest is relevant.

boto authentication plugin isn't going to work because gsutil isn't using them. However, gsutil allows passing the OAuth2 token through boto.cfg file or environment variables OAUTH2_CLIENT_ID and OAUTH2_CLIENT_SECRET so we could possibly write a wrapper around gsutil that mints the token using LUCI_CONTEXT and, sets the environment variables and executes gsutil.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c4560bb1d0ba03c48bee086b6363da9495292cd4

commit c4560bb1d0ba03c48bee086b6363da9495292cd4
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Oct 13 02:37:50 2017

kitchen: Refactor authentication in preparation for adding more stuff.

We are about to add support for Git and Devshell authentication (for gsutil).
They require doing environment modifications and/or dropping some temporary
files and/or running background goroutines.

Extract all authentication related logic into AuthContext struct that exists
in two instances: "system" context (used by kitchen itself), and "recipe"
context (used by the user-supplied recipe).

Each such context can be launched and stopped. When it is running, it can be
"exported" into environ, thus making subprocesses inherit it. This is more than
just LUCI_CONTEXT["local_auth"].DefaultAccountID modification, since we also
need to modify Git and Devshell environment variables.

This CL is mostly refactoring except one inconsequential (in theory) change: on
Buildbot we now use the on-disk token cache to reuse -luci-system-account-json
tokens between runs. (We should have been doing this from the start, it was
oversight).

R=nodir@chromium.org
CC=phosek@chromium.org
BUG= 756224 ,  756229 

Change-Id: Ide1b3d1dd43d5918b49bf44b47f5e9a97fcf76da
Reviewed-on: https://chromium-review.googlesource.com/714888
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/c4560bb1d0ba03c48bee086b6363da9495292cd4/go/src/infra/tools/kitchen/cook_test.go
[add] https://crrev.com/c4560bb1d0ba03c48bee086b6363da9495292cd4/go/src/infra/tools/kitchen/auth.go
[modify] https://crrev.com/c4560bb1d0ba03c48bee086b6363da9495292cd4/go/src/infra/tools/kitchen/monitoring_test.go
[modify] https://crrev.com/c4560bb1d0ba03c48bee086b6363da9495292cd4/go/src/infra/tools/kitchen/testdata/recipe_repo/recipes.py
[modify] https://crrev.com/c4560bb1d0ba03c48bee086b6363da9495292cd4/go/src/infra/tools/kitchen/cook.go
[modify] https://crrev.com/c4560bb1d0ba03c48bee086b6363da9495292cd4/go/src/infra/tools/kitchen/cook_mode.go
[modify] https://crrev.com/c4560bb1d0ba03c48bee086b6363da9495292cd4/go/src/infra/tools/kitchen/monitoring.go
[modify] https://crrev.com/c4560bb1d0ba03c48bee086b6363da9495292cd4/go/src/infra/tools/kitchen/cook_logdog.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/2e8c982c8ae4ed5840ebd3d39453de62c64daa96

commit 2e8c982c8ae4ed5840ebd3d39453de62c64daa96
Author: Petr Hosek <phosek@google.com>
Date: Tue Oct 17 07:12:45 2017

devshell: LUCI Devshell server implementation

This can be used with any application that supports the Devshell
protocol to supply authentication token obtained from LUCI_CONTEXT.

Bug:  756229 
Change-Id: I399aca4c2b9ecc2696fd38ece0c485a953714869
Reviewed-on: https://chromium-review.googlesource.com/670985
Commit-Queue: Petr Hosek <phosek@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[add] https://crrev.com/2e8c982c8ae4ed5840ebd3d39453de62c64daa96/client/cmd/devshell/main.go
[add] https://crrev.com/2e8c982c8ae4ed5840ebd3d39453de62c64daa96/common/devshell/server.go
[add] https://crrev.com/2e8c982c8ae4ed5840ebd3d39453de62c64daa96/common/devshell/server_test.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/9b21472876fc187e574ae812773692a43597cb97

commit 9b21472876fc187e574ae812773692a43597cb97
Author: Petr Hosek <phosek@google.com>
Date: Tue Oct 31 23:02:59 2017

kitchen: Hook up DevShell support

When requested through kitchen properties, Kitchen will start
a local DevShell server instance which can be used by tool that
supports the DevShell protocol (e.g. gsutil or gcloud) to obtain
authentication credentials from LUCI_CONTEXT.

Bug:  756229 
Change-Id: I8a6be6b26503d405017966096befae3cdf72989c
Reviewed-on: https://chromium-review.googlesource.com/724399
Commit-Queue: Petr Hosek <phosek@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/9b21472876fc187e574ae812773692a43597cb97/go/src/infra/tools/kitchen/cook.go
[modify] https://crrev.com/9b21472876fc187e574ae812773692a43597cb97/go/src/infra/tools/kitchen/auth.go
[modify] https://crrev.com/9b21472876fc187e574ae812773692a43597cb97/go/src/infra/tools/kitchen/cook_mode.go
[modify] https://crrev.com/9b21472876fc187e574ae812773692a43597cb97/go/src/infra/tools/kitchen/cook_test.go
[modify] https://crrev.com/9b21472876fc187e574ae812773692a43597cb97/go/src/infra/tools/kitchen/testdata/recipe_repo/recipes.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c44213d85a5a2c7af2e15cf63df3094ea7d15aae

commit c44213d85a5a2c7af2e15cf63df3094ea7d15aae
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Nov 02 04:17:10 2017

kitchen: Mint one token with cloud-platform scope instead of 3 different tokens.

Before we were using 3 flavors of tokens:
 * PubSub flavored for LogDog.
 * BigQuery flavored for events export to BigQuery.
 * Google Storage flavored for Devshell proxy (we didn't actually do this, due
   to a bug).

It means we did at least 3 round trips to grab each individual token. Now we use
single cloud-platform token that is good for all 3 cases.

Note that git and cipd etc. still do round trips to grab their corresponding
tokens. This CL applies only to kitchen guts.

R=phosek@chromium.org, nodir@chromium.org
BUG= 756229 

Change-Id: I49ec1ed7c725bd2e7544b847b8acc4f5602231ac
Reviewed-on: https://chromium-review.googlesource.com/750326
Reviewed-by: Petr Hosek <phosek@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/c44213d85a5a2c7af2e15cf63df3094ea7d15aae/go/src/infra/tools/kitchen/auth.go
[modify] https://crrev.com/c44213d85a5a2c7af2e15cf63df3094ea7d15aae/go/src/infra/tools/kitchen/monitoring.go
[modify] https://crrev.com/c44213d85a5a2c7af2e15cf63df3094ea7d15aae/go/src/infra/tools/kitchen/cook_logdog.go

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/b1142f1fb86735b69d591227bb70c36dd34fc6e8

commit b1142f1fb86735b69d591227bb70c36dd34fc6e8
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Nov 02 19:13:23 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/55a53f173fe6ce663336fb3f20c460ce6c68bdca

commit 55a53f173fe6ce663336fb3f20c460ce6c68bdca
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Nov 03 01:17:18 2017

This has been deployed to prod.