You're not the first person to ask, so I should definitely write this up. There is a CLI tool called "logdog" that can do this.

Long-term, the "logdog" tool will be distributed as part of "depot_tools". Short-term, you need to manually install it. Pick a directory to install into - let's call it LOGDOG_CIPD_ROOT.

mkdir $LOGDOG_CIPD_ROOT
cd $LOGDOG_CIPD_ROOT
/path/to/depot_tools/cipd init
/path/to/depot_tools/cipd install infra/tools/luci/logdog/logdog/linux-amd64

At this point, you will have the "logdog" binary available, and can run it:

./logdog cat <path>

<path> can be one of two things:
1) A <project>/<path> tuple, like the one you would see the viewer: https://screenshot.googleplex.com/wfmB4a05Dip

e.g.:
./logdog cat chromium/bb/chromium.fyi/CrWin7Goma_dbg_/32153/+/recipes/steps/taskkill/0/stdout

2) A LogDog URL, like the one you would see in a build property: https://screenshot.googleplex.com/pQL9CJGRLvD

./logdog cat logdog://luci-logdog.appspot.com/chromium/bb/chromium.fyi/CrWin7Goma_dbg_/32153/+/recipes/annotations

Both of those should work.

Oh, if you're running on not 64-bit Linux, you can pick a different supported platform in the CIPD path. See the full list here: https://chrome-infra-packages.appspot.com/#/?path=infra/tools/luci/logdog/logdog

Is there a reason we can't expose a service that just serves the log in response to a simple HTTP request?

No technical reason, but such a service doesn't exist, and the CLI tool fills that role.

(Renaming bug) This has been a fairly frequent request.  The tool fills that role atm but the API could be much simpler as a simple HTTP GET endpoint.

Well, the tool fills another important role: auth. The "logdog" CLI tool takes care of obtaining and utilizing authentication to view internal waterfalls. Developing a special service just to serve public waterfalls b/c users want to type "curl" instead of "logdog cat" seems like a waste (IMO). This is why it hasn't been a priority: "curl" is inferior to the "logdog" tool for this purpose.

We can allow GET requests to attach oauth2 tokens too.  This isn't just for curl, it's also for making requests from my_random_python_script.py to logdog easier.  IE instead of depending on a Go tool, it can just depend on the oauth2client library.

For a majority of logs, is there even a reason to require oauth2 tokens?

Depends what team you're on.  If you're in perf, chromeos, or android then the majority of your logs will require an oauth2 token.

Currently, OAuth2 tokens are only required for logs that require authentication to view. If the log does not require authentication, it can currently be viewed without an OAuth2 token.

copying hinoka's message from https://bugs.chromium.org/p/chromium/issues/detail?id=698968#c9

Alright, you can try running this instead:

cit logdog cat logdog://luci-logdog.appspot.com/chromium/bb/chromium.fyi/CrWin7Goma_dbg_/32153/+/recipes/annotations

It'll spend a couple minutes doing basically a full gclient sync, but after that should be fast.

I'm converting some of my scripts to use logdog cat now, and just learned it doesn't do HTTP unescaping, e.g.

logdog cat /work/cipd/logdog cat chromium/bb/chromium.chrome/Google%20Chrome%20Win/16456/+/recipes/steps/sizes/0/stdout

results in

[E2017-03-22T14:28:53.224119-07:00 2778 0 subcommandCat.go:119] Invalid command-line stream path.           {"error":"Illegal charater (%) at index 25.", "index":0, "path":"bb/chromium.chrome/Google%20Chrome%20Win/16456/+/recipes/steps/sizes/0/stdout", "project":"chromium"}


I realize it expects a valid path, but maybe it would be easier for folks copying URLs into it if it could do the unescaping itself.

Oh actually, it doesn't like me unescaping %20 with ' ', it wants an underscore there instead. Interesting.

RE #13, that's correct - the "cat" tool doesn't regard its input as a URL. However, since "%" is not a valid LogDog path character, it should be safe to unquote the command-line argument. That's a really good idea. Can you please file a feature request bug for it?

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/ffe5df245684d2c4c13dc60cf9f3a1df44f65eeb

commit ffe5df245684d2c4c13dc60cf9f3a1df44f65eeb
Author: Ryan Tseng <hinoka@google.com>
Date: Thu Apr 20 23:22:18 2017

Bump luci/gae

To pick up https://codereview.chromium.org/2829903002

BUG= 698429 
TBR=iannucci

Change-Id: Iaf5ef500761e4441fee5192dfb36d2550f02f892
Reviewed-on: https://chromium-review.googlesource.com/483703
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/ffe5df245684d2c4c13dc60cf9f3a1df44f65eeb/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/d7caba160073c8b83e1b7d393227162f910d46ee

commit d7caba160073c8b83e1b7d393227162f910d46ee
Author: hinoka <hinoka@google.com>
Date: Mon May 01 23:14:38 2017

Milo flex raw log viewer endpoint

This is a simple flex app that runs the logdog fetcher and serves back log text.  It runs in flex to avoid the GAE classic restrictions such as response size and response time.

Most of the logic is setting up a flex environment, most notably, memcache is not yet available.  The idea is to refactor this to look more like the GAE Classic middleware once a public memcache API is released.  Until then, memcache is simply not used.

BUG= 698429 

Review-Url: https://codereview.chromium.org/2796743004

[modify] https://crrev.com/d7caba160073c8b83e1b7d393227162f910d46ee/milo/Makefile
[modify] https://crrev.com/d7caba160073c8b83e1b7d393227162f910d46ee/milo/appengine/common/middleware.go
[modify] https://crrev.com/d7caba160073c8b83e1b7d393227162f910d46ee/milo/appengine/frontend/dispatch.yaml
[add] https://crrev.com/d7caba160073c8b83e1b7d393227162f910d46ee/milo/appengine/logs/README.md
[add] https://crrev.com/d7caba160073c8b83e1b7d393227162f910d46ee/milo/appengine/logs/logs.go
[add] https://crrev.com/d7caba160073c8b83e1b7d393227162f910d46ee/milo/appengine/logs/main.go
[add] https://crrev.com/d7caba160073c8b83e1b7d393227162f910d46ee/milo/appengine/logs/module-logs.yaml

 Issue 722218  has been merged into this issue.

 Issue 724249  has been merged into this issue.

When using a python script to retrieve logs with logdog.exe I often want it to keep going until there are no more logs. However the "logdog cat" command doesn't seem to have an option to return immediately when there is no data. Instead it repeatedly prints this:

[I2017-11-16T11:27:37.895249-08:00 99932 0 fetcher.go:366] No logs returned. Sleeping...               {"delay":"5s", "index":4266}

Is there a recommended way of doing this? An option to the cat command to tell it to return immediately would work. Alternately, if logdog.exe would respond to Ctrl+C then I could abort it.

> c#20
i've filed a separate bug 786142

Is there any update on this?

The major development here is that I moved the LogDog client backend to AppEngine Flex environment Q3 of last year, removing the need to host a separate service just for this. The Flex environment app has all of the facilities and connections needed to do this, and (unlike other GAE environments) does not have an imposed content deadline or buffering restriction.

That was most of the work - it shouldn't be too hard to just add an endpoint and use LogDog's streaming structures to have it stream a raw text request.

Yep dnj already did all the hard work, I'm just tying it all together now.  It is in a (relatively lightweight) design stage.

I've circulated a doc on how it should behave:
https://docs.google.com/document/d/1nydZKUFQH8arLMIYjuciEYdulrrjnl26meYm91twLeQ/edit

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/gae/+/8511a0dd8cd1e24b351ba3458b207d077043fa6c

commit 8511a0dd8cd1e24b351ba3458b207d077043fa6c
Author: Ryan Tseng <hinoka@google.com>
Date: Tue May 01 21:54:37 2018

[gae] Implement http.Flusher for capturingResponseWriter

The Flusher interface needs to be exposed for higher level caller to
flush correctly.

Bug:  698429 
Change-Id: I2e5f81650329936e068f4e9f3027377da2f890d2
Reviewed-on: https://chromium-review.googlesource.com/1038435
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/8511a0dd8cd1e24b351ba3458b207d077043fa6c/impl/cloud/request.go

 Issue 695703  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/5857162208bfe988ac6247a1dbf95ab2ec176f31

commit 5857162208bfe988ac6247a1dbf95ab2ec176f31
Author: Ryan Tseng <hinoka@google.com>
Date: Wed May 16 00:20:59 2018

[logdog] Plaintext http log endpoint

Adds a /logs/... endpoint to fetch logs directly in GAE flex.
This implementation uses storage.Storage directly.

BUG:  698429 
Change-Id: Ie377144eb7e6987e1170f37645049a92e0f0a3e8
Reviewed-on: https://chromium-review.googlesource.com/1038883
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/appengine/gaemiddleware/flex/env.go
[modify] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/grpc/grpcutil/errors.go
[modify] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/grpc/prpc/encoding.go
[modify] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/logdog/appengine/cmd/coordinator/default/dispatch.yaml
[modify] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/logdog/appengine/cmd/coordinator/logs/main.go
[modify] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/logdog/appengine/cmd/coordinator/logs/module-logs.yaml
[modify] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/logdog/appengine/coordinator/context.go
[modify] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/logdog/appengine/coordinator/flex/logs/get.go
[add] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/logdog/appengine/coordinator/flex/logs/http.go
[modify] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/logdog/appengine/coordinator/flex/logs/util.go
[modify] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/logdog/appengine/coordinator/logStream.go
[modify] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/logdog/common/storage/archive/storage.go
[modify] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/logdog/common/types/sizes.go
[modify] https://crrev.com/5857162208bfe988ac6247a1dbf95ab2ec176f31/server/tsmon/responsewriter.go

Cool!

hinoka: Is this ready so we can give it a try?

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/3a5e7d20be2391cc3fc8ad527349f4b933d40440

commit 3a5e7d20be2391cc3fc8ad527349f4b933d40440
Author: Ryan Tseng <hinoka@google.com>
Date: Wed May 16 19:00:03 2018

[logdog] Fix off-by-one typo in plaintext endpoint

Bug:  698429 
Change-Id: I9d7569fbefd7af8459b1ddf182e23a40d5df7d0a
Reviewed-on: https://chromium-review.googlesource.com/1060618
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/3a5e7d20be2391cc3fc8ad527349f4b933d40440/logdog/appengine/coordinator/flex/logs/http.go

It's not quite ready since it's missing some features and polish (backlink to build, auth flow, etc), but if you'd like to beta-test it, just copy the string at the top: https://screenshot.googleplex.com/S0nRqCT9poY

And construct the URL at /logs/<path>,  ie:
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8946347424693195344/+/steps/bot_update/0/stdout

> And construct the URL at /logs/<path>,  ie: https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8946347424693195344/+/steps/bot_update/0/stdout

Wow, that loaded much faster than the usual LogDog version. This is very exciting :-)

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/8cc0d4ce4fb9c7ef9fafbf25cb4f4f371de95f84

commit 8cc0d4ce4fb9c7ef9fafbf25cb4f4f371de95f84
Author: Ryan Tseng <hinoka@google.com>
Date: Tue Jun 26 21:58:53 2018

[logdog] Redirect unauthenticated request to plain-text to login page

Bug:  698429 
Change-Id: I2d5bc4c5fe20771f520666c13ddc32dacaa86f8c
Reviewed-on: https://chromium-review.googlesource.com/1114252
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/8cc0d4ce4fb9c7ef9fafbf25cb4f4f371de95f84/logdog/appengine/coordinator/flex/logs/http.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/4e71a66a9bbca54d56914c19b3f1197ff02bc23e

commit 4e71a66a9bbca54d56914c19b3f1197ff02bc23e
Author: Ryan Tseng <hinoka@google.com>
Date: Tue Jun 26 23:30:13 2018

Add link to plain-text viewer

Bug:  698429 
Change-Id: If81b1dff9b5ee57c99d7d8e749f4b065d45bf709
Reviewed-on: https://chromium-review.googlesource.com/1115916
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/4e71a66a9bbca54d56914c19b3f1197ff02bc23e/web/apps/logdog-view/elements/logdog-view/logdog-view.html

Not yet feature complete, but there's a dogfood link in the log viewer now.

Nice! I just tried it on a build and it loaded much faster than through the regular logdog interface. Maybe the plaintext interface should be the default? :-)

It's still missing a few features, but I'm working on it :)

 Issue 705015  has been merged into this issue.

Okay I think we have enough features to switch plaintext to the default view, with the TS viewer as a fallback link.

Missing features are:
P0 - Snap to bottom
P1 - Merged Multiple streams
P1 - Feedback button
P1 - ?tail
P2 - Datagram streams
P2 - Timestamps
P2 - Cache control headers

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/eaa1f1aade19f25c4f1484a2bf1cb18d51be60a9

commit eaa1f1aade19f25c4f1484a2bf1cb18d51be60a9
Author: Ryan Tseng <hinoka@google.com>
Date: Tue Jul 10 21:32:40 2018

[logdog] Add old viewer link to plaintext viewer

Bug:  698429 
Change-Id: I26f6d3701c4773d9d00051396d624e13466e60db
Reviewed-on: https://chromium-review.googlesource.com/1132079
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/eaa1f1aade19f25c4f1484a2bf1cb18d51be60a9/logdog/appengine/coordinator/flex/logs/http.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/e282bdb337fa2753a3f28bceaaed1057dd329b5e

commit e282bdb337fa2753a3f28bceaaed1057dd329b5e
Author: Ryan Tseng <hinoka@google.com>
Date: Tue Jul 10 22:52:39 2018

[logdog] Add feedback link to plaintext viewer

Bug:  698429 
Change-Id: I92e1dcd789d518f64969e2e1d6ce66e862d78d86
Reviewed-on: https://chromium-review.googlesource.com/1132433
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/e282bdb337fa2753a3f28bceaaed1057dd329b5e/logdog/appengine/coordinator/flex/logs/http.go

Re comment 40: FWIW I dislike the current snap to bottom -- I want to look at the top bits often enough where the snapping gets in the way, and when I do want to look at the bottom then hitting cmd-down isn't so hard. Also, no other web page has snap to bottom, making the logdog pages feel pretty un-webby to me. So I wouldn't mind if that wasn't implemented.

I agree with comment 43, FWIW.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/082513b02eaf7d1d5058fd6ba4f4d6baa81df085

commit 082513b02eaf7d1d5058fd6ba4f4d6baa81df085
Author: Ryan Tseng <hinoka@google.com>
Date: Wed Jul 11 22:24:21 2018

[milo] Use logdog's plain-text endpoint for single stream log links

Also removes the query view in the test.
It's not actively used and this will reduce the API surface later.

Bug:  698429 
Change-Id: Id7fd1d6bf8a565b390a6a79f5e1ccfd7a2ae87ea
Reviewed-on: https://chromium-review.googlesource.com/1134354
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/082513b02eaf7d1d5058fd6ba4f4d6baa81df085/logdog/client/butlerlib/bootstrap/bootstrap_test.go
[modify] https://crrev.com/082513b02eaf7d1d5058fd6ba4f4d6baa81df085/logdog/common/viewer/url.go
[modify] https://crrev.com/082513b02eaf7d1d5058fd6ba4f4d6baa81df085/logdog/common/viewer/url_test.go
[modify] https://crrev.com/082513b02eaf7d1d5058fd6ba4f4d6baa81df085/milo/buildsource/swarming/expectations/build-running-logdog.json
[modify] https://crrev.com/082513b02eaf7d1d5058fd6ba4f4d6baa81df085/milo/frontend/expectations/swarming.build-build-running-logdog.html

Fixed now.

Features can be tracked in other bugs.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/63806c9c90e04f774f4988ddf1e34508c5dab81f

commit 63806c9c90e04f774f4988ddf1e34508c5dab81f
Author: Ryan Tseng <hinoka@google.com>
Date: Mon Jul 16 17:15:41 2018

[logdog] Escape html lines in plaintext viewer

Bug:  698429 
Change-Id: Iccef804a92bacaaaccc41a8580d0c6a42a96bc4d
Reviewed-on: https://chromium-review.googlesource.com/1137361
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>

[modify] https://crrev.com/63806c9c90e04f774f4988ddf1e34508c5dab81f/logdog/appengine/coordinator/flex/logs/http.go