Simplified error case: https://luci-logdog.appspot.com/rpcexplorer/services/logdog.Logs/Get?request={%20%20%20%20%22path%22:%20%22bb/chromeos/cyan-paladin/609/+/recipes/steps/cbuildbot__cyan-paladin_/0/stdout%22,%20%20%20%20%22project%22:%20%22chromeos%22,%20%20%20%20%22index%22:%201}

This almost certainly has something to do with index offsets from archived builds.

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/fcfcaa45107e466fa6b0a07986fe87a5da5c1cf7

commit fcfcaa45107e466fa6b0a07986fe87a5da5c1cf7
Author: dnj <dnj@chromium.org>
Date: Tue Oct 18 02:12:07 2016

Fix sparse index handling, add index params.

Fix a bug in index archival where sparse indexes without a last log
entry will act as if they have no logs.

Augment the index protobuf with stream-wide parameters, namely the last
prefix and stream indexes in the stream and the total number of entries
in the stream. This will help supplement index information for sparse
indexes. This is also backwards-compatible with indexes without these
parameters.

Finally, add a tool to help query and interact with archived data
records. This was useful in diagnosing this bug, fixing this bug, and
is expected to be useful for future purposes.

BUG= chromium:654880 
TEST=local,production
  - Tested against production data. Tool without patch exhibits
    production bug. Tool with patch successfully loads the stream.

Review-Url: https://codereview.chromium.org/2422393002

[modify] https://crrev.com/fcfcaa45107e466fa6b0a07986fe87a5da5c1cf7/logdog/api/logpb/log.pb.go
[modify] https://crrev.com/fcfcaa45107e466fa6b0a07986fe87a5da5c1cf7/logdog/api/logpb/log.proto
[modify] https://crrev.com/fcfcaa45107e466fa6b0a07986fe87a5da5c1cf7/logdog/common/archive/archive_test.go
[modify] https://crrev.com/fcfcaa45107e466fa6b0a07986fe87a5da5c1cf7/logdog/common/archive/index.go
[add] https://crrev.com/fcfcaa45107e466fa6b0a07986fe87a5da5c1cf7/logdog/common/storage/archive/logdog_archive_test/main.go
[modify] https://crrev.com/fcfcaa45107e466fa6b0a07986fe87a5da5c1cf7/logdog/common/storage/archive/storage.go

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70

commit d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70
Author: dnj <dnj@chromium.org>
Date: Wed Oct 26 18:20:49 2016

LogDog: Fix archival Get/Tail implementations.

The sparse index tail algorithm assumed that indexes always had an entry
for the last log in the index. This is not necessarily true for spare
indexes. Conseqently, archive Tail could return a non-tail entry in that
situation.

This also fixes the Get and Tail to be more robust against malformed index
and stream data files, to the point where it will happily operate
without an index if needed now.

However, this is actually a good idea, so update index generation to
always include a final entry to the index. Tail optimizes for this case.

This introduced yet another situation where the storage layer was
unmarshalling LogEntry protobufs, then passing the binary data up to the
higher level to unmarshal again! To prevent this from introducing
performance regressions, we now have the storage layer kick up a lazy
unmarshal-at-most-once struct instead of raw data. This will let higher
levels take advantage of storage layers that have to unmarshal
protobufs.

Finally, log archival fetching had no supporting unit tests. Write some,
with very high coverage.

TBR=nodir@chromium.org
BUG= chromium:654880 
TEST=unit

Review-Url: https://codereview.chromium.org/2435883002

[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/common/iotools/countingreader.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/common/iotools/countingreader_test.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/appengine/coordinator/endpoints/logs/get.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/appengine/coordinator/endpoints/logs/get_test.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/client/butler/streamserver/handshake.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/common/archive/archive.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/common/archive/archive_test.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/common/archive/index.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/common/storage/archive/logdog_archive_test/main.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/common/storage/archive/storage.go
[add] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/common/storage/archive/storage_test.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/common/storage/bigtable/storage.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/common/storage/bigtable/storage_test.go
[add] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/common/storage/entry.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/common/storage/memory/memory.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/common/storage/memory/memory_test.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/common/storage/storage.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/server/archivist/archivist.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/server/archivist/storageSource.go
[modify] https://crrev.com/d65c2b2a5ebadb3ebfa0383630bbc1fa2b755a70/logdog/server/collector/utils_test.go

This looks to be fixed now!