I have good news and bad news

Bad news: We probably lost the annotation stream.
Good news: We have the build information in buildbucket
You can access the build via the not-yet-completed buildbucket page:
https://ci.chromium.org/p/chromium/builders/try/linux_angle_rel_ng/4689

It looks like this isn't as widespread as "We lost everything", since I can still load a lot of the builds, just a lot of the other builds aren't loadable.  This graph confirms we lost builds, but not a catastrophic number of them:
https://screenshot.googleplex.com/WGaBJpSPEoq

Obviously we don't want to lose any builds, but plugging up this pipeline is an ongoing project.

Thanks. Those logs were accessible sometime in the past, because they were linked from old bugs. So they were lost recently?

I mostly care about the build logs, so that buildbucket page is fine for me. Looks like you just deleted "luci.chromium" from my first URL, but that doesn't work for the second one. How can I fix up my second URL?

Unfortunately the second one pre-dated buildbucket storing full information about a build.

Root cause found, it was due to tumble backlog, which causes
Tumble started to have a backlog starting early december due to GAE running out of quota (used up $5000/day), and reducing tumble shard count to mitigate that.
This drops the tumble processing rate below the threshold required.

Tumble queue size: https://screenshot.googleplex.com/DqiOLvYCZRQ
Tumble processing rate: https://screenshot.googleplex.com/jxPRUNNmKzh

Tumble shard count was increased, which caused LogDog to run out of quota again last night.  This isn't really a sustainable strategy.

This class of failure should go away with the work tracked in 923557

Thanks for investigating!

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/e098879524490dc32608b810d577bc825a47feec

commit e098879524490dc32608b810d577bc825a47feec
Author: Ryan Tseng <hinoka@google.com>
Date: Fri Jan 18 22:54:10 2019

[tumble] Hardcode the luci.chromium namespace to always use 64 shards

Bug: 920852
Change-Id: I18a80f99ce841b0b94ebab10019922999d1a19b6
Reviewed-on: https://chromium-review.googlesource.com/c/1423401
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/e098879524490dc32608b810d577bc825a47feec/tumble/config.go
[modify] https://crrev.com/e098879524490dc32608b810d577bc825a47feec/tumble/fire_tasks_test.go
[modify] https://crrev.com/e098879524490dc32608b810d577bc825a47feec/tumble/model_mutation.go
[modify] https://crrev.com/e098879524490dc32608b810d577bc825a47feec/tumble/process.go
[modify] https://crrev.com/e098879524490dc32608b810d577bc825a47feec/tumble/service.go