Yeah Milo needs to propagate the 404.

milo should not return 404, the build exists. It is normal that logdog annotation stream is not available for some time.

"some time"? There is an ingest delay between the Butler emitting data and it getting loaded into LogDog. That delay is generally pretty short, but can be longer. If the stream doesn't exist in LogDog, there are two possibilities:
1) It will, but ingest delay. Maybe retry?
2) It will never exist, so retry is wasteful.

We could have Milo respond to 404s by retrying for some threshold (30s?) and then propagating the 404. Is there a better option here?

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/887de917e6e7d48a151293fd298d22519c86b6af

commit 887de917e6e7d48a151293fd298d22519c86b6af
Author: dnj <dnj@chromium.org>
Date: Fri Feb 24 03:38:40 2017

Milo: Handle missing / transient LogDog failures.

If a build is labelled as having a LogDog stream, Milo will
unconditionally try and load that stream. This falls apart if the build
has either not started yet, or if the LogDog annotation stream is not
available due to natural ingest delay.

This will prevent stream loading if the build hasn't started yet, and
will retry LogDog transient and not-found errors prior to failing
absolutely.

BUG= chromium:692245 ,  chromium:694913 
TEST=None
R=hinoka@chromium.org, nodir@chromium.org

Review-Url: https://codereview.chromium.org/2717623002

[modify] https://crrev.com/887de917e6e7d48a151293fd298d22519c86b6af/logdog/common/types/streamaddr.go
[add] https://crrev.com/887de917e6e7d48a151293fd298d22519c86b6af/milo/appengine/frontend/expectations/bootstrap-swarming.TestableBuild-build-pending-logdog.html
[add] https://crrev.com/887de917e6e7d48a151293fd298d22519c86b6af/milo/appengine/frontend/expectations/bootstrap-swarming.TestableBuild-build-running-logdog-no-annotation-stream.html
[add] https://crrev.com/887de917e6e7d48a151293fd298d22519c86b6af/milo/appengine/frontend/expectations/buildbot-swarming.TestableBuild-build-pending-logdog.html
[add] https://crrev.com/887de917e6e7d48a151293fd298d22519c86b6af/milo/appengine/frontend/expectations/buildbot-swarming.TestableBuild-build-running-logdog-no-annotation-stream.html
[modify] https://crrev.com/887de917e6e7d48a151293fd298d22519c86b6af/milo/appengine/logdog/build.go
[modify] https://crrev.com/887de917e6e7d48a151293fd298d22519c86b6af/milo/appengine/logdog/http.go
[modify] https://crrev.com/887de917e6e7d48a151293fd298d22519c86b6af/milo/appengine/swarming/build.go
[add] https://crrev.com/887de917e6e7d48a151293fd298d22519c86b6af/milo/appengine/swarming/expectations/build-pending-logdog.json
[add] https://crrev.com/887de917e6e7d48a151293fd298d22519c86b6af/milo/appengine/swarming/expectations/build-running-logdog-no-annotation-stream.json
[modify] https://crrev.com/887de917e6e7d48a151293fd298d22519c86b6af/milo/appengine/swarming/html_data.go
[add] https://crrev.com/887de917e6e7d48a151293fd298d22519c86b6af/milo/appengine/swarming/testdata/build-pending-logdog.swarm
[add] https://crrev.com/887de917e6e7d48a151293fd298d22519c86b6af/milo/appengine/swarming/testdata/build-running-logdog-no-annotation-stream.swarm

Did we actually roll out a new Milo with this patch?

I did for both luci-milo and luci-milo-dev

A W E S O M E!