So realistically we have a few problems:

- LogDog needs to report failures, else we can't see when things fail.
- LogDog currently never gives up on sending logs, since it is the sole owner of log data and giving up means losing data permanently.


Transient errors typically resolve themselves eventually, so this isn't usually a problem.

BuildBot has three ways of terminating builds:
1) Maximum time without output. Since LogDog needs to report status, this is not going to work, since LogDog will be causing output when it fails like this.
2) Maximum per-step execution time.
3) Maximum per-build execution time.

On Swarming, and on *most* builders, (2) and (3) kick in and stop this sort of thing from happening. LogDog supersedes (2) b/c it runs even after steps finish, so the simple and standard solution here would be to configure maximum timeouts to the overall build.

Internally, I would caution strongly against having LogDog actually give up sending logs, since that causes irrecoverable data loss. It might be worth having LogDog's logger detect duplicate messages, but that's a fair amount of work and there is an art to doing that right. I think the right solution here is to just have the builder enforce an upper limit.

This is mostly an issue with logdog on buildbot.  While the root cause is unknown, this is known not to happen on LUCI, so the solution is to migrate to luci.