It's becoming a bit awkward when tile manager wants to schedule new tasks that it has to keep around a ref on the old tasks until we actually complete the schedule. It's very subtle and I don't think it should be a requirement that we hold on to a ref on a task that might only logically exist within the worker pool.