Right now, worker context visibility (as set on ContextCacheController) is controlled by both LTHI visibility as well as memory policy changes. We should try to remove the memory policy control, as this makes the logic harder to reason about.