Flake and non-flake try jobs often work on revisions that are not in the same periods of time (flake analysis happens on older revisions)

Before swarmbucket we had separate builders for flake and non-flake and this would help with two things:
 - Prevent that tree-closure analysis (compile failure) is not starved off resources by long-running flake analysis
 - Keep non-flake builders with relatively fresh caches and checkouts compared to flake builders.

In swarmbucket, since our approach is to use a single builder with multiple flexible slaves we need to find a way to guarantee the same advantages as we had with the separate builders.

There is a one-pager with some design considerations here:
https://docs.google.com/document/d/1bwcSf08Q8MT_4t0W5zouTjD_lDFbsJeHx0yxpcdur98/edit#heading=h.7hjufwbmerml