I can confirm that all of image decode time is spent in upscaling the image (there is no actual decode happening). This is due to the fact that we get different subrects that need to be scaled to different sizes. Other than that, the caching is behaving as expected.

I don't know what we can do here, short of pretty agressive heuristics to try and reuse a different subrect scale in a smaller thing. That won't help if the subrect grows however, since the data would be missing in a smaller subrect scale.

I'm going to WontFix this, but please reopen if you feel strongly that this should be investigated in more detail.