It doesn't look like the remote server that has the caption is responding with Access-Control-Allow-Origin headers, which would be required for this to work.
Seems like this is working as intended.

hubbe@ Please label this as WontFix if you feel so.

hubbe@ Thank you for the response. 
Do you mean the video is not responding with Access-Control-Allow-Origin headers instead of the caption?
The video is the thing that does not load when we set crossorigin attribute.
Also, the video and the caption are from two different sources, so we need a way to load them both. Is there a workaround to make this work, or can you give us some kind of guidance on how we should tackle this?

Thank you!

Yes, you're right, the video does not seem to respond with access-control-alllow-origin headers. Additionally, since the video is accessed through a redirect, multiple origins are in play, which chrome translates to a "Origin: null" request header, which seems reasonable.

I think this will work with crossorigin="anonymous" if you make sure that the the video returns the cross-origin headers properly.

I'm going to close this bug now, feel free to re-open it if it still doesn't work with the right response headers.

Thanks hubbe and karandeepb!

While we can ask content hosts and publishers to add the correct headers to their video files, it will be difficult for some of them and there is a catch.

The catch is that access-control-alllow-origin headers are not required to load this media when no crossorigin attribute is added to the video tag. However the attribute is required to load VTT files. So our player adds the attribute to load the VTT and in doing so cripples the ability to load and play the media.

This seems like a problem with the spec or the behavior of the crossorigin attribute compared to the behavior without it.

Is this something we should take up with whatwg?