Thanks for the report.

To be clear, you're referring to x-ref annotations specifically? Or are you seeing elevated latency for other types of annotation? (e.g. blame).

It does appear that x-ref latency has gotten worse recently at high percentiles. I see it spiked up around the middle of May. It seems telling that you're experiencing bad latency consistently, despite the latency being reasonable at lower percentiles. This suggests to me that there's something pathological about either:
1) The files you're requesting
2) Your location

For the first point, do you get equally bad latency across all files? Can you give me an example file with consistently bad latency?

For the second point, do others in your office experience this too? It would be strange for this to be the cause considering your office location and where all the services we hit are located. However, looking at our monitoring I can see some significant network latency for x-refs, so I think this is the most likely explanation. I'll look into it further.

It looks like the latency gets much worse during PT business hours - about 06:45-19:00 PT. If it's not too inconvenient, can you try testing outside that time range? Obviously this isn't a long-term solution (or even a good workaround), but it would help narrow down the issue.

Yep, this is about x-ref annotations. It didn't used to be this way (e.g., a month or two ago). It used to be consistently fast. Now, even when it's good, I notice the delay between the file loading and the x-refs showing up.

Yesterday I was noticing it during PT business hours. It seems fine now (5am PT). 

But when it's bad it's consistently bad, for small files and big. Though small are more likely to succeed it seems.

I'm out until next week but I'll update this bug with the times that I notice it being bad.
 

We've made some improvements here, latency should be significantly lower as of a few hours ago. Is it okay for you now? Please give it a try and let me know if it's still a problem.

Sorry, I forgot to respond. Yes, it seems y'all fixed the problem. Thanks!