Matt, I've got a fix in progress for this, but it might be better to fix this in the Overlay Panel instead of in Contextual Search.

The problem seems be be that a recent change to handle interaction with the Ephemeral Tab introduced a delay as a workaround to not having an effective way to check if the TTS panel is currently shown.
This is because when the panel is first requested to show it is not yet visible, and that's the measure of it being shown or not.

CL https://chromium-review.googlesource.com/c/chromium/src/+/1414171

Matt, Jinsuk gave a drive-by review encouraging me to look at pushing the change down to Overlay Panel and that looks like it will work pretty well.  I will send it out for your review soon.