See Issue 781302 for background / context.

Slow queries began piling up on several shards, cascading into slow scheduler and widespread timeouts and failures.

It took longer than it should have to identify this problem. A slow-scheduler or slow-queries alert on shards would have identified it much sooner.