At around 5pm pst today, martiniss noticed that some chromium.perf builders were not scheduling builds. About fifteen minutes later, a bunch of builds were scheduled at once. 

There were builders with pending queues, and bots available but not utilized.

For the builds that are running, they look as though they've all been scheduled in a batch. This is what I see right now for Linux Builder Currently Building:

85902 ETA: 17:34:37 [3 mins, 7 secs] [Running for 20 mins, 52 secs] compile
85903 ETA: 17:34:34 [3 mins, 3 secs] [Running for 20 mins, 52 secs] compile
85904 ETA: 17:34:38 [3 mins, 8 secs] [Running for 20 mins, 52 secs] compile
85905 ETA: 17:34:34 [3 mins, 3 secs] [Running for 20 mins, 52 secs] compile
85906 ETA: 17:31:50 [19 secs] [Running for 20 mins, 52 secs] compile
85907 ETA: 17:31:50 [19 secs] [Running for 20 mins, 52 secs] compile
85908 ETA: 17:31:46 [15 secs] [Running for 20 mins, 52 secs] compile
85910 ETA: 17:31:44 [13 secs] [Running for 20 mins, 52 secs] compile
85909 ETA: 17:31:45 [14 secs] [Running for 20 mins, 52 secs] compile
85911 ETA: 17:31:45 [14 secs] [Running for 20 mins, 52 secs] compile
85912 ETA: 17:31:44 [13 secs] [Running for 20 mins, 52 secs] compile
85913 ETA: 17:31:44 [13 secs] [Running for 20 mins, 52 secs] compile
85914 ETA: 17:31:44 [13 secs] [Running for 20 mins, 52 secs] compile
85915 ETA: 17:31:44 [14 secs] [Running for 20 mins, 52 secs] compile
85916 ETA: 17:31:42 [11 secs] [Running for 11 mins, 16 secs] compile 

It looks like calls to get master.json are taking a long time. This might be a contributing factor. Looking at the viceroy graphs (https://viceroy.corp.google.com/chrome_infra/Machines/masters?duration=30d), nothing looks all that out of the ordinary. We need to do more investigation.