Dirk, are you aware of any changes in browser test that might cause this?

Also on Windows, i.e. my job here is pending for > 30 minutes https://chromium-swarm.appspot.com/task?id=34f6f49e4f5eeb10&refresh=10

I think browser tests takes more time => task duration increased => swarming capacity is not enough => all swarming tasks using the same pool of machines is affected.

Do we have any evidence that browser_tests suddenly started taking more time?

rate of HTTP 500s on swarming elevated http://shortn/_iOGvBQjBWv
(we should have an alert on this)

... => # of pending tasks is too high => task query takes too much time => http request timeout of 1min is hit => http 500

Run time of each swarming test on linux_chromium_rel_ng: http://shortn/_HuFQkyFGUu

something happened at about 14:20 PDT

http://shortn/_oupxZGZ4T9 is only browser tests

there were no swarming deployments today http://shortn/_QxzpD1fNiV

I don't see a problem with infrastructure, so, as a trooper, I am not sure what to do here besides escalation. jam, dpranke?

It seems the outage has been resolved: http://shortn/_anRPIjnlww

Test times are back down and current tryjobs don't seem to be purple.

I believe rcabebfee33715afb9e577f6d133898b408390ab1 might have been the breaking change. It's since been reverted. (Though the timing doesn't really line up, so it could have been something else.)

This was reverted 4 hours ago but Windows and Mac try runs are still timing out on swarming jobs.

The test runtimes of both mac_chromium_rel_ng and win_chromium_rel_ng returned to normal at about the same time as linux_chromium_rel_ng:
mac_chromium_rel_ng: http://shortn/_qFyySetpb1
win_chromium_rel_ng: http://shortn/_rfaDsNPXwp

Any current expiring jobs could just be blowback from that short period of time when nearly everything was expiring. Is there a particular trybot that's still affected?

Looks like this outage was resolved source-side.

I'm going to reopen this a bit to do some follow-up, but we can take this out of the trooper queue.

It sounds like this actually went reasonably well in terms of handling a src-induced overload, and so I think it might be interesting to look at it for places to improve.

nodir@ - how did you notice this? Did we get step failures alerts, or did someone ping you, or something else?

Regarding the comments in #5-#6, maruel@, are the queries taking too long expected behavior? Should we be able to shed load better so that queries complete in a timely manner?

Did we get alerts on pending times for tasks being too long? If not, do we need to adjust thresholds there? 

Do we know what the problem with the offending CL was? Was it hanging the task?

Do we know how quickly we ran out of capacity, or what the performance characteristics are like to know how quickly we should've recovered and how well we did?

Oh, also, were we planning to write a postmortem for this?

I received pages for high infra failure rate in some builders, see tickets in o/369081. Then I've received an alert for high 500x on Swarming (which was caused by long pending queue, caused by the same thing)

There were no alerts from metrics mentioned in #7. I am sure such alerts would be useful because those metrics are precomputations with a very large window (2h). I think stip@ set these metrics up, but bpastene should know better.

I didn't dig into Chromium side of this outage, e.g. what caused the timeout, was it slowdown or hang, etc. I think I don't have enough domain-specific knowledge.

> Do we know how quickly we ran out of capacity, or what the performance characteristics are like to know how quickly we should've recovered and how well we did?
here is a graph of pending tasks http://shortn/_YUk933miHf
I think "active jobs" is close to "number of bots".
at ~1pm pending queue started to grow
at ~2pm it was so long, tasks started to expire
i interpret the graph is that at ~6pm service got back to normal, but maruel@ or sergeyberezin@ could tell better

---

I think it would be useful to set up few more swarming alerts:
- # of pending tasks, i.e. http://shortn/_feQCpt0Mk8
- rate of task expiration

i didn't plan to write a postmortem

From my perspective the outcome of this postmortem is:
- alerts on InfraFailures worked
- we need more alerts on swarming to detect the problem even earlier. # of pending tasks would catch it early
- it would be great if Swarming had a button "EMERGENCY add more bots" (like we have "emergency add quota" in AppEngine), but it is possible only for Linux (thanks to MachineProvider), but not Macs/Windows I think.

The long pending queue leading to sluggish performance is being worked on. That's my main project at the moment.