503 Server Errors for api/swarming/v1/tasks/new |
||
Issue descriptionA lots of bots on Chromium waterfall ran into this error. One example: https://uberchromegw.corp.google.com/i/chromium.gpu/builders/Linux%20Release%20%28NVIDIA%29/builds/78832 9294 2016-05-25 20:32:11.486 E: Unable to open given url, https://chromium-swarm.appspot.com/_ah/api/swarming/v1/tasks/new, after 30 attempts. 503 Server Error: Service Unavailable for url: https://chromium-swarm.appspot.com/_ah/api/swarming/v1/tasks/new ---------- Alternate-protocol: 443:quic X-xss-protection: 1; mode=block X-content-type-options: nosniff Content-encoding: gzip Transfer-encoding: chunked Expires: Wed, 25 May 2016 20:32:11 GMT Server: GSE Cache-control: private, max-age=0 Date: Wed, 25 May 2016 20:32:11 GMT X-frame-options: SAMEORIGIN Alt-svc: quic=":443"; ma=2592000; v="34,33,32,31,30,29,28,27,26,25" Content-type: application/json; charset=UTF-8 { "error": { "errors": [ { "domain": "global", "reason": "backendError", "message": "Internal Server Error" } ], "code": 503, "message": "Internal Server Error" } }
,
May 25 2016
https://codereview.chromium.org/2006263005 has been put live. tasks.new is now working. The problem was OverQuotaError generated by the search API. Since this API is not strictly needed, commented out its use. Failures started at around 16:18 EDT and was resolved at 16:49. This was not a complete outage, a subset of the tasks were still succeeding.
,
May 25 2016
https://screenshot.googleplex.com/SkF0A2ZrnK9 Rocking at 220 QPS.
,
May 25 2016
Peaked at 10 new tasks/second but it has now slowed down at 2 tasks/sec.
,
May 25 2016
I confirmed that this error is not showing up anymore on the waterfall.
,
May 26 2016
|
||
►
Sign in to add a comment |
||
Comment 1 by st...@chromium.org
, May 25 2016