New issue
Advanced search Search tips

Issue 614811 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: May 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

503 Server Errors for api/swarming/v1/tasks/new

Project Member Reported by st...@chromium.org, May 25 2016

Issue description

A lots of bots on Chromium waterfall ran into this error.
One example: https://uberchromegw.corp.google.com/i/chromium.gpu/builders/Linux%20Release%20%28NVIDIA%29/builds/78832

9294 2016-05-25 20:32:11.486 E: Unable to open given url, https://chromium-swarm.appspot.com/_ah/api/swarming/v1/tasks/new, after 30 attempts.
503 Server Error: Service Unavailable for url: https://chromium-swarm.appspot.com/_ah/api/swarming/v1/tasks/new
----------
Alternate-protocol: 443:quic
X-xss-protection: 1; mode=block
X-content-type-options: nosniff
Content-encoding: gzip
Transfer-encoding: chunked
Expires: Wed, 25 May 2016 20:32:11 GMT
Server: GSE
Cache-control: private, max-age=0
Date: Wed, 25 May 2016 20:32:11 GMT
X-frame-options: SAMEORIGIN
Alt-svc: quic=":443"; ma=2592000; v="34,33,32,31,30,29,28,27,26,25"
Content-type: application/json; charset=UTF-8

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "backendError",
    "message": "Internal Server Error"
   }
  ],
  "code": 503,
  "message": "Internal Server Error"
 }
}
 

Comment 1 by st...@chromium.org, May 25 2016

Cc: hajimehoshi@chromium.org

Comment 2 by mar...@chromium.org, May 25 2016

Labels: -Pri-0 Pri-1
https://codereview.chromium.org/2006263005 has been put live. tasks.new is now working.
The problem was OverQuotaError generated by the search API. Since this API is not strictly needed, commented out its use.

Failures started at around 16:18 EDT and was resolved at 16:49. This was not a complete outage, a subset of the tasks were still succeeding.

Comment 3 by mar...@chromium.org, May 25 2016

https://screenshot.googleplex.com/SkF0A2ZrnK9

Rocking at 220 QPS.

Comment 4 by mar...@chromium.org, May 25 2016

Peaked at 10 new tasks/second but it has now slowed down at 2 tasks/sec.

Comment 5 by st...@chromium.org, May 25 2016

I confirmed that this error is not showing up anymore on the waterfall.

Comment 6 by mar...@chromium.org, May 26 2016

Status: Fixed (was: Assigned)
Fixed in https://codereview.chromium.org/2012843003/

Sign in to add a comment