New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 807899 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Feb 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

cros tryjob failing: 503 from cr-buildbucket.appspot.com

Reported by vpalatin@chromium.org, Feb 1 2018

Issue description

When trying to launch a tryjob, e.g. cros tryjob meowth-release-tryjob
it's failing (most of the time, 100%?) with the following error message:
Submitting tryjob...
08:27:15: ERROR: cros tryjob failed before completing.
08:27:15: ERROR: Got a 503 response from buildbucket with url: https://cr-buildbucket.appspot.com/_ah/api/buildbucket/v1/builds

followed with a long JSON stuff (see below) containing HTML containing some messaging about over-quota.

content: {
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "backendError",
    "message": "\n\u003chtml\u003e\u003chead\u003e\n\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003ctitle\u003e503 Over Quota\u003c/title\u003e\n\u003cstyle\u003e\u003c!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--\u003e\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody text=#000000 bgcolor=#ffffff\u003e\n\u003ctable border=0 cellpadding=2 cellspacing=0 width=100%\u003e\n\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cfont face=arial,sans-serif color=#ffffff\u003e\u003cb\u003eError\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cH1\u003eOver Quota\u003c/H1\u003e\nThis application is temporarily over its serving quota.  Please try again later.\n\n\u003cp\u003e\n\u003c/blockquote\u003e\n\u003ctable width=100% cellpadding=0 cellspacing=0\u003e\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cimg alt=\"\" width=1 height=4\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/body\u003e\u003c/html\u003e\n",
    "debugInfo": "Lily SPI request failed: status=503 URL='https://13708-4184003-dot-cr-buildbucket.appspot.com/_ah/spi/BuildBucketApi.put' error='\n\u003chtml\u003e\u003chead\u003e\n\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003ctitle\u003e503 Over Quota\u003c/title\u003e\n\u003cstyle\u003e\u003c!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--\u003e\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody text=#000000 bgcolor=#ffffff\u003e\n\u003ctable border=0 cellpadding=2 cellspacing=0 width=100%\u003e\n\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cfont face=arial,sans-serif color=#ffffff\u003e\u003cb\u003eError\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cH1\u003eOver Quota\u003c/H1\u003e\nThis application is temporarily over its serving quota.  Please try again later.\n\n\u003cp\u003e\n\u003c/blockquote\u003e\n\u003ctable width=100% cellpadding=0 cellspacing=0\u003e\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cimg alt=\"\" width=1 height=4\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/body\u003e\u003c/html\u003e\n'\ncom.google.api.server.core.Fault: ImmutableErrorDefinition{base=BACKEND_ERROR, category=BACKEND_ERROR, cause=null, debugInfo=Lily SPI request failed: status=503 URL='https://13708-4184003-dot-cr-buildbucket.appspot.com/_ah/spi/BuildBucketApi.put' error='\n\u003chtml\u003e\u003chead\u003e\n\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003ctitle\u003e503 Over Quota\u003c/title\u003e\n\u003cstyle\u003e\u003c!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--\u003e\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody text=#000000 bgcolor=#ffffff\u003e\n\u003ctable border=0 cellpadding=2 cellspacing=0 width=100%\u003e\n\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cfont face=arial,sans-serif color=#ffffff\u003e\u003cb\u003eError\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cH1\u003eOver Quota\u003c/H1\u003e\nThis application is temporarily over its serving quota.  Please try again later.\n\n\u003cp\u003e\n\u003c/blockquote\u003e\n\u003ctable width=100% cellpadding=0 cellspacing=0\u003e\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cimg alt=\"\" width=1 height=4\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/body\u003e\u003c/html\u003e\n', domain=global, extendedHelp=null, httpHeaders={}, httpStatus=serviceUnavailable, internalReason=Reason{arguments={}, cause=null, code=lilySpiBackendError, createdByBackend=false, debugMessage=null, errorProtoCode=null, errorProtoDomain=null, filteredMessage=null, location=null, message=null, unnamedArguments=[]}, location=null, message=\n\u003chtml\u003e\u003chead\u003e\n\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003ctitle\u003e503 Over Quota\u003c/title\u003e\n\u003cstyle\u003e\u003c!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--\u003e\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody text=#000000 bgcolor=#ffffff\u003e\n\u003ctable border=0 cellpadding=2 cellspacing=0 width=100%\u003e\n\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cfont face=arial,sans-serif color=#ffffff\u003e\u003cb\u003eError\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cH1\u003eOver Quota\u003c/H1\u003e\nThis application is temporarily over its serving quota.  Please try again later.\n\n\u003cp\u003e\n\u003c/blockquote\u003e\n\u003ctable width=100% cellpadding=0 cellspacing=0\u003e\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cimg alt=\"\" width=1 height=4\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/body\u003e\u003c/html\u003e\n, reason=backendError, rpcCode=-32099} \n\u003chtml\u003e\u003chead\u003e\n\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003ctitle\u003e503 Over Quota\u003c/title\u003e\n\u003cstyle\u003e\u003c!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--\u003e\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody text=#000000 bgcolor=#ffffff\u003e\n\u003ctable border=0 cellpadding=2 cellspacing=0 width=100%\u003e\n\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cfont face=arial,sans-serif color=#ffffff\u003e\u003cb\u003eError\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cH1\u003eOver Quota\u003c/H1\u003e\nThis application is temporarily over its serving quota.  Please try again later.\n\n\u003cp\u003e\n\u003c/blockquote\u003e\n\u003ctable width=100% cellpadding=0 cellspacing=0\u003e\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cimg alt=\"\" width=1 height=4\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/body\u003e\u003c/html\u003e\n: Lily SPI request failed: status=503 URL='https://13708-4184003-dot-cr-buildbucket.appspot.com/_ah/spi/BuildBucketApi.put' error='\n\u003chtml\u003e\u003chead\u003e\n\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003ctitle\u003e503 Over Quota\u003c/title\u003e\n\u003cstyle\u003e\u003c!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--\u003e\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody text=#000000 bgcolor=#ffffff\u003e\n\u003ctable border=0 cellpadding=2 cellspacing=0 width=100%\u003e\n\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cfont face=arial,sans-serif color=#ffffff\u003e\u003cb\u003eError\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cH1\u003eOver Quota\u003c/H1\u003e\nThis application is temporarily over its serving quota.  Please try again later.\n\n\u003cp\u003e\n\u003c/blockquote\u003e\n\u003ctable width=100% cellpadding=0 cellspacing=0\u003e\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cimg alt=\"\" width=1 height=4\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/body\u003e\u003c/html\u003e\n'\n\tat com.google.api.server.rest.adapter.lily.LilyErrorConverter.createFault(LilyErrorConverter.java:117)\n\tat com.google.api.server.rest.adapter.lily.LilyHandler$Transformer.apply(LilyHandler.java:211)\n\tat com.google.api.server.rest.adapter.lily.LilyHandler$Transformer.apply(LilyHandler.java:175)\n\tat com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:241)\n\tat com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:231)\n\tat com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:122)\n\tat com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:401)\n\tat com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1019)\n\tat com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:675)\n\tat com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:107)\n\tat com.google.common.util.concurrent.AbstractTransformFuture.create(AbstractTransformFuture.java:51)\n\tat com.google.common.util.concurrent.Futures.transform(Futures.java:761)\n\tat com.google.api.server.rest.adapter.lily.LilyHandler$LilyCallHandler$1.callInContext(LilyHandler.java:404)\n\tat com.google.api.server.rest.adapter.lily.LilyHandler$LilyCallHandler$1.callInContext(LilyHandler.java:370)\n\tat com.google.tracing.TraceContext$TraceContextCallable$1.call(TraceContext.java:793)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.callInInheritedContextNoUnref(TraceContext.java:339)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.callInInheritedContext(TraceContext.java:328)\n\tat com.google.tracing.TraceContext$TraceContextCallable.call(TraceContext.java:790)\n\tat com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:130)\n\tat com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)\n\tat com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:83)\n\tat com.google.api.server.thread.ThreadTrackers$ThreadTrackingRunnable.run(ThreadTrackers.java:126)\n\tat com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)\n\tat com.google.api.server.server.CommonModule$ContextCarryingExecutorService$1.runInContext(CommonModule.java:846)\n\tat com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)\n\tat com.google.tracing.CurrentContext.runInContext(CurrentContext.java:351)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:320)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:312)\n\tat com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\n"
   }
  ],
  "code": 503,
  "message": "\n\u003chtml\u003e\u003chead\u003e\n\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003ctitle\u003e503 Over Quota\u003c/title\u003e\n\u003cstyle\u003e\u003c!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--\u003e\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody text=#000000 bgcolor=#ffffff\u003e\n\u003ctable border=0 cellpadding=2 cellspacing=0 width=100%\u003e\n\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cfont face=arial,sans-serif color=#ffffff\u003e\u003cb\u003eError\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cH1\u003eOver Quota\u003c/H1\u003e\nThis application is temporarily over its serving quota.  Please try again later.\n\n\u003cp\u003e\n\u003c/blockquote\u003e\n\u003ctable width=100% cellpadding=0 cellspacing=0\u003e\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cimg alt=\"\" width=1 height=4\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/body\u003e\u003c/html\u003e\n"
 }
}

 
Labels: -Pri-0 Pri-1
Seems the quota reset at midnight PST ...
downgrading to P1

Comment 2 by jkop@chromium.org, Feb 1 2018

Cc: dgarr...@chromium.org
+dgarrett

Could this be fixed by using swarming tryjobs?
Looking at nothing but that error message, I'm guessing that that was a general buildbucket outage since fixed, or else some kind of network proxy failure.

The current trooper should be able to confirm/deny that.


Interestingly, looking at logs on the PreCQ launcher (and if I converted timezones correctly), these are the two closest cros tryjob commands bracket the error above, and they both worked (along with every other command it ran).


01:19:17: INFO: RunCommand: cros tryjob --yes --timeout 14400 celes-no-vmtest-pre-cq fizz-no-vmtest-pre-cq reef-no-vmtest-pre-cq chell-no-vmtest-pre-cq gru-no-vmtest-pre-cq -g 897068 -g 897067 -g 897069 in /b/c/cbuild/repository
01:19:20: INFO: cbuildbot output: Verifying patches...
Submitting tryjob...
Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:celes-no-vmtest-pre-cq] [buildbucket_id:8955804005638291520].
Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:fizz-no-vmtest-pre-cq] [buildbucket_id:8955804005235508992].
Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:reef-no-vmtest-pre-cq] [buildbucket_id:8955804004920562112].
Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:chell-no-vmtest-pre-cq] [buildbucket_id:8955804004467926848].
Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:gru-no-vmtest-pre-cq] [buildbucket_id:8955804004148457056].
Tryjob submitted!

<snip>

2:06:13: INFO: Launching Pre-CQs for configs ['reef-pre-cq', 'amd64-generic-pre-cq', 'arm-generic-pre-cq', 'gru-pre-cq'] with changes CL:859176 CL:859177 CL:859178 CL:859179 CL:859180 CL:859181 CL:859182 CL:859183 CL:859184 CL:859185 CL:859186 CL:859187 CL:859188 CL:859189 CL:859190 CL:859191 CL:859192 CL:859193 CL:859194 CL:859195 CL:859196 CL:859197 CL:870018 CL:870019 CL:870020 CL:870021 CL:870022 CL:870023 CL:870024 CL:870025 CL:870026 CL:870027 CL:870190 CL:870191 CL:870790 CL:891039 CL:891040 CL:891041 CL:896904
02:06:13: INFO: RunCommand: cros tryjob --yes --timeout 14400 reef-pre-cq amd64-generic-pre-cq arm-generic-pre-cq gru-pre-cq -g 870021 -g 859186 -g 891039 -g 859190 -g 859180 -g 859185 -g 859195 -g 870020 -g 870190 -g 870019 -g 859181 -g 870790 -g 870025 -g 859184 -g 870022 -g 870026 -g 859187 -g 859193 -g 859197 -g 870023 -g 859196 -g 859176 -g 891041 -g 891040 -g 859188 -g 870191 -g 859179 -g 859192 -g 859178 -g 859182 -g 859194 -g 870024 -g 870018 -g 859183 -g 896904 -g 859177 -g 870027 -g 859191 -g 859189 in /b/c/cbuild/repository
02:06:16: INFO: cbuildbot output: Verifying patches...
Submitting tryjob...
Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:reef-pre-cq] [buildbucket_id:8955801052508892320].
Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:amd64-generic-pre-cq] [buildbucket_id:8955801052125751216].
Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:arm-generic-pre-cq] [buildbucket_id:8955801051705750048].
Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:gru-pre-cq] [buildbucket_id:8955801051359908688].
Tryjob submitted!

Owner: jkop@chromium.org
Note: that error message contains a Java backtrace.

But the "cros tryjob" command is in Python, and buildbuckets are in Go, so I'm not sure what's happening.

I would follow up with the current trooper.

Comment 5 by jkop@chromium.org, Feb 1 2018

Cc: katthomas@chromium.org
+Trooper
Status: Fixed (was: Untriaged)
This is likely due to omg/8040. Let's reopen if we see it again. 

Sign in to add a comment