cros tryjob failing: 503 from cr-buildbucket.appspot.com
Reported by
vpalatin@chromium.org,
Feb 1 2018
|
||||
Issue descriptionWhen trying to launch a tryjob, e.g. cros tryjob meowth-release-tryjob it's failing (most of the time, 100%?) with the following error message: Submitting tryjob... 08:27:15: ERROR: cros tryjob failed before completing. 08:27:15: ERROR: Got a 503 response from buildbucket with url: https://cr-buildbucket.appspot.com/_ah/api/buildbucket/v1/builds followed with a long JSON stuff (see below) containing HTML containing some messaging about over-quota. content: { "error": { "errors": [ { "domain": "global", "reason": "backendError", "message": "\n\u003chtml\u003e\u003chead\u003e\n\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003ctitle\u003e503 Over Quota\u003c/title\u003e\n\u003cstyle\u003e\u003c!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--\u003e\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody text=#000000 bgcolor=#ffffff\u003e\n\u003ctable border=0 cellpadding=2 cellspacing=0 width=100%\u003e\n\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cfont face=arial,sans-serif color=#ffffff\u003e\u003cb\u003eError\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cH1\u003eOver Quota\u003c/H1\u003e\nThis application is temporarily over its serving quota. Please try again later.\n\n\u003cp\u003e\n\u003c/blockquote\u003e\n\u003ctable width=100% cellpadding=0 cellspacing=0\u003e\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cimg alt=\"\" width=1 height=4\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/body\u003e\u003c/html\u003e\n", "debugInfo": "Lily SPI request failed: status=503 URL='https://13708-4184003-dot-cr-buildbucket.appspot.com/_ah/spi/BuildBucketApi.put' error='\n\u003chtml\u003e\u003chead\u003e\n\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003ctitle\u003e503 Over Quota\u003c/title\u003e\n\u003cstyle\u003e\u003c!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--\u003e\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody text=#000000 bgcolor=#ffffff\u003e\n\u003ctable border=0 cellpadding=2 cellspacing=0 width=100%\u003e\n\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cfont face=arial,sans-serif color=#ffffff\u003e\u003cb\u003eError\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cH1\u003eOver Quota\u003c/H1\u003e\nThis application is temporarily over its serving quota. Please try again later.\n\n\u003cp\u003e\n\u003c/blockquote\u003e\n\u003ctable width=100% cellpadding=0 cellspacing=0\u003e\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cimg alt=\"\" width=1 height=4\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/body\u003e\u003c/html\u003e\n'\ncom.google.api.server.core.Fault: ImmutableErrorDefinition{base=BACKEND_ERROR, category=BACKEND_ERROR, cause=null, debugInfo=Lily SPI request failed: status=503 URL='https://13708-4184003-dot-cr-buildbucket.appspot.com/_ah/spi/BuildBucketApi.put' error='\n\u003chtml\u003e\u003chead\u003e\n\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003ctitle\u003e503 Over Quota\u003c/title\u003e\n\u003cstyle\u003e\u003c!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--\u003e\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody text=#000000 bgcolor=#ffffff\u003e\n\u003ctable border=0 cellpadding=2 cellspacing=0 width=100%\u003e\n\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cfont face=arial,sans-serif color=#ffffff\u003e\u003cb\u003eError\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cH1\u003eOver Quota\u003c/H1\u003e\nThis application is temporarily over its serving quota. Please try again later.\n\n\u003cp\u003e\n\u003c/blockquote\u003e\n\u003ctable width=100% cellpadding=0 cellspacing=0\u003e\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cimg alt=\"\" width=1 height=4\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/body\u003e\u003c/html\u003e\n', domain=global, extendedHelp=null, httpHeaders={}, httpStatus=serviceUnavailable, internalReason=Reason{arguments={}, cause=null, code=lilySpiBackendError, createdByBackend=false, debugMessage=null, errorProtoCode=null, errorProtoDomain=null, filteredMessage=null, location=null, message=null, unnamedArguments=[]}, location=null, message=\n\u003chtml\u003e\u003chead\u003e\n\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003ctitle\u003e503 Over Quota\u003c/title\u003e\n\u003cstyle\u003e\u003c!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--\u003e\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody text=#000000 bgcolor=#ffffff\u003e\n\u003ctable border=0 cellpadding=2 cellspacing=0 width=100%\u003e\n\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cfont face=arial,sans-serif color=#ffffff\u003e\u003cb\u003eError\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cH1\u003eOver Quota\u003c/H1\u003e\nThis application is temporarily over its serving quota. Please try again later.\n\n\u003cp\u003e\n\u003c/blockquote\u003e\n\u003ctable width=100% cellpadding=0 cellspacing=0\u003e\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cimg alt=\"\" width=1 height=4\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/body\u003e\u003c/html\u003e\n, reason=backendError, rpcCode=-32099} \n\u003chtml\u003e\u003chead\u003e\n\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003ctitle\u003e503 Over Quota\u003c/title\u003e\n\u003cstyle\u003e\u003c!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--\u003e\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody text=#000000 bgcolor=#ffffff\u003e\n\u003ctable border=0 cellpadding=2 cellspacing=0 width=100%\u003e\n\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cfont face=arial,sans-serif color=#ffffff\u003e\u003cb\u003eError\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cH1\u003eOver Quota\u003c/H1\u003e\nThis application is temporarily over its serving quota. Please try again later.\n\n\u003cp\u003e\n\u003c/blockquote\u003e\n\u003ctable width=100% cellpadding=0 cellspacing=0\u003e\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cimg alt=\"\" width=1 height=4\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/body\u003e\u003c/html\u003e\n: Lily SPI request failed: status=503 URL='https://13708-4184003-dot-cr-buildbucket.appspot.com/_ah/spi/BuildBucketApi.put' error='\n\u003chtml\u003e\u003chead\u003e\n\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003ctitle\u003e503 Over Quota\u003c/title\u003e\n\u003cstyle\u003e\u003c!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--\u003e\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody text=#000000 bgcolor=#ffffff\u003e\n\u003ctable border=0 cellpadding=2 cellspacing=0 width=100%\u003e\n\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cfont face=arial,sans-serif color=#ffffff\u003e\u003cb\u003eError\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cH1\u003eOver Quota\u003c/H1\u003e\nThis application is temporarily over its serving quota. Please try again later.\n\n\u003cp\u003e\n\u003c/blockquote\u003e\n\u003ctable width=100% cellpadding=0 cellspacing=0\u003e\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cimg alt=\"\" width=1 height=4\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/body\u003e\u003c/html\u003e\n'\n\tat com.google.api.server.rest.adapter.lily.LilyErrorConverter.createFault(LilyErrorConverter.java:117)\n\tat com.google.api.server.rest.adapter.lily.LilyHandler$Transformer.apply(LilyHandler.java:211)\n\tat com.google.api.server.rest.adapter.lily.LilyHandler$Transformer.apply(LilyHandler.java:175)\n\tat com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:241)\n\tat com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:231)\n\tat com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:122)\n\tat com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:401)\n\tat com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1019)\n\tat com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:675)\n\tat com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:107)\n\tat com.google.common.util.concurrent.AbstractTransformFuture.create(AbstractTransformFuture.java:51)\n\tat com.google.common.util.concurrent.Futures.transform(Futures.java:761)\n\tat com.google.api.server.rest.adapter.lily.LilyHandler$LilyCallHandler$1.callInContext(LilyHandler.java:404)\n\tat com.google.api.server.rest.adapter.lily.LilyHandler$LilyCallHandler$1.callInContext(LilyHandler.java:370)\n\tat com.google.tracing.TraceContext$TraceContextCallable$1.call(TraceContext.java:793)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.callInInheritedContextNoUnref(TraceContext.java:339)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.callInInheritedContext(TraceContext.java:328)\n\tat com.google.tracing.TraceContext$TraceContextCallable.call(TraceContext.java:790)\n\tat com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:130)\n\tat com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)\n\tat com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:83)\n\tat com.google.api.server.thread.ThreadTrackers$ThreadTrackingRunnable.run(ThreadTrackers.java:126)\n\tat com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454)\n\tat com.google.api.server.server.CommonModule$ContextCarryingExecutorService$1.runInContext(CommonModule.java:846)\n\tat com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461)\n\tat com.google.tracing.CurrentContext.runInContext(CurrentContext.java:351)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:320)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:312)\n\tat com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\n" } ], "code": 503, "message": "\n\u003chtml\u003e\u003chead\u003e\n\u003cmeta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\"\u003e\n\u003ctitle\u003e503 Over Quota\u003c/title\u003e\n\u003cstyle\u003e\u003c!--\nbody {font-family: arial,sans-serif}\ndiv.nav {margin-top: 1ex}\ndiv.nav A {font-size: 10pt; font-family: arial,sans-serif}\nspan.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}\ndiv.nav A,span.big {font-size: 12pt; color: #0000cc}\ndiv.nav A {font-size: 10pt; color: black}\nA.l:link {color: #6f6f6f}\nA.u:link {color: green}\n//--\u003e\u003c/style\u003e\n\n\u003c/head\u003e\n\u003cbody text=#000000 bgcolor=#ffffff\u003e\n\u003ctable border=0 cellpadding=2 cellspacing=0 width=100%\u003e\n\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cfont face=arial,sans-serif color=#ffffff\u003e\u003cb\u003eError\u003c/b\u003e\u003c/td\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cH1\u003eOver Quota\u003c/H1\u003e\nThis application is temporarily over its serving quota. Please try again later.\n\n\u003cp\u003e\n\u003c/blockquote\u003e\n\u003ctable width=100% cellpadding=0 cellspacing=0\u003e\u003ctr\u003e\u003ctd bgcolor=#3366cc\u003e\u003cimg alt=\"\" width=1 height=4\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/body\u003e\u003c/html\u003e\n" } }
,
Feb 1 2018
+dgarrett Could this be fixed by using swarming tryjobs?
,
Feb 1 2018
Looking at nothing but that error message, I'm guessing that that was a general buildbucket outage since fixed, or else some kind of network proxy failure. The current trooper should be able to confirm/deny that. Interestingly, looking at logs on the PreCQ launcher (and if I converted timezones correctly), these are the two closest cros tryjob commands bracket the error above, and they both worked (along with every other command it ran). 01:19:17: INFO: RunCommand: cros tryjob --yes --timeout 14400 celes-no-vmtest-pre-cq fizz-no-vmtest-pre-cq reef-no-vmtest-pre-cq chell-no-vmtest-pre-cq gru-no-vmtest-pre-cq -g 897068 -g 897067 -g 897069 in /b/c/cbuild/repository 01:19:20: INFO: cbuildbot output: Verifying patches... Submitting tryjob... Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:celes-no-vmtest-pre-cq] [buildbucket_id:8955804005638291520]. Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:fizz-no-vmtest-pre-cq] [buildbucket_id:8955804005235508992]. Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:reef-no-vmtest-pre-cq] [buildbucket_id:8955804004920562112]. Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:chell-no-vmtest-pre-cq] [buildbucket_id:8955804004467926848]. Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:gru-no-vmtest-pre-cq] [buildbucket_id:8955804004148457056]. Tryjob submitted! <snip> 2:06:13: INFO: Launching Pre-CQs for configs ['reef-pre-cq', 'amd64-generic-pre-cq', 'arm-generic-pre-cq', 'gru-pre-cq'] with changes CL:859176 CL:859177 CL:859178 CL:859179 CL:859180 CL:859181 CL:859182 CL:859183 CL:859184 CL:859185 CL:859186 CL:859187 CL:859188 CL:859189 CL:859190 CL:859191 CL:859192 CL:859193 CL:859194 CL:859195 CL:859196 CL:859197 CL:870018 CL:870019 CL:870020 CL:870021 CL:870022 CL:870023 CL:870024 CL:870025 CL:870026 CL:870027 CL:870190 CL:870191 CL:870790 CL:891039 CL:891040 CL:891041 CL:896904 02:06:13: INFO: RunCommand: cros tryjob --yes --timeout 14400 reef-pre-cq amd64-generic-pre-cq arm-generic-pre-cq gru-pre-cq -g 870021 -g 859186 -g 891039 -g 859190 -g 859180 -g 859185 -g 859195 -g 870020 -g 870190 -g 870019 -g 859181 -g 870790 -g 870025 -g 859184 -g 870022 -g 870026 -g 859187 -g 859193 -g 859197 -g 870023 -g 859196 -g 859176 -g 891041 -g 891040 -g 859188 -g 870191 -g 859179 -g 859192 -g 859178 -g 859182 -g 859194 -g 870024 -g 870018 -g 859183 -g 896904 -g 859177 -g 870027 -g 859191 -g 859189 in /b/c/cbuild/repository 02:06:16: INFO: cbuildbot output: Verifying patches... Submitting tryjob... Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:reef-pre-cq] [buildbucket_id:8955801052508892320]. Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:amd64-generic-pre-cq] [buildbucket_id:8955801052125751216]. Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:arm-generic-pre-cq] [buildbucket_id:8955801051705750048]. Successfully sent PUT request to [buildbucket_bucket:master.chromiumos.tryserver] with [config:gru-pre-cq] [buildbucket_id:8955801051359908688]. Tryjob submitted!
,
Feb 1 2018
Note: that error message contains a Java backtrace. But the "cros tryjob" command is in Python, and buildbuckets are in Go, so I'm not sure what's happening. I would follow up with the current trooper.
,
Feb 1 2018
+Trooper
,
Feb 1 2018
|
||||
►
Sign in to add a comment |
||||
Comment 1 by vpalatin@chromium.org
, Feb 1 2018