Increase gitiles RPC deadline from 5 sec to something higher |
|||
Issue descriptionWe are getting alerts due to high rate of HTTP 500s. Most of them seem to be coming from: Error fetching state of the world: Get https://dart.googlesource.com/a/sdk/+refs%2Fheads?format=JSON: API error 5 (urlfetch: DEADLINE_EXCEEDED) :: {"InvID":9108383873276414992, "JobID":"dart/dart-gitiles-trigger-master"} Failed to LaunchTask :: {"error":"Get https://dart.googlesource.com/a/sdk/+refs%2Fheads?format=JSON: API error 5 (urlfetch: DEADLINE_EXCEEDED)", "InvID":9108383873276414992, "JobID":"dart/dart-gitiles-trigger-master"} (with different repos)
,
Jun 25 2018
5 sec is default RPC timeout. I see it by duration of handlers before they timeout.
,
Jun 25 2018
> 5 sec is default RPC timeout. I poked code and I still don't know what/where sets it, nor how to change it.
,
Jun 25 2018
Vadim answered in IM: https://github.com/golang/appengine/blob/master/urlfetch/urlfetch.go#L46 https://github.com/golang/appengine/blob/master/urlfetch/urlfetch.go#L141
,
Jun 27 2018
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/3a7f9b2b025a82d1e25191f5448af7924bde8aed commit 3a7f9b2b025a82d1e25191f5448af7924bde8aed Author: Vadim Shtayura <vadimsh@chromium.org> Date: Wed Jun 27 02:28:53 2018 [scheduler] Improve RPC deadline handling. Since https://chromium.googlesource.com/infra/luci/luci-go/+/9883c80 it is no longer necessary to setup the deadline when creating the client. In fact, it seems it no longer works (since gitiles should have been using 1 min deadline if it worked). Instead setup the deadline right before calling RPCs. R=tandrii@chromium.org BUG= 855823 Change-Id: I048ecc1305fe6ebf6eaed73de2365b9ab032d316 Reviewed-on: https://chromium-review.googlesource.com/1116061 Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Commit-Queue: Vadim Shtayura <vadimsh@chromium.org> [modify] https://crrev.com/3a7f9b2b025a82d1e25191f5448af7924bde8aed/scheduler/appengine/engine/controller.go [modify] https://crrev.com/3a7f9b2b025a82d1e25191f5448af7924bde8aed/scheduler/appengine/task/buildbucket/buildbucket.go [modify] https://crrev.com/3a7f9b2b025a82d1e25191f5448af7924bde8aed/scheduler/appengine/task/gitiles/gitiles.go [modify] https://crrev.com/3a7f9b2b025a82d1e25191f5448af7924bde8aed/scheduler/appengine/task/task.go [modify] https://crrev.com/3a7f9b2b025a82d1e25191f5448af7924bde8aed/scheduler/appengine/task/utils/tasktest/controller.go
,
Jun 29 2018
Deployed. |
|||
►
Sign in to add a comment |
|||
Comment 1 by tandrii@chromium.org
, Jun 25 2018