Treat context deadline as a transient error |
|||
Issue descriptionNoticed some luci-scheduler job stuck in "Starting" state. Debugging led to /internal/tasks/invocations task queue call: """ Job transaction failed: Call error 11: Deadline exceeded (timeout) HTTP 202: Error when executing the action - Call error 11: Deadline exceeded (timeout) """ The deadline error is treated as fatal here, and task queue task is declared finished (by returning HTTP 202). It should be treated as transient and the task should complete with HTTP 500 code (so Task Queue Service can retry it).
,
Sep 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/af2196e28a0b79b66902a31e6b9f75caf525eb25 commit af2196e28a0b79b66902a31e6b9f75caf525eb25 Author: Vadim Shtayura <vadimsh@chromium.org> Date: Tue Sep 12 00:20:56 2017 scheduler: Extract the transaction helper function. Make it retry on subset of transient errors. Consistently mark all unexpected datastore errors (usually deadlines) as transient, to make sure their status correctly propagates to the task queue level. R=tandrii@chromium.org BUG= 764043 , 691399 Change-Id: I31c289f12eb91cf932022bc85b9c7f7368309fe6 Reviewed-on: https://chromium-review.googlesource.com/660987 Commit-Queue: Vadim Shtayura <vadimsh@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> [modify] https://crrev.com/af2196e28a0b79b66902a31e6b9f75caf525eb25/scheduler/appengine/engine/controller.go [modify] https://crrev.com/af2196e28a0b79b66902a31e6b9f75caf525eb25/scheduler/appengine/engine/engine.go [modify] https://crrev.com/af2196e28a0b79b66902a31e6b9f75caf525eb25/scheduler/appengine/engine/invocation.go [modify] https://crrev.com/af2196e28a0b79b66902a31e6b9f75caf525eb25/scheduler/appengine/engine/utils.go [add] https://crrev.com/af2196e28a0b79b66902a31e6b9f75caf525eb25/scheduler/appengine/engine/utils_test.go
,
Sep 21 2017
Isn't this fixed now?
,
Sep 21 2017
Yes |
|||
►
Sign in to add a comment |
|||
Comment 1 by estaab@chromium.org
, Jun 22 2017Status: Available (was: Untriaged)