chromeos-skylab-bot-fleet failing to RefreshBots |
||||
Issue description
Failing with
line: [
0: {
logMessage: "original error: parsing time "" as "2006-01-02T15:04:05.999999999": cannot parse "" as "2006""
severity: "ERROR"
time: "2018-09-27T21:20:17.685117Z"
}
1: {
severity: "ERROR"
time: "2018-09-27T21:20:17.685120Z"
}
2: {
logMessage: "goroutine 297:"
severity: "ERROR"
time: "2018-09-27T21:20:17.685174Z"
}
3: {
logMessage: "#0 infra/appengine/crosskylabadmin/app/clients/swarming.go:264 - clients.TimeSinceBotTask()"
severity: "ERROR"
time: "2018-09-27T21:20:17.685177Z"
}
4: {
logMessage: " reason: swarming returned corrupted completed timestamp "
severity: "ERROR"
time: "2018-09-27T21:20:17.685179Z"
}
5: {
severity: "ERROR"
time: "2018-09-27T21:20:17.685180Z"
}
6: {
logMessage: "From frame 0 to 1, the following wrappers were found:"
severity: "ERROR"
time: "2018-09-27T21:20:17.685182Z"
}
7: {
logMessage: " internal reason: MultiError 1/1: following first non-nil error."
severity: "ERROR"
time: "2018-09-27T21:20:17.685184Z"
}
8: {
severity: "ERROR"
time: "2018-09-27T21:20:17.685185Z"
}
9: {
logMessage: "#1 infra/appengine/crosskylabadmin/app/frontend/tracker.go:217 - frontend.getIdleDuration()"
severity: "ERROR"
time: "2018-09-27T21:20:17.685187Z"
}
10: {
logMessage: " reason: failed to determine time since task 40358e0634070111"
severity: "ERROR"
time: "2018-09-27T21:20:17.685189Z"
}
...
Failed for task https://chrome-swarming.appspot.com/task?id=40358e0634070111
,
Sep 27
Looks like KILLED tasks have AbandonedTs set, not CompletedTs This similar to https://chromium-review.googlesource.com/c/infra/infra/+/1135773 (opposite interpretation though) Immediate way to unbreak the app is to run a task on that bot that isn't killed.
,
Sep 27
Rerunning an old admin task at: https://chrome-swarming.appspot.com/task?id=4035c128fc801e10&refresh=10
,
Sep 27
crosskylabadmin is no longer failing. Actual fix CL incoming.
,
Sep 27
,
Sep 28
No longer affecting prod, so off the deputy list.
,
Oct 1
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/929ef6f008b2727d10b0f6646901d4192c2339d6 commit 929ef6f008b2727d10b0f6646901d4192c2339d6 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Mon Oct 01 18:50:24 2018 crosskylbadmin: Fix handling of KILLED tasks BUG= chromium:890026 Change-Id: I98f106f62709192332c4b070fc41d9c1a6483229 Reviewed-on: https://chromium-review.googlesource.com/c/1249947 Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org> Auto-Submit: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> Cr-Commit-Position: refs/heads/master@{#17972} [modify] https://crrev.com/929ef6f008b2727d10b0f6646901d4192c2339d6/go/src/infra/appengine/crosskylabadmin/app/frontend/tasker_test.go [modify] https://crrev.com/929ef6f008b2727d10b0f6646901d4192c2339d6/go/src/infra/appengine/crosskylabadmin/app/clients/swarming.go
,
Oct 31
|
||||
►
Sign in to add a comment |
||||
Comment 1 by pprabhu@chromium.org
, Sep 27Labels: Hotlist-Skylab Hotlist-Deputy