Catapult Autoroller is broken |
||
Issue descriptionThe Catapult autoroller is in a weird state: * It says that the most recent roll https://chromium-review.googlesource.com/c/chromium/src/+/756964 failed, but it actually landed. * The full history shows this CL https://chromium-review.googlesource.com/c/chromium/src/+/757817 with an additional Catapult commit, but this roll is not displayed in the list on catapult-roll.skia.org * There's more than a day worth of Catapult commits to pick up after f83c79, which is the revision of the most recent roll, but the roller is sitting idle. * Clicking the stop button seems to hang and not actually stop the roller.
,
Nov 9 2017
Looks like "git cl upload" was the last thing to run, on 11/7/2017 at 6:59 pm EST. That matches the CL from #2. There are no errors in the logs, so it's not clear what happened here. Based on the CL (https://chromium-review.googlesource.com/c/chromium/src/+/757817), the CQ bit was never set, which indicates to me that "git cl upload" did not finish. If that's the case, the fact that a sub-process was allowed to hang for two days is a big problem. In the meantime, I restarted the server and it seems to be working again.
,
Nov 9 2017
I'm re-enabling the roller now, to see how it behaves.
,
Nov 9 2017
Okay, I'm still not sure exactly what happened here. We didn't get alerts because a previous change of mine did not get pushed to prod (user error). In the future we'll be alerted when any roller does not complete a cycle for 10 minutes. I'm adding a timeout for "git cl upload" in case that was in fact the cause.
,
Nov 9 2017
The following revision refers to this bug: https://skia.googlesource.com/buildbot/+/03988495131986efc96da2b290db1d1604526699 commit 03988495131986efc96da2b290db1d1604526699 Author: Eric Boren <borenet@google.com> Date: Thu Nov 09 19:11:08 2017 [autoroll] Add timeout to "git cl upload" Bug: 783259 Change-Id: I932d05db952110c832f1f77acdfb5189e1dc1b88 Reviewed-on: https://skia-review.googlesource.com/69560 Reviewed-by: Ravi Mistry <rmistry@google.com> Commit-Queue: Eric Boren <borenet@google.com> [modify] https://crrev.com/03988495131986efc96da2b290db1d1604526699/autoroll/go/repo_manager/deps_repo_manager.go
,
Nov 9 2017
Woohoo, looks like the Catapult roller picked up the right roll. Thanks Eric for your extremely prompt attention to this issue!
,
Nov 10 2017
No problem! Looks like we're in good shape now. |
||
►
Sign in to add a comment |
||
Comment 1 by bore...@google.com
, Nov 9 2017