[elm] M71 Elm Paygen causing bots to die |
||||
Issue descriptionAttempted a number of payload rebuilds for M71 Elm Stable; failing on an internal process error: 08:17:26: INFO: OAuth token TTL expired, auto-refreshing (attempt 1/2) https://logs.chromium.org/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8927334844383177072/+/steps/PaygenBuildStable/0/stdout Not sure if this is a red herring since it's a purlple internal error, but need eyes regardless. Thanks
,
Dec 12
The bot died, I don't see any reason why though. Can you kick the build off again please? OAuth TTL expiring is a normal part of the life-cycle, it's irreverent.
,
Dec 12
We've run it 3 times no; no joy. I can do it again.
,
Dec 12
Let me look into the other failures as well then.
,
Dec 12
Kevin was this still a non-pass for you guys (yesterday at 20:56)? https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8927377100704938912
,
Dec 12
Oddly enough the bot died in both of these runs: Run: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8927334844383177072 Task: https://chrome-swarming.appspot.com/task?id=41bbc4220462b510&refresh=10 Run: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8927380697030539088 Task: https://chrome-swarming.appspot.com/task?id=41b928e35e608910&refresh=10 Which makes me thing 'died' might mean restarted mid-task.
,
Dec 13
Hi Alec, anyone else we should add to this bug, etc. since the board won't be included in the first stable? Otherwise it would be lower priority, most likely. Thanks
,
Dec 13
Hey Kevin. What is this builder for? I'm unsure what's causing the builder deaths, I'm pretty sure it's something to do with the build itself though.
,
Dec 13
These builds generate payloads. If the payload phase fails in the main builder, we have the ability to regenerate them after the fact using this build type. Without payloads, we have nothing to push to users, so this is a critical portion of the process.
,
Dec 13
But the 5th try passed (and there was much rejoicing). We should still figure out why we had so many failed attempts to keep this from happening, however.
,
Dec 13
I suspect it was crrev.com/c/1375531 being chumped in. That was breaking master release as well.
,
Dec 13
Removing the RBS since the last attempt passed.
,
Dec 13
I'm going to mark this fixed for now, watching a few other paygen related bugs at the moment. Looks like it's in a bad way all around. We can revisit this if it fails again once things stabilize. |
||||
►
Sign in to add a comment |
||||
Comment 1 by kbleicher@google.com
, Dec 12