paladin builders not starting |
|||||||||||
Issue descriptionmaster-paladin has encountered infra failures: nyan-paladin did not start parrot-paladin did not start glados-paladin did not start daisy-paladin did not start falco-paladin did not start leon-paladin did not start oak-paladin did not start veyron_pinky-paladin did not start samus-paladin did not start lumpy-incremental-paladin did not start auron-paladin did not start nyan_freon-paladin did not start peach_pit-paladin did not start See https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/10450 It looks like builds are pending, but buildslaves were offline, eg: https://uberchromegw.corp.google.com/i/chromeos/builders/nyan-paladin (Actually, now one of the nyan-paladin buildslaves seems to have come back...)
,
Mar 11 2016
Many buildslaves are still offline: https://uberchromegw.corp.google.com/i/chromeos/buildslaves Are there any known infra outages occurring now?
,
Mar 11 2016
+ dshi, secondary, nothing obvious broken in the lab.
,
Mar 11 2016
,
Mar 11 2016
,
Mar 11 2016
Could this be related to the file system corruption that happened on the pineview-release builder ( crbug.com/593942 )?
,
Mar 11 2016
No known infra outages right now, except goma which does not seem to be related
,
Mar 11 2016
The waterfall was restarted this morning. Then the bots went offline after the restart. Any config changes cause this? Can we recover the offline bots now?
,
Mar 11 2016
We really needs to stop assigning me owner for infra-related bugs. Infra has a trooper rotation designed to handle exactly this sort of thing. If trooper needs to loop me in, that's the intended workflow. In this case, it looks like the problem is that several build slaves are offline: https://uberchromegw.corp.google.com/i/chromeos/buildslaves nodir@, can you kick them back alive?
,
Mar 11 2016
Leaving untriaged so this gets picked up by trooper queue.
,
Mar 11 2016
,
Mar 11 2016
Rebooted offline slaves
build{260,316}-m2 are still rebooting, offline, can't ssh.
,
Mar 11 2016
apparently build260-m2 didn't start buildbot automatically ran it manually, peach_pit-paladin is building: https://uberchromegw.corp.google.com/i/chromeos/builders/peach_pit-paladin/builds/11752
,
Mar 11 2016
Thanks! build316-m2 seems to be up also. I'll re-open the tree.
,
Mar 11 2016
I think this is practically fixed because builders started. master-paladin says 14:17:26: INFO: Still waiting for the following builds to complete: ['auron-paladin', 'daisy-paladin', 'falco-paladin', 'glados-paladin', 'leon-paladin', 'lumpy-incremental-paladin', 'nyan-paladin', 'nyan_freon-paladin', 'parrot-paladin', 'peach_pit-paladin', 'samus-paladin', 'stumpy-paladin', 'veyron_minnie-cheets-paladin', 'veyron_pinky-paladin'] 14:17:26: INFO: 2:18:05.546017 until timeout... no "No status found for build config" log lines anymore
,
Mar 18 2016
,
Apr 27 2016
,
Aug 12 2016
Closing. please reopen if its not fixed. |
|||||||||||
►
Sign in to add a comment |
|||||||||||
Comment 1 by shuqianz@chromium.org
, Mar 11 2016