New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 594233 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Mar 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

paladin builders not starting

Project Member Reported by sha...@chromium.org, Mar 11 2016

Issue description

master-paladin has encountered infra failures:

nyan-paladin did not start

parrot-paladin did not start

glados-paladin did not start

daisy-paladin did not start

falco-paladin did not start

leon-paladin did not start

oak-paladin did not start

veyron_pinky-paladin did not start

samus-paladin did not start

lumpy-incremental-paladin did not start

auron-paladin did not start

nyan_freon-paladin did not start

peach_pit-paladin did not start

See https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/10450



It looks like builds are pending, but buildslaves were offline, eg:

https://uberchromegw.corp.google.com/i/chromeos/builders/nyan-paladin

(Actually, now one of the nyan-paladin buildslaves seems to have come back...)
 
Cc: chrome-troopers@google.com

Comment 2 by sha...@chromium.org, Mar 11 2016

Many buildslaves are still offline:

https://uberchromegw.corp.google.com/i/chromeos/buildslaves

Are there any known infra outages occurring now?
Cc: dshi@chromium.org
+ dshi, secondary, nothing obvious broken in the lab. 
Owner: d...@chromium.org
Labels: -Pri-1 Pri-0
Cc: pschmidt@chromium.org
Could this be related to the file system corruption that happened on the pineview-release builder ( crbug.com/593942 )?


Comment 7 by no...@chromium.org, Mar 11 2016

No known infra outages right now, except goma which does not seem to be related
The waterfall was restarted this morning. Then the bots went offline after the restart. Any config changes cause this? Can we recover the offline bots now?

Comment 9 by d...@chromium.org, Mar 11 2016

We really needs to stop assigning me owner for infra-related bugs. Infra has a trooper rotation designed to handle exactly this sort of thing. If trooper needs to loop me in, that's the intended workflow.

In this case, it looks like the problem is that several build slaves are offline: https://uberchromegw.corp.google.com/i/chromeos/buildslaves

nodir@, can you kick them back alive?

Comment 10 by d...@chromium.org, Mar 11 2016

Owner: ----
Leaving untriaged so this gets picked up by trooper queue.

Comment 11 by no...@chromium.org, Mar 11 2016

Labels: Infra-Troopers
Owner: no...@chromium.org
Status: Started (was: Untriaged)

Comment 12 by no...@chromium.org, Mar 11 2016

Rebooted offline slaves
build{260,316}-m2 are still rebooting, offline, can't ssh.

Comment 13 by no...@chromium.org, Mar 11 2016

apparently build260-m2 didn't start buildbot automatically
ran it manually, peach_pit-paladin is building:  https://uberchromegw.corp.google.com/i/chromeos/builders/peach_pit-paladin/builds/11752
Thanks! build316-m2 seems to be up also. I'll re-open the tree.

Comment 15 by no...@chromium.org, Mar 11 2016

Labels: -Pri-0 Pri-1
I think this is practically fixed because builders started. master-paladin says

14:17:26: INFO: Still waiting for the following builds to complete: ['auron-paladin', 'daisy-paladin', 'falco-paladin', 'glados-paladin', 'leon-paladin', 'lumpy-incremental-paladin', 'nyan-paladin', 'nyan_freon-paladin', 'parrot-paladin', 'peach_pit-paladin', 'samus-paladin', 'stumpy-paladin', 'veyron_minnie-cheets-paladin', 'veyron_pinky-paladin']
14:17:26: INFO: 2:18:05.546017 until timeout...

no "No status found for build config" log lines anymore

Comment 16 by no...@chromium.org, Mar 18 2016

Status: Fixed (was: Started)
Components: Infra>Client>ChromeOS
Labels: -Infra-ChromeOS
Status: Verified (was: Fixed)
Closing. please reopen if its not fixed.

Sign in to add a comment