New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 619815 link

Starred by 3 users
Status: Fixed
Owner:
friedman@chromium.org
Closed: Jun 2016
Cc:
chrishall@chromium.org
█ dsansome@chromium.org
chrome-trooper-bugs@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 0
Type: ----

Blocking:
issue 619814



Sign in to add a comment

all masters on build.chromium.org are down

Project Member Reported by martiniss@chromium.org, Jun 14 2016 
Infra team is aware of the problem; it appears some infrastructure code accidentally deleted the postgres database which backs our buildbot instances. 

we appear to have found a fix; ETA on everything being back up is unknown.

Comment 1 by martiniss@chromium.org, Jun 14 2016

Summary: all masters on build.chromium.org are down (was: build.chromium.org is down)
It looks like we have to run a command for each master/builder combination, so ETA is a while. 

chromium.infra.cron is back up (with lost history) so fix is available. Need to apply everywhere

https://build.chromium.org/p/chromium.infra.cron/

Comment 2 by dimu@chromium.org, Jun 14 2016

Blocking: 619814

Comment 3 by martiniss@chromium.org, Jun 14 2016 

master1 is recovering right now; some masters are slowly starting to not 500

Comment 4 by martiniss@chromium.org, Jun 14 2016 

Master 1 (chromium.*), 3 (client.*), and 4 (try*) should be fixed. Not opening the tree until more is fixed.

Comment 5 by martiniss@chromium.org, Jun 14 2016 

Looks like all masters are back up (database). We've seen a couple good builds, but still need to verify everything.

Comment 6 by martiniss@chromium.org, Jun 14 2016 

(internal only) https://docs.google.com/spreadsheets/d/1OZXbwYXLxq9T3957mnCIFMAG-tz8EizlHoHiENiOueQ/edit#gid=0 list of masters which are up and seem good.

Comment 7 by fried...@google.com, Jun 14 2016

Status: Fixed (was: Started)
This was resolved at 23:24.

https://docs.google.com/document/d/187LmN11xtMc5EzyovSZNlKYjWklP0x5Ywbpl3DRbJ0c/edit?ts=575fa405

Comment 8 by friedman@chromium.org, Jun 14 2016

Owner: friedman@chromium.org

Comment 9 by chrishall@google.com, Jun 15 2016

Cc: chrishall@chromium.org
►

Sign in to add a comment