also https://crrev.com/

also https://crosbug.com/

Cert request is here: https://gutsv3.corp.google.com/#ticket/24551295 - Priority set to Urgent.

Pinged agl@, sleevi@, awarner@ via hangouts.  No responses yet.

Certificate key has been placed on the netscalers at /nsconfig/ssl/2016_12_14/build.chromium.org.key.  The Cert should be placed there as well once it has been created via the ticket.

Got bif@ to respond.  He's currently working on the cert request for us.

 Issue 674339  has been merged into this issue.

I seem to have had a redirect for build.chromium.org in place that goes to https://chromium-build.appspot.com/ -- that seems to work fine. (the redirect.. I think it comes from an extension I installed at some point).

Outage ends.

We have 265 more days until this happens again.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome-golo/chrome-golo.git/+/8db6d8053935172c5869aaf6c3e5697b156f00d9

commit 8db6d8053935172c5869aaf6c3e5697b156f00d9
Author: Elliott Friedman <friedman@google.com>
Date: Thu Dec 15 00:52:28 2016

Is there a bug to fix this longer term, or for monitoring to not have this happen again?

I thought we did have basic probers up (at least for crbug.com) that should have at least notified us.

But +1 to not doing this emergency dance annually ...

Probers would be an improvement but ideally we'd catch these weeks early. I'll check w/ the sysadmins tomorrow but in the worst case we can have a dumb builder that just runs:

% echo crbug.com build.chromium.org chromium.org crrev.com codereview.chromium.org | fmt -1 | xargs -l ssl-cert-check -b -p 443 -s
crbug.com:443                                   Valid        Sep 7 2017   267
build.chromium.org:443                          Valid        Sep 7 2017   267
chromium.org:443                                Valid        Mar 2 2017   78
crrev.com:443                                   Valid        Sep 7 2017   267
codereview.chromium.org:443                     Valid        Sep 9 2017   269

and fails when certs are 14 days away from expiring.

Alerting level increased here: https://chrome-internal-review.googlesource.com/312051

Postmortem in progress.

Slightly off-topic, sorry: is there a newer equivalent of the old "crosbug.com/p" short URL? Something short to type that would redirect to the longer:
https://partnerissuetracker.corp.google.com/issues/35587084

The old syntax still works but only for old bugs, not for newer ones: http://crosbug.com/p/35587084 

Replying to the subject on this bug gave me a minor heart attack!  Glad this isn't an outage.  Can you open a new bug at go/infrasys-bug?  It seems crosbug.com/p is already implemented and I'm not entirely sure how you want it modified.

https://chrome-internal.googlesource.com/chrome-golo/chrome-golo/+/master/services/web/apache2/sites-available/crosbug.com