Issue 657059 has been merged into this issue.

PSA threads:
internal: https://groups.google.com/a/google.com/forum/#!topic/chrome-team/KkgTghI3kO8
public: https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/Kp76ADDuuxc

Note that this has much wider ramifications than yesterday. In particular:
* CQ is down because it can't reach Rietveld
* 'git cl land' won't work because it can't close the issues
* Trybots are down because they can't download patches

tandrii@ is working on workarounds for some of these, but no promises that they'll land before the root issue is resolved.

friedman@ is currently talking directly to the folks who wrote the Cloud Console to see if they can help set the appropriate certs which we can't.

Hi,

Is "git cl upload" expected to be affected as well?

Yes, basically all aspects of "git cl" for Rietveld (upload, status, land, probably some more) are affected at the moment.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/a8fb3a2892ae47c87a2562743b0e7a6fd1c7cf6d

commit a8fb3a2892ae47c87a2562743b0e7a6fd1c7cf6d
Author: Dan Jacques <dnj@chromium.org>
Date: Tue Oct 18 18:25:31 2016

git_cl: Allow server overriding.

BUG= chromium:657064 
TEST=local
  - Ran "git cl status" during outage with override, works.

R=tandrii@chromium.org

Review URL: https://chromiumcodereview-hr.appspot.com/2425673005 .

[modify] https://crrev.com/a8fb3a2892ae47c87a2562743b0e7a6fd1c7cf6d/git_cl.py

Note that you can still edit your git config file ('git config -e') and update the rietveldserver entries to point to chromiumcodereview.appspot.com

(I'm not from the infra team and I'm not so good with Git, so I might be wrong here, but it worked for me).

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal.git/+/59e407798eded1d61dbb746fad61b9f1897f6e8e

commit 59e407798eded1d61dbb746fad61b9f1897f6e8e
Author: tandrii <tandrii@google.com>
Date: Tue Oct 18 18:29:53 2016

For code review uploading and interfacing, you can use the (new) flag, "--force-codereview-server", to override your repository's code review server during the outage. You will have to update "depot_tools" to access the new flag and re-authenticate with the new server URL.

The flag to use for "codereview.chromium.org" is: --force-codereview-server=https://chromiumcodereview.appspot.com

Note that this will not work with "git cl try".

Is there any workaround for git drover?

yuweih@: There is not. Git drover simply calls "git cl upload" at the end of its run. Its usage is low enough that it is probably not worth adding a workaround to pass that flag before the outage is over.

If you want, you can edit your local "git_drover.py" ( https://chromium.googlesource.com/chromium/tools/depot_tools/+/master/git_drover.py#278 ) to include the new flag.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal.git/+/4058a8fcdd6d945f8e7f8667a0df5698be803a52

commit 4058a8fcdd6d945f8e7f8667a0df5698be803a52
Author: tandrii <tandrii@google.com>
Date: Tue Oct 18 18:45:08 2016

dnj@ SGTM. Thanks!

A new VIP has been set, and appears to have propagated. Everyone should have access to both codereview.chromium.org and crashpad.chromium.org now.

Please verify that you have access with a hard-refresh, and comment here if you don't.

Follow-up: remove the need for VIP at all, so this (exact thing) can't happen again.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/50f8208811016b04a278111a2e4103fa3a5f309d

commit 50f8208811016b04a278111a2e4103fa3a5f309d
Author: recipe-roller <recipe-roller@chromium.org>
Date: Tue Oct 18 19:06:31 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

depot_tools:
  https://crrev.com/a8fb3a2892ae47c87a2562743b0e7a6fd1c7cf6d git_cl: Allow server overriding. (dnj@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG= chromium:657064 

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://chromiumcodereview.appspot.com/2424153003

[modify] https://crrev.com/50f8208811016b04a278111a2e4103fa3a5f309d/infra/config/recipes.cfg

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal.git/+/f72c739c4d4ece96dd4de744f2d093164efc6164

commit f72c739c4d4ece96dd4de744f2d093164efc6164
Author: tandrii <tandrii@google.com>
Date: Tue Oct 18 19:10:43 2016

Reverted add flag CL: https://chromium.googlesource.com/chromium/tools/depot_tools/+/78de30e1bb2a3065ea78c28305be6782cc7c4e43

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/fd2c3e85c8a23ccb2e9a88a66604b2d749e303f7

commit fd2c3e85c8a23ccb2e9a88a66604b2d749e303f7
Author: recipe-roller <recipe-roller@chromium.org>
Date: Tue Oct 18 19:17:20 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/23484abd691e2e776f2c8a4aec706046d2ca7919

commit 23484abd691e2e776f2c8a4aec706046d2ca7919
Author: recipe-roller <recipe-roller@chromium.org>
Date: Tue Oct 18 19:24:28 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/3d68b83bc307ab0f55225956ce533d18fce26bac go/bisectrefactor (robertocn@chromium.org)
  https://crrev.com/50f8208811016b04a278111a2e4103fa3a5f309d Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
  https://crrev.com/e6351d31b52c10d59a56d72e626802c57546805c Revert of libyuv: Set symbol_level=1 for Android builders. (patchset #1 id:1 of https://codereview.chromium.org/2431633002/ ) (kjellander@chromium.org)
depot_tools:
  https://crrev.com/a8fb3a2892ae47c87a2562743b0e7a6fd1c7cf6d git_cl: Allow server overriding. (dnj@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG= chromium:657064 , chromium:652188 

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://chromiumcodereview.appspot.com/2425703008
Cr-Commit-Position: refs/heads/master@{#426021}

[modify] https://crrev.com/23484abd691e2e776f2c8a4aec706046d2ca7919/infra/config/recipes.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/43c57e94454da6858120e01f50133598cc31de85

commit 43c57e94454da6858120e01f50133598cc31de85
Author: recipe-roller <recipe-roller@chromium.org>
Date: Tue Oct 18 19:36:40 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/50f8208811016b04a278111a2e4103fa3a5f309d Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
  https://crrev.com/e6351d31b52c10d59a56d72e626802c57546805c Revert of libyuv: Set symbol_level=1 for Android builders. (patchset #1 id:1 of https://codereview.chromium.org/2431633002/ ) (kjellander@chromium.org)
depot_tools:
  https://crrev.com/a8fb3a2892ae47c87a2562743b0e7a6fd1c7cf6d git_cl: Allow server overriding. (dnj@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG= chromium:657064 , chromium:652188 

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://codereview.chromium.org/2428973002

[modify] https://crrev.com/43c57e94454da6858120e01f50133598cc31de85/infra/config/recipes.cfg

Postmortem: https://docs.google.com/document/d/1xHOHapH7X6ggw5DJeBTjcc3ywCNGPPXeqNV1693CltY/edit#heading=h.79bfjjbj7om6