The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5035f577b33f3b101b1f2e4a73927545cc44d8e4

commit 5035f577b33f3b101b1f2e4a73927545cc44d8e4
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed Feb 28 20:24:56 2018

Drain CQ for all CLs submitted to CQ after 12pm today.

No CLs submitted after that time to CQ will be procesed until
this CL is reverted.

R=iannucci@chromium.org, jchinlee@chromium.org

Bug:  817498 
Change-Id: I9e0f8a542cff370013d46166f12d5e81b3c1c58c
No-Try: True
Reviewed-on: https://chromium-review.googlesource.com/941967
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Cr-Commit-Position: refs/heads/master@{#539942}
[modify] https://crrev.com/5035f577b33f3b101b1f2e4a73927545cc44d8e4/infra/config/branch/cq.cfg

Could this be related to http://b/74010885?

If so, https://chromium-review.googlesource.com/c/chromium/src/+/940196 needs to be reverted, but I'm not sure how that happens with the code freeze in place.

Issue 817481 has been merged into this issue.

Thanks for that information, Tim! It's possible to have contributed; we think the likeliest root cause CL was something that caused a lot of bots to go offline, which once we quickly reverted it, caused the suddenly connected bots to exhaust [Gerrit] short term quota.

Both quotas will reset soon, so we can see if things improve then.

 Issue 817485  has been merged into this issue.

Any timeline as to when this will be fixed?

Quota usage and bot connectivity are stabilizing, and we hope to have everything fully open within the next hour.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2cbf4ed191dfd95cfb6adab309ae7c2e1ed2a67e

commit 2cbf4ed191dfd95cfb6adab309ae7c2e1ed2a67e
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed Feb 28 23:44:19 2018

Undrain CQ.

But leave doc for troopers in case this is needed again in the future.

R=jchinlee@chromium.org

Bug:  817498 
Change-Id: I46c33c17752f52394bb96cbd3d7bbbe944f89a17
No-Try: True
Reviewed-on: https://chromium-review.googlesource.com/942240
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Cr-Commit-Position: refs/heads/master@{#539948}
[modify] https://crrev.com/2cbf4ed191dfd95cfb6adab309ae7c2e1ed2a67e/infra/config/branch/cq.cfg

Affected machines have been respawned to pick up reverts
https://chrome-internal.googlesource.com/infra/puppet/+/66a2c59fc6d18a22b4dc07d7fb8cd1244c6135e8
(and https://chrome-internal.googlesource.com/infra/puppet/+/adcfe2470739f0147eaee9d5addb941cfa605ec9 to resolve conflict) and almost entirely online now.

CQ has been opened, and we're keeping an eye on quota consumption in order to unthrottle soon.

Issue 817511 has been merged into this issue.

And tree is open!

 Issue 817532  has been merged into this issue.

Detected 26 new flakes for test/step "update_scripts". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "update_scripts". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww. This message was posted automatically by the chromium-try-flakes app.

Detected 15 new flakes for test/step "update_scripts". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "update_scripts". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww. This message was posted automatically by the chromium-try-flakes app.

Detected 12 new flakes for test/step "update_scripts". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww. This message was posted automatically by the chromium-try-flakes app.

Detected 11 new flakes for test/step "update_scripts". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww. This message was posted automatically by the chromium-try-flakes app.

Latest additions to this bug seem totally unrelated. Quota issues have been fixed. What's happening here is general flakiness of update_script, which is KI and one of the reasons we have CIPD-based recipe bundler :)