+alexclarke@ FYI

adding sheriffs as FYI.

duplicate of https://bugs.chromium.org/p/gerrit/issues/detail?id=9546 ?

Yes, sit's duplicate but in the different sub-tracker, so can't merge.
Let's keep it here for a bit since Gerrit Monorail doesn't assign priorities. Hopefully P0 will attract attention :)

See also https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/YnUEOtzCrzs

Issue 872729 has been merged into this issue.

I'm talking to the git-on-borg oncall to see if a new gerrit landed recently. Do we have a timeline on this?

Not that issue 872729 that was merged into this one, is also talking about commits that landed without Git number (Git Numberer plugin is not working), which broke many systems.

dart-review.googlesource.com  is not broken in any of these ways

An example of a working CL is 
https://dart-review.googlesource.com/c/sdk/+/68682

A rollback of gerrit pieces has started which is believe will relieve this problem. You can follow along in #gob

In V8 I force pushed back to the last good commit. The LUCI services seem to have tolerated that. Sadly CQ landed another one without CP afterwards.

Is force push generally advisable here? Is there any downside other than the authors of the commits that are omitted having to reland? (and they don't get any message about that...)

Issue 872789 has been merged into this issue.

As another mitigation I disabled "Submit" for V8 committers for refs/heads/master. But also CQ seems to land things without CP, so this might not help much.

Assigning to myself since I am the current incident commander.

 Issue 872835  has been merged into this issue.

https://groups.google.com/a/chromium.org/d/msg/chromium-dev/YnUEOtzCrzs/LW3JKUyFCwAJ

This appears to be fixed. Postmortem doc: https://docs.google.com/document/d/11UwPvlhjK5DLKSOBOpF5fRsT8zx1urEIDLavuAcTotY/edit?usp=sharing

V8 lkgr finder broke by this:
https://ci.chromium.org/p/v8/builders/luci.v8.ci/V8%20lkgr%20finder

It might impact chromium lkgr finder too...

Looks like after the force push some builds reported by milo are associated to non-existing revisions.

It is not fixed - while gerrit is back up, the fallout has not yet been dealt with.

Assigning to pkasting@ as part of handing off IC.

I filed a separate issue https://crbug.com/872853 for the lkgr finder.

 Issue 872775  has been merged into this issue.

Issue 872745 has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/78fe025a82e693c8e4b53ca887c503d5804f5faa

commit 78fe025a82e693c8e4b53ca887c503d5804f5faa
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Aug 09 19:54:32 2018

whitespace: to test git-numberer is working.

TBR=pkasting

Bug:  872722 
Change-Id: Iffb249c1d61d468ac8f301b4808049e03ab52127
Reviewed-on: https://chromium-review.googlesource.com/1169912
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#581888}
[modify] https://crrev.com/78fe025a82e693c8e4b53ca887c503d5804f5faa/build/whitespace_file.txt

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/17139b0d44c20c4aa6982adab731cf7aedb04305

commit 17139b0d44c20c4aa6982adab731cf7aedb04305
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Aug 09 20:11:21 2018

When the issue is fixed, would the "RELAND" button on the now-reverted CLs work properly (i.e. land through CQ), or I'd need to upload and land new CLs?

Re #31: yes, it will. It already does, but CQ won't currently land your CL -- but you can dry-run it already.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/019603a8c470db9940854761b52d1634ef8d2f9d

commit 019603a8c470db9940854761b52d1634ef8d2f9d
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Aug 09 20:16:12 2018

Re #32: Well, I tried it and it created a new CL mentioning "Reland" in title and referring to a deleted commit hash which I think would make the commit history confusing. And, I think I need to get LGTMs again. So, probably it's better to just re-upload?

I think the tools should be OK, but if you think readers would be confused, you're welcome to handle however you think best.

OK. Thanks

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/d35d16a9783a5197b915bd819f86e058c94afc44

commit d35d16a9783a5197b915bd819f86e058c94afc44
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Aug 09 20:43:23 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/86c31f358c6a86f3f8cad5e535aad98fc5b93f9b

commit 86c31f358c6a86f3f8cad5e535aad98fc5b93f9b
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Aug 09 21:07:23 2018

whitespace: now test CQ.

NOTREECHECKS=True

No-Presubmit: True
No-Try: True
Bug:  872722 
Change-Id: I78a357132559302f59848a712363d02cb2f0e91a
Reviewed-on: https://chromium-review.googlesource.com/1169949
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Peter Kasting <pkasting@chromium.org>
Cr-Commit-Position: refs/heads/master@{#581889}
[modify] https://crrev.com/86c31f358c6a86f3f8cad5e535aad98fc5b93f9b/build/whitespace_file.txt

Submit rights have been restored and the tree is open.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/ca18334db49aa3bafd05c7011429a0cc113501a2

commit ca18334db49aa3bafd05c7011429a0cc113501a2
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Aug 09 21:50:30 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/5bfb5f2acdd8190af83dfec174b614dffbab1b9f

commit 5bfb5f2acdd8190af83dfec174b614dffbab1b9f
Author: Andrii Shyshkalov <tandrii@google.com>
Date: Thu Aug 09 23:25:54 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal/+/6055bf46c124158848430cc6e68c6e4ec7b1bed9

commit 6055bf46c124158848430cc6e68c6e4ec7b1bed9
Author: Andrii Shyshkalov <tandrii@google.com>
Date: Fri Aug 10 00:31:56 2018

 Issue gerrit:9546  has been merged into this issue.

For some reason, I think this CL went it bypassing CQ.

I didn't see the Submit button on the main screen, so I clicked "..." menu, then Submit.

Then it just submitted to master without going through CQ.

Happened just now.

CL link: https://chromium-review.googlesource.com/c/chromium/src/+/1232525

After talking to some folks, I think the answer is:

There are two different buttons "Submit to CQ" and "Submit", which do different things. If that's the case, then things are working fine, I just misunderstood.