and fyi - this also spams s-o-m

trooper here, looking

This is also blocking canary branch gnumbd, not all other branches like master. Lowering priority. IIUC this is what happened:

1. Branching script has tried to push a6376c2c38f5bbd6f56a89794f5003cce2aa89c2 from refs/pending/master into 
refs/pending-tags/branch-heads/2716, refs/pending-tags/branch-heads/2716 and refs/branch-heads/2716
2. It has succeeded for the first two refs, but failed for the third one. Even though we push all 3 refs in a single command, Git is doing this in 3 independent processes, which is why this is possible.
3. When retrying a push to refs/branch-heads/2716, Git looks up latest commit on refs/pending/master, which by this time has advanced to 550eecbe6487337a11088e59cb894b75a174c659 and that is what has been pushed to refs/pending/master.
4. Things got inconsistent and gnumbd started to issue the warning despite successfully updating all other refs.

We show 4 as an error because the branching script would be blocked by the inconsistency and thus no new canaries would be released.

Short-term solution would be to create a fake canary release by pushing a6376c2c38f5bbd6f56a89794f5003cce2aa89c2 to the refs/pending-tags/branch-heads/2717, refs/pending/branch-heads/2717 and refs/branch-heads/2717 manually.

Long-term solution, however, is to fix the branching script to check whether things are consistent after pushing things and if needed - correct them. Unless long-term solution is implemented soon, one should at least update the playbook explaining why this happens and what to do.

P.S. I have no idea where the branch script is.

It is blocking tomorrow's dev release.

I tried to delete the 2716 refs following the instruction in https://bugs.chromium.org/p/chromium/issues/detail?id=580493

dimu@dimu0:~/chromium/src$ git push --delete origin refs/pending-tags/branch-heads/2617
remote: Processing changes: refs: 1, done    
To https://chromium.googlesource.com/chromium/src.git
 ! [remote rejected] refs/pending-tags/branch-heads/2617 (cannot delete references)
error: failed to push some refs to 'https://chromium.googlesource.com/chromium/src.git'

not sure if it is failed due to permission issue.

Could someone on trooper please help to take a look? 

The branch script located in chrome-cron@wpih18.hot, in the src/pmo/chrome-cron/src/pmo/buildspec/scripts

sry wrong paste

the refs need to be deleted is 2716


dimu@dimu0:~/chromium/src$ git push --delete origin refs/pending-tags/branch-heads/2716
remote: Processing changes: refs: 1, done    
To https://chromium.googlesource.com/chromium/src.git
 ! [remote rejected] refs/pending-tags/branch-heads/2716 (cannot delete references)
error: failed to push some refs to 'https://chromium.googlesource.com/chromium/src.git'

I'll remove 2716 entirely and then you can re-create it

This should be cleaned up now.

Please let me know when you've successfully created this branch.

http://chromepmo.appspot.com/trigger/view?key=ag9zfmNocm9tZXBtby1ocmRyHAsSD0NvbW1hbmRSZXNwb25zZRiAgICQ6riNCgw

Looks like still the same error 

To https://chromium.googlesource.com/a/chromium/src.git
a6376c2..4226bdb 2716 -> refs/pending/branch-heads/2716
* [new branch] 2716 -> refs/pending-tags/branch-heads/2716
! [remote rejected] 2716 -> refs/branch-heads/2716 (prohibited by Gerrit)
error: failed to push some refs to 'https://chromium.googlesource.com/a/chromium/src.git'

Branch success!! 2716 looks happy now.