actually, looking at history, master-lakitu-release has been pushing a lot of weird stuff.

commit 32dbd502e45598347e6c5874e410288ccd834bb9
Author: chrome-bot <chrome-bot@chromium.org>
Date:   Thu Dec 20 10:16:54 2018 -0800

    Automatic: master-lakitu-release - Updating to a new version number from 11437.0.0
    
    Change-Id: Icaf2c17b6938c68cb29e38cb630a0af274e1dcfd

commit 646a401afbe4c9a33ce19b8428db4a6b134302cc
Author: chrome-bot <chrome-bot@chromium.org>
Date:   Thu Dec 20 10:12:57 2018 -0800

    Automatic: master-release - Updating to a new version number from 11436.0.0
    
    Change-Id: If0355658239f4ad9179bd26d7ecac26e4569062c

It's expected to increment version numbers, but not to publish ebuild uprevs. This should be fixable by tweaking the build config.

It's a full master builder for release builds with a different set of boards.

> It's expected to increment version numbers

why ?  that means we're increasing our OS version by 2x now ?  and any other master you want to add is going to increase it by another 1x ?

Yes. That's how master builders work today, and development resources for a system that's in the process of being deprecated are minimal.

I don't like it, but it was the expedient answer, especially since the new master has to be able to increment version numbers for branch builds. Having it work the same way everywhere seemed easiest.

There doesn't seem to be any relevant config difference between master-release and master-lakitu-release.

Looking closer at these two builds:

https://ci.chromium.org/p/chromeos/builders/luci.chromeos.general/Prod/b8926568821006741568

https://ci.chromium.org/p/chromeos/builders/luci.chromeos.general/Prod/b8926538622254762752

Why did I think it was publishing uprevs? No idea, I shouldn't have trusted whatever I found before coffee. Sigh.

it is publishing uprevs and it's very very bad that it is.  i linked to a bad commit in comment #0.

That CL came from this build:

https://ci.chromium.org/p/chromeos/builders/luci.chromeos.general/Prod/b8926568821012648560

  18:16:29: INFO: Updating manifest-versions checkout.
  18:16:54: INFO: Running cidb query on pid 13268, repr(query) starts with u'SELECT id, build_config, start_time, finish_time, status, waterfall, build_number, builder_name, p
  18:17:03: INFO: Updating version file to: 11440.0.0
  18:17:03: INFO: Pushing to branch (_RemoteRef(remote=u'cros', ref=u'refs/heads/master', project_name=u'chromiumos/overlays/chromiumos-overlay')) with message: Automatic: master-lakitu-release - Updating to a new version number from 11439.0.0 
  18:17:34: INFO: Incremented version number to  11440.0.0
  18:17:39: INFO: Publishing build spec for: 11440.0.0
  18:17:39: INFO: Publishing with commit message: Automatic: Start master-lakitu-release master 11440.0.0
  CrOS-Build-Id: 3257028
  18:17:48: INFO: Pushing to branch (_RemoteRef(remote='origin', ref='refs/heads/master', project_name=None)) with message: Automatic: Start master-lakitu-release master 11440.0.0
  CrOS-Build-Id: 3257028 
  18:17:49: ERROR: Failed to generate buildspec. error: return code: 1; command: git push origin temp_auto_checkin_branch:refs/heads/master
  To https://chrome-internal.googlesource.com/chromeos/manifest-versions
   ! [rejected]                temp_auto_checkin_branch -> master (fetch first)
  error: failed to push some refs to 'https://chrome-internal.googlesource.com/chromeos/manifest-versions'
  hint: Updates were rejected because the remote contains work that you do

The problem is that the builder in question hadn't yet performed an uprev.

And the previous build on that builder was a gandof-postsubmit build, there really, really shouldn't have been any state left behind.

i don't think the state of the tree matters.  we should only have one bot performing & committing uprevs here.

 Issue 918121  has been merged into this issue.

 issue 918121  showed another case where lakitu-release pushed an uprev it shouldn't have and it broke the tree

Happened to spot this. https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8925824206830075056/+/steps/Uprev/0/stdout

23:28:08: WARNING: Ignoring stable ebuild revision 0.0.3-r122 in /b/swarming/wD_wku7/ir/cache/cbuild/repository/src/third_party/chromiumos-overlay/chromeos-base/chromeos-firmware-null
...
23:28:38: WARNING: Ignoring stable ebuild revision 0.0.1-r7387 in /b/swarming/wD_wku7/ir/cache/cbuild/repository/src/third_party/chromiumos-overlay/chromeos-base/autotest-chrome

I looked at a few of these, and many (all?) of them were created by the CL in #0.

We should fix the root cause, but we should also remove all these extra ebuilds before they come back to bite us later (it took a while for the kernel issue to show up, when ${PV} was bumped)...

Having the master create the uprevs, and comitting them locally, is normal an expected.

Having it push them them externally, is not. Finding where it pushes them is what is confusing me.