I suspect that this is because of the stabilize branch from the same release number. Both are creating pinned manifests in manifest-versions-internal, and incrementing on top of each other.

However, I don't understand why the patch number is incrementing.

This looks very suspicious to me, as I was digging through the version increment code.

From chromite/cbuildbot/stages/branch_stages.py:
  @staticmethod
  def DetermineBranchIncrParams(version_info):
    """Determines the version component to bump for the new branch."""
    # We increment the left-most component that is zero.
    if version_info.branch_build_number != '0':
      if version_info.patch_number != '0':
        raise BranchError('Version %s cannot be branched.' %
                          version_info.VersionString())
      return 'patch', 'patch number'
    else:
      return 'branch', 'branch number'

It seems to me the problem is around [1] but I can't predict the result from self.db.GetBuildHistory(). So may I know how to check the returned data? Thanks.

On the other hand, when increment type is chromeos_branch the returned value of _LatestSpecFromDir normally would be 11512.X.0 . But in this case, it became 11512.6.X since there is no 11512.7.0 yet. Since chromeos_version.sh currently of ocotpus factory branch is 11512.1.0 which is not equal to 11512.6.X, the version didn't increase one automatically.

Last update, I tried to bump build number to greater then 6 (11512.6), the version becomes 11512.1.0 now. will see whether it can be 11512.2.0 next time.

from [1]:
    # Calculate the status of the latest build, and whether the build was
    # processed.
    if version is None:
      self.latest = self._LatestSpecFromDir(version_info, self.all_specs_dir)
      if self.latest is not None:
        latest_builds = None
        if self.db is not None:
          latest_builds = self.db.GetBuildHistory(
              self.build_names[0], 1, platform_version=self.latest)
        if not latest_builds:
          self.latest_unprocessed = self.latest
        else:
          self._latest_build = latest_builds[0]

[1] https://chromium.googlesource.com/chromiumos/chromite/+/master/cbuildbot/manifest_version.py#753

Evan, you just looked at this code. Do you know what might be going on here?

From this information, it looks like branch-util did the right thing. The error probably occurs in the builder's increment logic, but I'm not sure. 

dgarrett@, was the stabilize branch you are referring to created from factory-11512.B? It could be that branch-util unexpectedly bumped a version on factory-* (the source branch)...but I think if that were the case, we would have discovered this issue sooner.

I have no idea what branches they were created from, but that's a very good question.

Convention says the factory branch would have come from master, and so be named factory-octopus-11512.B, and that the stabilize branch would have come from a release branch and so be named stabilize-11512.N.B where N is the release branch build it branched from.

However, if the branches came from odd places (say, factory came from the stabilize branch) that would break convention, and possibly our code.

The history was:

  Timeline 1: Both stabilize-altas-11512.B (Bob) and factory-octopus-11512.B (me) were created from 11512.0.0 (R73 in the ToT) in the similar time (within 10 minutes) so we didn't notice there is another 11512 branch already before creating them.

  Timeline 2: Then stablilize-altas-11512.B started to build product versions from 'tryjob' (not builder) and bumped to 11512.6.0 in buildspec.

  Timeline 3: We found this situation so altas decided to branch again - stablilize-altas-11512.6.B and continue to build product versions from 'tryjob' to 11512.6.3

  Timeline 4: factory builder - factory-octopus-11512.B in waterfall was ready so triggered build to verify. But the image version built was 11512.6.3 not 11512.1.0 .

  Timeline 5: I triggered several builds of factory-octopus-11512.B to build number - 7 then the image version became 11512.1.0 .

  Timeline 6: But builds after are still all kept in 11512.1.0 only and not bumped to 11512.2.0 and so forth.

> https://uberchromegw.corp.google.com/i/chromeos.branch/builders/octopus%20factory%20factory-octopus-11512.B/builds/9

The latest build still set version as 11512.1.0 and failed to push manifest to record pass result in manifest-version-internal.

[1]
cmd=['git', 'commit', '-m', u'Automatic checkin: status=pass build_version 11512.1.0 for octopus-factory'], cwd=/b/c/cbuild/repository/manifest-versions-internal

19:54:40: INFO: Translating result Failed to update the status for octopus-factory during remote command: return code: 1; command: git commit -m 'Automatic checkin: status=pass build_version 11512.1.0 for octopus-factory'

Hi all,

Ping for the question about [1]. What will be returned values for self.db.GetBuildHistory() there? It looks like _latest_build is assigned by that returned value. Probably we can manually set it as 11512.7.0?

On the other hand, the codes shows that if there is not returned value then the _latest_build depends on _LatestSpecFromDir(). As a result, probably I can submit a FAKE 11512.7.0 which becomes the maximum version for incr_type - branch and change chrome_version.sh to be with 11512.7.0 as well then next manifest version would be increased by the logic to 11512.8.0?

Sorry for if it didn't make sense since I am not that familiar with codes.

[1]
from [1]:
    # Calculate the status of the latest build, and whether the build was
    # processed.
    if version is None:
      self.latest = self._LatestSpecFromDir(version_info, self.all_specs_dir)
      if self.latest is not None:
        latest_builds = None
        if self.db is not None:
          latest_builds = self.db.GetBuildHistory(
              self.build_names[0], 1, platform_version=self.latest)
        if not latest_builds:
          self.latest_unprocessed = self.latest
        else:
          self._latest_build = latest_builds[0]

(+vapier who I assume you meant instead of apier)

Seems pretty clear that there's some cbuildbot/tryjob logic that reads the looks at existing buildspecs to determine what to build. As we've already hit on, it probably does not expect multiple branches off the same version.

I can't look this week because I am sheriffing, but I can look next week if it is still unresolved.

Note: Branching twice from the same version will not be allowed in the new branching tool.

As comment 12, can I commit 11512.7.0.xml to manfiest-version in order to make it as maxmium buildsspec of 11512.X.0? Thanks.