+dgarrett, akeshet

Any comment on how to fix this?

I don't know much about how the firmware builders build.

It sounds like we have both firmware and release builders incrementing the software version? That's probably bad. On the other hand, we probably want a unique version for firmware builders.

I'd have to understand the processes around firmware builds to give a strong recommendation about how to fix this.

Is this still happening?

I suspect so, we would have to manually start some firmware builders to be sure, but we have not changed much in this old logic. The priority is not super high since we can get reliable numbers by starting the pre flight builder first. 

https://bugs.chromium.org/p/chromium/issues/detail?id=536857 has more explanation of what is happening, not sure if it is worth duping into this, quoting the interesting bits from that:

"
Somewhere between 7132 and 7287 the version number in source built firmware started coming out wrong for the first member of a firmware branch build group. This manifests as the version string in the resulting firmware containing the build date instead of the proper third version number. For example you might get Google_Cyan.7287.57.2015_09_22_1357 instead of Google_Cyan.7287.57.10. Additionally when the branch is only one deep, the minor revision number also appears to be one too low, for example Intel_Strago.7287.62.2015_09_23_1919 instead of Intel_Strago.7287.63.0. 

Since this only happens on the first builder in the build group, this appears to have something to do with the uprev logic, making this difficult to reproduce locally.

Tracing down the logic in the firmware generation process it looks like:
chromeos-bootimage seems to put in this string (it only shows up in final bundled images).
chromeos-bootimage calls cros_bundle_firmware to generate the bundled images.
cros_bundle_firmware eventually gets into bundle_firmware.py.
bundle_firmware.py calls GetChromeosVersion() from tools.py.
GetChromeosVersion() is a wrapper to src/third_party/chromiumos-overlay/chromeos/config/chromeos_version.sh.
chromeos_version.sh gets updated by the builder, and contains the actual version number, it has an if statement on CHROMEOS_OFFICIAL to replace the last digit of the version number with a date or non official images. 

Based on this, it looks like somehow when the builder gets to building chromeos-bootimage, that the chromeos_version.sh may not yet be updated, and the CHROMEOS_OFFICIAL environment variable is not properly set. This appears to conflict with the BuildPackages log from the builder, which seems to run chromeos_version.sh and dump the output to the log before building the packages, only the values it outputs there are correct (minor revision is updated and no date in the version string).

Currently the workaround is to add a dummy preflight and first builder into a group, then the rest of the builds in the group get a valid number. The down side here is that it means we are generating more build artifacts than we need and it takes longer to get a build out.

This is a follow up generic bug for the internal issue https://code.google.com/p/chrome-os-partner/issues/detail?id=45708.

Any ideas?
 
Comment 1 by reinauer@chromium.org, Oct 2 2015
Delete comment ⚐
Owner: sosa@chromium.org
Status: Assigned
Comment 2 by bhthompson@chromium.org, Oct 7 2015
Delete comment ⚐
Adding another observation, if the builders are left to run on their own, the version number embedded in the rest of the builds from the build group are one behind (as  noted in https://code.google.com/p/chrome-os-partner/issues/detail?id=46022#c5). 

If one manually runs the build again, it revs again and the version number matches, for another example of this https://chromegw.corp.google.com/i/chromeos.branch/builders/cyan%20firmware%20firmware-cyan-7287.57.B/builds/23 appears to be 7287.57.22 but the firmware inside is labeled 7287.57.21, when run again manually in https://chromegw.corp.google.com/i/chromeos.branch/builders/cyan%20firmware%20firmware-cyan-7287.57.B/builds/24 the builder shows 7287.57.23 and the firmware now matches with 7287.57.23. 

It seems the only way to get a properly aligned firmware version is to manually run each build step and wait for completion of the prior stage before starting the next.
"