It looks like we are somehow serving the differential installer for the Dev channel, for new installs.  Suggests that there might be a problem w/ lorry redirects.

We have another dev release tomorrow (06/14) and would be really great if we can fix this ASAP.

Thank you!

I see the problem.
* I have fixed the current dev install. The change will be effective at 10:19 PDT.
* I am working to make sure this will not recur in future pushes.

The bug has a chance of affecting future Dev, Beta, and Stable pushes (50% each, independently, on a push to any).

I'm pushing the fix now but it may be several hours until it's globally deployed. It will affect differential installers built during that window of time; so until then, I'll be manually monitoring the service and fixing up any issues that do occur.

I've confirmed that this only affects Mac.

I'll further update this bug when the change is fully deployed.

Thank you Joshua. I will keep an eye on all future releases.

Thank you Joshua.

Release plan for this week:
M53 Dev: Tomorrow (Tuesday)
M52 Beta: Wednesday
M51 Stable: TBD (This week some time)

This is now fixed:

I've corrected all differential installers this issue affected and future differential installers past this point will not be affected. We should be all set now.

Thanks for the quick fix!

What was broken, and why didn't we notice that it was broken until a user told us?

We serve certain payloads (such as the first-install .dmg files for Mac) off of known stable URLs. The release tool is programmed with a mapping of update-related-files to stable-URLs. A refactoring by yours truly introduced a bug where Mac differential updaters were marked as having the same stable URL basename as the full installers (instead of no stable URL basename at all).

Therefore, when executing a push, the system would make two payloads available on the same path in the download server: the full and diff installer. The download server behavior is well-defined in such a case, but the release tool's upload behavior is not. In at least some of the cases (maybe all?) the diff installer would win the race and be served instead of the full installer.

The code in question is not covered by a unit test, and the nature of this bug is that the end-to-end issue only manifests when we push, therefore it could not be caught in pre-push manual testing. (I don't know if we do pre-push manual testing on dev anyways.)

It could have been caught by us in a post-push manual test, but my guess is that we don't do that for dev channel updates, and there has not been a stable or beta push to trigger that process.