Moving to Infra>SDK. 

This isn't SDK: it's entirely about the toolchain used on bots, not by developers on their local machines. Adding jbuorick for the Platform/John's team.

This appears to be related to some of what +sergeyberezin is currently working on for iOS.

Indeed, it heavily intersects with issue 475693 and http://go/hermetic-xcode . I'm going to block this bug on the above for now, but maybe it should be merged instead.

This request appears to have 2 separate components:
 (A) Documentation for the toolchain updates;
 (B) Updating recipes [to use swarming?].

I'm leaving out the original 1b), because the impending LUCI transition will leave nothing but swarming anyway.

I'm not entirely sure how swarming is relevant to the SDK rolls, but I'll be updating recipes as part of issue 475693 to support the new flow, including specifying custom toolchain on swarming.

I'll definitely need to document the process when I'm done, it's part of the plan. So, unless I'm missing something else in this issue, it appears this request is a subset of issue 475693.

WDYT?

Xcode has OS build requirement. I thought issue 475693 was just for rolling a new version of Xcode, not for updating/changing the dimensions for the machines that compile chrome?

issue 475693 is about specifying the version of Xcode (and its toolchain) on demand. You don't need to pick a machine with the right Xcode anymore; instead, you just request what version you want and your build will install the right version.

Additionally, for the time being, we do not (yet) build Chrome on swarming, only on buildbot slaves. This is rapidly changing with LUCI, but I consider it an implementation detail that ideally you shouldn't worry about.

> you just request what version you want and your build will install the right version.
Right, assuming that the non-swarming machine supports the version you want. This is not always going to be the case.

>Additionally, for the time being, we do not (yet) build Chrome on swarming, only on buildbot slaves.
Agreed. Building Chrome on swarming is a pre-requisite for this crbug.

> Right, assuming that the non-swarming machine supports the version you want. This is not always going to be the case.

Why not? The only constraint I know is the OS version (as you already mentioned). We'll be upgrading the OS as needed (about once a year, it seems) for the entire fleet, so hopefully this will not be a problem. Do you know of any other constraints?

And by the time we need to upgrade the OS, I'm hoping buildbot will be a distant memory, and we'll be able to pick machines using swarming dimensions e.g. during the upgrade period when we'd have a mix of OS versions.

> Why not? The only constraint I know is the OS version (as you already mentioned). We'll be upgrading the OS as needed (about once a year, it seems) for the entire fleet, so hopefully this will not be a problem. Do you know of any other constraints?

We cannot upgrade to the latest Xcode 9.1 + 10.13 SDK, because it requires 10.12.6, whereas when we  were performing the massive set of upgrades needed to support 10.12 SDK, many machines were upgraded to 10.12.2.

> And by the time we need to upgrade the OS, I'm hoping buildbot will be a distant memory, and we'll be able to pick machines using swarming dimensions e.g. during the upgrade period when we'd have a mix of OS versions.

That would be very cool.

> We cannot upgrade to the latest Xcode 9.1 + 10.13 SDK, because it requires 10.12.6, whereas when we  were performing the massive set of upgrades needed to support 10.12 SDK, many machines were upgraded to 10.12.2.

I see. Yes, that's a good point - we may need to figure out how to upgrade the OS more often and more efficiently, at least for a subset of the fleet. But that's a different problem than this issue, it seems. It may deserve a separate bug blocking this issue, if it makes sense.

For this issue:

1a) will likely be taken care by the Foundation team as they migrate everything to LUCI;
1b) is a non-issue, buildbot goes away so we have no choice;
2) and 3) are already part of issue 475693, the way I envision it.

Sounds good. 

Is there a tracking bug/timeline for (1a)? 

I don't have a bug number, not sure a single such bug exists (estaab@ probably knows better). I get my info from http://go/chops-foundation notes (also posted to http://g/chops-all) and from occasional conversations in the hall.

IIUC, they want to start waterfall migrations likely next week, but I don't know how long it will take (I'm guessing all Q4 if not longer).

Thanks. Let's keep around this crbug to capture the feature that the macOS team wants. If there comes a point in time when the relevant work for (1a) is tracked, we can add that as a blocking bug.

SGTM, thanks!

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

FYI, LUCI migration for chromium CQ / CI is complete. There is no more buildbot, everything is on swarming. Is there anything else that this issue needs to track? If not, can we close it?

We may still need to update documentation:

"""
3) Document the steps needed to roll the toolchain, so that any Chrome macOS developer can roll the toolchain.
"""

Rolling the toolchain also requires being able to roll the OS version of devices that build Chrome. Is this process now automated and easy? If so this would unblock Issue 780980.

+ ellyjones

Documentation exists as an internal doc: http://shortn/_pLOeSQvVLN . I don't think we can make it public, since only Googlers can have access to uploading new Xcode versions. Maybe it can be referenced somewhere from Mac OS internal docs, for easier discovery.

Updating the OS, unfortunately, is still a manual process, and I don't believe we can easily automate it. Labs are getting pretty good at massive upgrades though (LUCI migration helped :-)

E.g. mac_chromium_rel_ng is now all on 10.13.3: https://chromium-swarm.appspot.com/botlist?c=id&c=task&c=os&c=status&d=asc&f=os%3AMac&f=builder%3Amac_chromium_rel_ng&k=builder&s=id
and can be used to try out Xcode 9.2, and maybe even Xcode 10. The new Xcode versions are already in CIPD - iOS team keeps them at bleeding edge for their builds.