Issue 797838  has been merged into this issue.

Xcode 9.2 is the GM version of xcode, I'd recommend that version. I was thinking xcode 9.3 would be released soon but we don't have to wait for that.

Okay - what needs to be done to make this happen? +cc erikchen@ who might know :)

There are a few parallelizable steps:

1) Locally, bump the min SDK to 10.13, and deal with all compile issues that come up.
2) Upload a new Xcode 10.12.6 package to CIPD.

At the same time,

2) The minimum build requirement for xcode 9.2 is 10.12.6 [https://en.wikipedia.org/wiki/Xcode#9.x_series]. We need to create a list of all macOS devices on the infra waterfall that build with the SDK that are lower than 10.12.6, and ask infra-labs to update them.

Hi Erik,

Couple of questions. 

The current GM Xcode is now 9.3.1, which requires OSX 10.13. Do we just want to go with this or is there still a desire to target xcode 9.2  and OSX 12.6 anyway? My recommendation would be to just target OSX 10.13.4 and Xcode9.3.

"2) Upload a new Xcode 10.12.6 package to CIPD."

Whether we go with Xcode9.2 or Xcode9.3, both have already been uploaded to CIPD already by the iOS team.

"We need to create a list of all macOS devices on the infra waterfall that build with the SDK that are lower than 10.12.6"

As for gathering the list of builders, if it's fine to upgrade all builders to 10.13, which I assume it is since we'd want to bump the target to Xcode9.3 in theory, infra/labs can compile this list when they receive the ticket from us, we don't have to provide the list to them.

We can just file a blocking bug at http://go/infrasys-bugs when Step 1 (from Comment #12) is done.

> My recommendation would be to just target OSX 10.13.4 and Xcode9.3.

Agreed.

I think that's the only question for me to answer. Looks like you've got things well under control. Thanks!

FYI I don't think Xcode8 will run on 10.14...

Building on 10.14 with xcode8 doesn't seem to be working on the mojave bot I'm bringing up

So just a consideration for this bug. I'll make this a simple tester in the mean time, no build. 

Just chatted with erikchen@ about this. Here's what needs to happen, in order:

1a) Mac team creates and uploads a 10.13 hermetic environment, containing Xcode 9 and the 10.13 SDK.
1b) Infra creates a macOS 10.13 FYI bot for us, which builds and runs tests locally, on macOS 10.13, with the existing hermetic environment (which uses the 10.12 SDK and Xcode 8)
2) Mac team does any necessary work to get that bot green
3) Infra starts upgrading the fleet of build hosts from whatever OS they're on right now to 10.13.
4) Mac team updates the FYI bot to the Xcode 9 / 10.13 SDK hermetic environment
5) Mac team does any necessary work to get that bot green (again)
6) Mac team updates the default hermetic environment from Xcode 8 / 10.12 SDK to Xcode 9 / 10.13 SDK

... and then we repeat for Xcode 10 / 10.14 SDK.

Will follow up with jbudorick@ on Thursday.

The NextAction date has arrived: 2018-11-15

I'm wondering if this relevant? --I upgraded to macOS Mojave 10.14.1 and I can't compile chrome anymore due to not being able to find xcode. I've tried reinstalling it, running "sudo xcode-select -s", re-fetching chromium, etc. but no luck.
Did I hit the "upper bound"? https://cs.chromium.org/chromium/src/build/mac_toolchain.py?type=cs&q=PlatformMeetsHermeticXcodeRequirements&sq=package:chromium&g=0&l=37

hbos: See  issue 904400  for a temporary fix. You may need to git pull / gclient sync to the very latest revision of everything, and have Xcode 10 installed on the system, with xcode-select -s pointing to it.

sergeyberezin: Thank you :D

I sent to the CQ a CL that tried to use Xcode 9: <https://chromium-review.googlesource.com/c/chromium/src/+/1351377>. This CL failed to build:

[209/43785] OBJCXX obj/base/base/bundle_locations.o
FAILED: obj/base/base/bundle_locations.o 
export DEVELOPER_DIR=/b/s/w/ir/cache/builder/src/build/mac_files/Xcode.app;  /b/s/w/ir/cache/goma/client/gomacc ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF obj/base/base/bundle_locations.o.d -DSYSTEM_NATIVE_UTF8 -DV8_DEPRECATION_WARNINGS -DDCHECK_ALWAYS_ON=1 -DNO_TCMALLOC -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -DFIELDTRIAL_TESTING_ENABLED -D_LIBCPP_HAS_NO_ALIGNED_ALLOCATION -DCR_XCODE_VERSION=0930 -DCR_CLANG_REVISION=\"346388-1\" -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -D__ASSERT_MACROS_DEFINE_VERSIONS_WITHOUT_UNDERSCORE=0 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DBASE_IMPLEMENTATION -I../.. -Igen -fno-strict-aliasing -fstack-protector -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -fcolor-diagnostics -fmerge-all-constants -Xclang -mllvm -Xclang -instcombine-lower-dbg-declare=0 -no-canonical-prefixes -arch x86_64 -Wall -Werror -Wextra -Wimplicit-fallthrough -Wthread-safety -Wunguarded-availability -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-unneeded-internal-declaration -Wno-undefined-var-template -Wno-null-pointer-arithmetic -Wno-ignored-pragma-optimize -fno-omit-frame-pointer -g1 -isysroot ../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -mmacosx-version-min=10.9.0 -fvisibility=hidden -Xclang -load -Xclang ../../third_party/llvm-build/Release+Asserts/lib/libFindBadConstructs.dylib -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang enforce-in-thirdparty-webkit -Xclang -plugin-arg-find-bad-constructs -Xclang check-enum-max-value -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Wno-char-subscripts -Wglobal-constructors -Wexit-time-destructors -Wshadow -Wexit-time-destructors -O2 -std=c++14 -stdlib=libc++ -fobjc-call-cxx-cdtors -Wobjc-missing-property-synthesis -fno-exceptions -fno-rtti -fvisibility-inlines-hidden -c ../../base/mac/bundle_locations.mm -o obj/base/base/bundle_locations.o
In file included from ../../base/mac/bundle_locations.mm:5:
In file included from ../../base/mac/bundle_locations.h:12:
In file included from ../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/Foundation.framework/Headers/Foundation.h:159:
In file included from ../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSAppleEventDescriptor.h:8:
../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/ApplicationServices.framework/Headers/ApplicationServices.h:31:10: fatal error: 'ColorSync/ColorSync.h' file not found
#include <ColorSync/ColorSync.h>
         ^~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

I don't fully understand this, since that header file is definitely present for me locally - perhaps it has to do with how CIPD deploys the new SDK? Anyway, trying this CL somehow poisoned the bot such that a clobber was needed to make it build with Xcode 8 again :\.

https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8928771526502315520/+/steps/gclient_runhooks__with_patch_/0/stdout from your try jobs has lots of 

[P50229 10:33:22.031 fs.go:358 W] fs: failed to remove /b/s/w/ir/cache/builder/src/build/mac_files/Xcode.app/Contents/Frameworks/IDEKit.framework/Versions/A/Resources/English.lproj/IDEReviewFilesView.nib - remove /b/s/w/ir/cache/builder/src/build/mac_files/Xcode.app/Contents/Frameworks/IDEKit.framework/Versions/A/Resources/English.lproj/IDEReviewFilesView.nib: directory not empty


which doesn't make me feel super confident about the xcode update code in cipd. Maybe check where these errors come from, check who wrote that code, and ask them for input?

+cc sergeyberezin@, iannucci@, vadimsh for #28 & #29

I tried repro'ing this locally by running:

cipd auth-login
build/mac_toolchain.py  # Hacked up to make _UseHermeticToolchain() return "true"

cd out/gn
DEVELOPER_DIR=$PWD/../../build/mac_files/Xcode.app GOMA_USE_LOCAL=false ~/goma/gomacc ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF obj/base/base/bundle_locations.o.d -DSYSTEM_NATIVE_UTF8 -DV8_DEPRECATION_WARNINGS -DDCHECK_ALWAYS_ON=1 -DNO_TCMALLOC -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -DFIELDTRIAL_TESTING_ENABLED -D_LIBCPP_HAS_NO_ALIGNED_ALLOCATION -DCR_XCODE_VERSION=0930 -DCR_CLANG_REVISION=\"346388-1\" -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -D__ASSERT_MACROS_DEFINE_VERSIONS_WITHOUT_UNDERSCORE=0 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DBASE_IMPLEMENTATION -I../.. -Igen -fno-strict-aliasing -fstack-protector -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -fcolor-diagnostics -fmerge-all-constants -Xclang -mllvm -Xclang -instcombine-lower-dbg-declare=0 -no-canonical-prefixes -arch x86_64 -Wall -Werror -Wextra -Wimplicit-fallthrough -Wthread-safety -Wunguarded-availability -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-unneeded-internal-declaration -Wno-undefined-var-template -Wno-null-pointer-arithmetic -Wno-ignored-pragma-optimize -fno-omit-frame-pointer -g1 -isysroot ../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -mmacosx-version-min=10.9.0 -fvisibility=hidden -Xclang -load -Xclang ../../third_party/llvm-build/Release+Asserts/lib/libFindBadConstructs.dylib -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang enforce-in-thirdparty-webkit -Xclang -plugin-arg-find-bad-constructs -Xclang check-enum-max-value -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Wno-char-subscripts -Wglobal-constructors -Wexit-time-destructors -Wshadow -Wexit-time-destructors -O2 -std=c++14 -stdlib=libc++ -fobjc-call-cxx-cdtors -Wobjc-missing-property-synthesis -fno-exceptions -fno-rtti -fvisibility-inlines-hidden -c ../../base/mac/bundle_locations.mm -o obj/base/base/bundle_locations.o


...but that worked fine over here. Still, that DEVELOPER_DIR is making me suspicious and as mentioned on b/119236037 I think we should remove it for compiles -- let me make a CL for that. (Maybe the fix for b/119236037 helped with this issue?)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c70ff02e893652d6b3e2f3e91bcfd47cf0320ebf

commit c70ff02e893652d6b3e2f3e91bcfd47cf0320ebf
Author: Nico Weber <thakis@chromium.org>
Date: Mon Dec 03 17:08:23 2018

mac: Don't set DEVELOPER_DIR env var for compiles.

Compiles shouldn't shell out to non-clang processes, and we know that
clang doesn't read DEVELOPER_DIR, so this shouldn't be needed.

Bug: 780980
Change-Id: I811786f567c517aee9a82867336da7b2adb3bf44
Reviewed-on: https://chromium-review.googlesource.com/c/1358698
Reviewed-by: Erik Chen <erikchen@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#613129}
[modify] https://crrev.com/c70ff02e893652d6b3e2f3e91bcfd47cf0320ebf/build/toolchain/mac/BUILD.gn

Re #29: I can reproduce this locally and the update indeed butchers Xcode directory. Working now on figuring out what's going on.

#34: Any luck here? It's still busted as far as I can tell: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_compile_dbg_ng/202911

It is now busted for some other reason.  Issue 911229  has been fixed. 'gclient runhooks' finishes without warnings.

I tried a new build on a bot today and got the same error about ColorSync/ColorSync.h being absent - do you think there's another root cause of the same problem?

ColorSync/ColorSync.h error was indeed due to CIPD bugs. I think I fixed it this time. I apologize I didn't test the previous "fix" more rigorously.

New CIPD version were able to update (https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_compile_dbg_ng/207483) and revert (https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_compile_dbg_ng/207558) Xcode successfully.

After the fixes on  issue 915278 , when I try my 10.13 roll CL <https://chromium-review.googlesource.com/c/chromium/src/+/1334092>, I get this: <https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_compile_dbg_ng/212477> and specifically:

Traceback (most recent call last):
  File "/b/s/w/ir/cache/builder/src/build/mac/find_sdk.py", line 104, in <module>
    print main()
  File "/b/s/w/ir/cache/builder/src/build/mac/find_sdk.py", line 77, in main
    raise Exception('No %s+ SDK found' % min_sdk_version)
Exception: No 10.13+ SDK found

I had expected that setting min_sdk_version would suffice to cause the bots to install that SDK version. Maybe I also need to set MAC_TOOLCHAIN_VERSION?

Yes. mac_sdk_min (which I guess you mean?) is a gn-only thing that's passed to build/mac/find_sdk.py to find an SDK with that version. It doesn't cause that SDK to be installed -- fin_sdk.py just checks what's available locally. (It predates the hermetic xcode stuff by a few years.)

MAC_TOOLCHAIN_VERSION determines what build/mac_toolchain.py downloads, which runs as a gclient hook.

A test fails when rolling to Xcode 9:

[ RUN      ] NativeWidgetMacTest.MiniaturizeExternally
../../ui/views/widget/native_widget_mac_unittest.mm:616: Failure
Value of: widget->IsMinimized()
  Actual: false
Expected: true
Stack trace:
0   views_unittests                     0x0000000102bbb41b testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop(int) + 91
1   views_unittests                     0x0000000102bbad39 testing::internal::AssertHelper::operator=(testing::Message const&) const + 89
2   views_unittests                     0x0000000101e4e84a views::test::NativeWidgetMacTest_MiniaturizeExternally_Test::TestBody() + 12010

[  FAILED  ] NativeWidgetMacTest.MiniaturizeExternally (2402 ms)

After much digging, it turns out that the behavior of [NSWindow miniaturize:] changed to early-out if !(window.styleMask & NSMiniaturizableWindowMask), and this is controlled by __NSAppGetBoolConfig(@"NSWindowRespectMiniaturizableMask", ...). The default value function for this checks whether the SDK version is > 0xa0cff.

Probably the best way forward is for us to override [NSWindow _canMiniaturize] in NativeWidgetMacNSWindow.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1c966377e14e9a1f54fdcae1403e01ff5f771d9b

commit 1c966377e14e9a1f54fdcae1403e01ff5f771d9b
Author: Elly Fong-Jones <ellyjones@chromium.org>
Date: Thu Jan 03 16:39:09 2019

macviews: fix window miniaturization test with newer SDKs

This change:

1) Splits up NativeWidgetMacTest.MiniaturizeExternally, which was a bit too big;
2) Adds an override on NativeWidgetMacNSWindow of _canMiniaturize, which
   always allows Views windows to be miniaturized; this is the behavior that
   the rest of Views (and the tests) expect, and is what happened on older SDKs.

Bug: 780980
Change-Id: Ic3356787433d9325aa13f1e01a8165c4ddd07b0a
Reviewed-on: https://chromium-review.googlesource.com/c/1394010
Reviewed-by: Avi Drissman <avi@chromium.org>
Commit-Queue: Elly Fong-Jones <ellyjones@chromium.org>
Cr-Commit-Position: refs/heads/master@{#619656}
[modify] https://crrev.com/1c966377e14e9a1f54fdcae1403e01ff5f771d9b/ui/views/widget/native_widget_mac_unittest.mm
[modify] https://crrev.com/1c966377e14e9a1f54fdcae1403e01ff5f771d9b/ui/views_bridge_mac/native_widget_mac_nswindow.mm

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d86a65614611266e5eaea2ffbda3463740e0b627

commit d86a65614611266e5eaea2ffbda3463740e0b627
Author: Elly Fong-Jones <ellyjones@chromium.org>
Date: Fri Jan 04 19:40:34 2019

mac: roll to 10.13 SDK

This change causes the default hermetic SDK to be version 10.13 (aka Xcode 9.3).
This also bumps the minimum build host version to 10.13.2, since that
is the earliest version on which Xcode 9.3 runs.

Bug: 780980
Change-Id: I6da3f7d3a060a22d71d5ee18bb88f51d5aedb0a2
Reviewed-on: https://chromium-review.googlesource.com/c/1394742
Commit-Queue: Elly Fong-Jones <ellyjones@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#620032}
[modify] https://crrev.com/d86a65614611266e5eaea2ffbda3463740e0b627/build/config/mac/mac_sdk_overrides.gni
[modify] https://crrev.com/d86a65614611266e5eaea2ffbda3463740e0b627/build/mac_toolchain.py

\o/

if that sticks, I'll mark this Fixed and move on to 10.14 :)

Sheriff here - #45 is strongly suspected as culprit for the consistently failing generate_buildfiles step (gn gen) beginning at https://ci.chromium.org/buildbot/chromium.chrome/Google%20Chrome%20Mac/40240

I'll revert #45 shortly.

https://ci.chromium.org/buildbot/chromium.chrome/Google%20Chrome%20Mac/40240 runs on https://build.chromium.org/deprecated/chromium.chrome/buildslaves/vm669-m1 which is os version: macOS 10.12.2 (16C68). The CL made 10.13 a requirement, so I agree that looks like a likely cause.

We need to revert and make sure all builders are on 10.13.2 before relanding.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b4c44dd165f1f3c3e60db8f77b823594e9f1bc2e

commit b4c44dd165f1f3c3e60db8f77b823594e9f1bc2e
Author: Matthew Wolenetz <wolenetz@chromium.org>
Date: Fri Jan 04 21:16:17 2019

Revert "mac: roll to 10.13 SDK"

This reverts commit d86a65614611266e5eaea2ffbda3463740e0b627.

Reason for revert: Strongly suspected as culprit for consistent "gn gen" failures during generate_build_files beginning at https://ci.chromium.org/buildbot/chromium.chrome/Google%20Chrome%20Mac/40240

Original change's description:
> mac: roll to 10.13 SDK
> 
> This change causes the default hermetic SDK to be version 10.13 (aka Xcode 9.3).
> This also bumps the minimum build host version to 10.13.2, since that
> is the earliest version on which Xcode 9.3 runs.
> 
> Bug: 780980
> Change-Id: I6da3f7d3a060a22d71d5ee18bb88f51d5aedb0a2
> Reviewed-on: https://chromium-review.googlesource.com/c/1394742
> Commit-Queue: Elly Fong-Jones <ellyjones@chromium.org>
> Reviewed-by: Nico Weber <thakis@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#620032}

TBR=ellyjones@chromium.org,thakis@chromium.org

Change-Id: I43c3e9d6e476238d992cf2f703b08ab7875daf2f
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 780980
Reviewed-on: https://chromium-review.googlesource.com/c/1396153
Reviewed-by: Matthew Wolenetz <wolenetz@chromium.org>
Commit-Queue: Matthew Wolenetz <wolenetz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#620066}
[modify] https://crrev.com/b4c44dd165f1f3c3e60db8f77b823594e9f1bc2e/build/config/mac/mac_sdk_overrides.gni
[modify] https://crrev.com/b4c44dd165f1f3c3e60db8f77b823594e9f1bc2e/build/mac_toolchain.py

The revert landed in #50 enabled generate_build_files to go back to green on the bot as of https://ci.chromium.org/buildbot/chromium.chrome/Google%20Chrome%20Mac/40253

bump! this is a Q1 KR :)