As a short-term workaround, we should make build/mac/should_use_hermetic_xcode.py return "no" on mojave.

+ sergeyberezin, justincohen

That's surprising. That suggests that the Xcode we're packaging is not working properly on 10.14. Apple is usually quite good about backwards compatibility. Maybe there's a problem with how we're packaging/deploying Xcode?

sergeyberezin -- Could you take a look at this?

Are you sure this is an issue with hermetic?

I just downloaded Xcode 8.3.3 and it does not run on macOS 10.14.  This is the full Xcode version, not hermetic.

 <snip>/Xcode.app/Contents/MacOS/Xcode
dyld: Symbol not found: _OBJC_IVAR_$_NSTextViewIvars.sharedData
  Referenced from: <snip>/Xcode.app/Contents/MacOS/../SharedFrameworks/DVTKit.framework/Versions/A/DVTKit
  Expected in: /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
 in <snip>/Xcode.app/Contents/MacOS/../SharedFrameworks/DVTKit.framework/Versions/A/DVTKit
Abort trap: 6

Erp, thanks for investigating. We're now stuck in an awkward situation, because we'd like to avoid rolling the Xcode version until we're ready to use the new SDK, but the old Xcode version doesn't work on 10.14.

c#1 sounds like the correct short term workaround, I'll implement that.

FYI, if we implement #c1, we would effectively be looking for a system-installed Xcode, which I don't think we should have on 10.14 machines (unless you specifically install newer Xcode manually, or request an exception from Labs).

Another option is to migrate the chromium recipe to use osx_sdk recipe module and specify different Xcode version for 10.14 in builder configs. Docs: https://chrome-internal.googlesource.com/infra/infra_internal/+/master/doc/troopers/playbooks/common.md#Xcode-is-missing-on-Mac-bots

Or, for a quicker but hackier solution, update mac_toolchain.py to do the same.

The priority here is devs upgrading to 10.14 and getting a broken build. We don't have 10.14 build machines yet.

I made should_use_hermetic_xcode.py to always return "0", but it didn't help. Do I need to do something else? Should I install Xcode 10 and point xcode-select to it?

btw, the issue doesn't look like P2 to me if everyone on Mojave is affected. 

Re #c7: yes, you'd need a working Xcode installed on the system in order for the workaround to work.
However, IIUC, it's a catch 22: Xcode 8.* doesn't work on Mojave, and Chromium might not compile with a newer Xcode. So basically, Mojave is a no go at this point for Chromium development.
The right fix (and possibly the only fix) is to roll the SDK to a newer Xcode, but that might take time.

You might be able to use Xcode 10 so long as you install the right SDK. We have some instructions for that at http://go/mac-10.12-sdk, but I don't know if anyone has tested that yet.

Building on Mojave with Xcode 10 did work for me. Although I did get a lot of linker warnings.

Installing Xcode 10 with:
sudo xcode-select --switch /Applications/Xcode.app/
worked for me with the workaround from #c1

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a537a9050e6a084e72ee43ad2e41eb2bd508e02c

commit a537a9050e6a084e72ee43ad2e41eb2bd508e02c
Author: Erik Chen <erikchen@chromium.org>
Date: Tue Nov 13 01:44:43 2018

Disable Hermetic Mac toolchain on macOS 10.14+.

The hermetic mac toolchain currently uses Xcode 8. Xcode 8 does not run on macOS
10.14+.

Bug:  904400 
Change-Id: I79afe5563631aac5c608c6ef57b594ae5de81e73
Reviewed-on: https://chromium-review.googlesource.com/c/1331907
Commit-Queue: Erik Chen <erikchen@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#607420}
[modify] https://crrev.com/a537a9050e6a084e72ee43ad2e41eb2bd508e02c/build/mac_toolchain.py

Hmmm, the patch didn't work for me.

$ gn gen out/Release
ERROR at //build_overrides/build.gni:53:3: Assertion failed.
  assert(_result != 2,
  ^-----
Do not allow building targets with the defaulthermetic toolchain if the minimum OS version is not met.
See //build/toolchain/toolchain.gni:8:1: whence it was imported.
import("//build_overrides/build.gni")
^-----------------------------------
See //build/config/coverage/coverage.gni:5:1: whence it was imported.
import("//build/toolchain/toolchain.gni")
^---------------------------------------
See //build/config/sanitizers/sanitizers.gni:7:1: whence it was imported.
import("//build/config/coverage/coverage.gni")
^--------------------------------------------
See //build/config/compiler/compiler.gni:8:1: whence it was imported.
import("//build/config/sanitizers/sanitizers.gni")
^------------------------------------------------
See //BUILD.gn:11:1: whence it was imported.
import("//build/config/compiler/compiler.gni")
^--------------------------------------------

Adding
use_system_xcode = true
to gn args seems to help.

> and Chromium might not compile with a newer Xcode.

That's usually not true. We generally try to keep chrome building with newer xcodes (newer SDKs, really), and most of the time that works.

I ran into this issue as well, #c14 fixed it for me. Could you update https://chromium.googlesource.com/chromium/src/+/HEAD/docs/mac_build_instructions.md to mention how to build on macOS 10.14?

c14 being needed is a bug. c12 should have had that effect without any user settings. That it didn't means that c12 needs more work. People should not have to set any vars, and once c12 has been amended people who did set these vars should unset them so that once we get hermetic working, they automatically pick it up.

Hm. The reason that c14 is needed is because we intentionally wanted to avoid a graceful fallback on our bots, which could potentially mask underlying problems.

I'm going to temporarily disable this assert.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/caca9998a7d4121ff8b6edec49c2def4d5dfb9e4

commit caca9998a7d4121ff8b6edec49c2def4d5dfb9e4
Author: Erik Chen <erikchen@chromium.org>
Date: Tue Nov 13 19:16:21 2018

Temporarily allow toolchain/OS mismatch to fallback on local toolchain.

Historically, we've disallowed toolchain/OS version mismatches to fallback on
local toolchain. This prevents bots from silently succeeding [but building with
the wrong toolchain].

Since the hermetic toolchain currently does not run on the
latest OS version, this check is also affecting all developers, which we'd like
to avoid. This CL temporarily disables the check.

Bug:  904400 
Change-Id: I4a3edc1218b50017a1dba96b54088986f48a6862
Reviewed-on: https://chromium-review.googlesource.com/c/1333877
Reviewed-by: Elly Fong-Jones <ellyjones@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#607695}
[modify] https://crrev.com/caca9998a7d4121ff8b6edec49c2def4d5dfb9e4/build_overrides/build.gni

alph, ahaas: Can you verify that things work for you after syncing without having to set any gn args? If so, please stop setting that gn arg mentioned in comment 14 :-)

Removing the setting breaks it :-(

$ gn args out/Release/
Waiting for editor on "/Users/alph/chromium/src/out/Release/args.gn"...
Generating files...
ERROR at //build/config/mac/mac_sdk.gni:68:5: Script returned non-zero exit code.
    exec_script("//build/mac/find_sdk.py", find_sdk_args, "list lines")
    ^----------
Current dir: /Users/alph/chromium/src/out/Release/
Command: python /Users/alph/chromium/src/build/mac/find_sdk.py --print_sdk_path --developer_dir /Users/alph/chromium/src/build/mac_files/Xcode.app 10.12
Returned 1.
stderr:

Traceback (most recent call last):
  File "/Users/alph/chromium/src/build/mac/find_sdk.py", line 104, in <module>
    print main()
  File "/Users/alph/chromium/src/build/mac/find_sdk.py", line 71, in main
    'to continue.')
__main__.SdkError: 'Install Xcode, launch it, accept the license agreement, and run `sudo xcode-select -s /path/to/Xcode.app` to continue.'

See //build/config/sysroot.gni:64:3: whence it was imported.
  import("//build/config/mac/mac_sdk.gni")
  ^--------------------------------------
See //chrome/installer/BUILD.gn:7:1: whence it was imported.
import("//build/config/sysroot.gni")
^----------------------------------
See //BUILD.gn:69:5: which caused the file to be included.
    "//chrome/installer",
    ^-------------------

I can confirm this is still broken with latest checkout and mojave. I need the following patch to be able to successfully run "gn gen" on mojave (I have Xcode 10 installed):

$ git diff
diff --git a/build_overrides/build.gni b/build_overrides/build.gni
index c2fc4775dae7..78bf8c39b123 100644
--- a/build_overrides/build.gni
+++ b/build_overrides/build.gni
@@ -56,5 +56,5 @@ if (host_os == "mac" && use_system_xcode == "") {
   # assert(_result != 2,
   #        "Do not allow building targets with the default" +
   #            "hermetic toolchain if the minimum OS version is not met.")
-  use_system_xcode = _result == 0
+  use_system_xcode = _result != 1
 }
$ gn gen out/foo
Done. Made 10136 targets from 1658 files in 10628ms

If I revert, I get build failure (as build tries to use hermetic Xcode which does not run):

$ git checkout -- :/
$ gn gen out/foo
ERROR at //build/config/mac/mac_sdk.gni:68:5: Script returned non-zero exit code.
    exec_script("//build/mac/find_sdk.py", find_sdk_args, "list lines")
    ^----------
Current dir: /Users/sdefresne/Developer/chromium/src/out/foo/
Command: python /Users/sdefresne/Developer/chromium/src/build/mac/find_sdk.py --print_sdk_path --developer_dir /Users/sdefresne/Developer/chromium/src/build/mac_files/Xcode.app 10.12
Returned 1.
stderr:

dyld: Symbol not found: _OBJC_IVAR_$_NSTextViewIvars.sharedData
  Referenced from: /Users/sdefresne/Developer/chromium/src/build/mac_files/Xcode.app/Contents/SharedFrameworks/DVTDocumentation.framework/Versions/A/../../../../SharedFrameworks/DVTKit.framework/Versions/A/DVTKit
  Expected in: /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
 in /Users/sdefresne/Developer/chromium/src/build/mac_files/Xcode.app/Contents/SharedFrameworks/DVTDocumentation.framework/Versions/A/../../../../SharedFrameworks/DVTKit.framework/Versions/A/DVTKit
xcrun: error: unable to locate xcodebuild, please make sure the path to the Xcode folder is set correctly!
xcrun: error: You can set the path to the Xcode folder using /usr/bin/xcode-select -switch
Traceback (most recent call last):
  File "/Users/sdefresne/Developer/chromium/src/build/mac/find_sdk.py", line 104, in <module>
    print main()
  File "/Users/sdefresne/Developer/chromium/src/build/mac/find_sdk.py", line 96, in main
    ['xcrun', '-sdk', 'macosx' + best_sdk, '--show-sdk-path']).strip()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['xcrun', '-sdk', 'macosx10.12', '--show-sdk-path']' returned non-zero exit status 71

See //build/config/sysroot.gni:64:3: whence it was imported.
  import("//build/config/mac/mac_sdk.gni")
  ^--------------------------------------
See //BUILD.gn:70:5: which caused the file to be included.
    "//net:net_unittests",
    ^--------------------

:( I've been writing patches on 10.13. 

I'll grab a 10.14 machine to test the patches...

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d2619f3b0f4807321f18065a469c2c371b006bb4

commit d2619f3b0f4807321f18065a469c2c371b006bb4
Author: Erik Chen <erikchen@chromium.org>
Date: Thu Nov 15 14:31:30 2018

macOS: Use system Xcode on 10.14.

A previous CL: https://chromium-review.googlesource.com/c/1333877 commented out
an assert to allow default fallback to the local toolchain, but failed to update
the subsequent conditional correctly.

Bug:  904400 
Change-Id: I8d299e60645db4821dbb79c45b57ac1c01767d57
Reviewed-on: https://chromium-review.googlesource.com/c/1335949
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#608359}
[modify] https://crrev.com/d2619f3b0f4807321f18065a469c2c371b006bb4/build_overrides/build.gni

alph, ahaas: can you try again after c25, please?

Works just fine for me. Thanks for the fixes!

I had this problem too after upgrading macOS to 10.14, I tried reinstalling xcode and doing lots of things suggested in various threads but no luck. Today I pulled the latest changes and it's compiling :) my laptop sounds like a helicopter trying to take off, I'm so happy.

Try using goma?

The workaround appears to have stuck. The long-term solution is rolling hermetic Xcode, which Elly is working on.