I've started a bisect.

And I can reproduce both failure and success, so it is an LLVM-side change.

Bisection points to r339807

commit 37e912ce6bb71dff5d150f49ce82c654e6826456
Author: Chris Bieneman <chris.bieneman@me.com>
Date:   Wed Aug 15 20:09:38 2018 +0000

    Refactor Darwin driver to refer to runtimes by component
    
    Summary:
    In r335809, Petr Hosek lays out support for what he calls the multiarch
    runtimes layout. This new way of laying out the directories for runtime
    libraries is workable for all platforms. Petr did some of the common
    infrastructure and made it work for Linux and Fuscia. This patch is a
    cleanup to the Darwin and MachO drivers to serve as a step toward
    supporting it in Darwin.
    
    This patch does primarily two things:
    (1) Changes the APIs for how the Darwin driver refers to compiler-rt
    libraries to use the component names, similar to how Linux and Fuscia do
    
    (2) Removes some legacy functionality for supporting macOS versions
    before 10.6. This functionality is effectively dead code because in
    r339277, the support was removed from compiler-rt for generating the 10.4
    runtime support library, and Xcode 10 (currently in beta) removes
    libgcc_s.10.4 and libgcc_s.10.5 from the macOS SDK.
    
    With this patch landed a subsequent patch can modify
    MachO::AddLinkRuntimeLib to support the multiarch runtimes layout.
    
    Worth noting: None of the removed functionality was actually covered in
    the test suite. So no test case updates are required.
    
    Reviewers: phosek, bruno, arphaman
    
    Reviewed By: phosek, arphaman
    
    Subscribers: cfe-commits
    
    Differential Revision: https://reviews.llvm.org/D50618

The linker step is invoked like this:

export DEVELOPER_DIR=/work/chromium/src/build/mac_files/Xcode.app;  TOOL_VERSION=1521192979 ../../build/toolchain/mac/linker_driver.py ../../../../llvm.combined/build.release/bin/clang++  -stdlib=libc++ -arch x86_64 -segprot PROTECTED_MEMORY rw r -Werror -Wl,-ObjC -Wl,-rpath,@loader_path/. -Wl,-rpath,@loader_path/../../.. -isysroot /work/chromium/src/build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -mmacosx-version-min=10.5 -o "./gcapi_example" -Wl,-filelist,"./gcapi_example.rsp"  -framework Cocoa

-### before the change:

clang version 8.0.0 
Target: x86_64-apple-darwin17.7.0
Thread model: posix
InstalledDir: /work/chromium/src/out/release/../../../../llvm.combined/build.release/bin
 "/usr/bin/ld" "-demangle" "-lto_library" "/work/llvm.combined/build.release/lib/libLTO.dylib" "-dynamic" "-arch" "x86_64" "-macosx_version_min" "10.5.0" "-segprot" "PROTECTED_MEMORY" "rw" "r" "-syslibroot" "/work/chromium/src/build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk" "-o" "./gcapi_example" "-lcrt1.10.5.o" "-ObjC" "-rpath" "@loader_path/." "-rpath" "@loader_path/../../.." "-filelist" "./gcapi_example.rsp" "-framework" "Cocoa" "-lSystem" "-lgcc_s.10.5" "/work/llvm.combined/build.release/lib/clang/8.0.0/lib/darwin/libclang_rt.osx.a"



-### after the change:

clang version 8.0.0 
Target: x86_64-apple-darwin17.7.0
Thread model: posix
InstalledDir: /work/chromium/src/out/release/../../../../llvm.combined/build.release/bin
 "/usr/bin/ld" "-demangle" "-lto_library" "/work/llvm.combined/build.release/lib/libLTO.dylib" "-dynamic" "-arch" "x86_64" "-macosx_version_min" "10.5.0" "-segprot" "PROTECTED_MEMORY" "rw" "r" "-syslibroot" "/work/chromium/src/build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk" "-o" "./gcapi_example" "-lcrt1.10.5.o" "-ObjC" "-rpath" "@loader_path/." "-rpath" "@loader_path/../../.." "-filelist" "./gcapi_example.rsp" "-framework" "Cocoa" "-lSystem" "/work/llvm.combined/build.release/lib/clang/8.0.0/lib/darwin/libclang_rt.osx.a"


It's missing the "-lSystem" "-lgcc_s.10.5" part.

It seems that for gcapi, Chromium sets -mmacosx-version-min=10.5 which is lower than what's used for the browser in general: https://cs.chromium.org/chromium/src/chrome/installer/gcapi_mac/BUILD.gn?l=9

r339807 removed the ability to target that version, saying the Xcode 10 SDK won't include a required library anyway.


thakis: What is gcapi, does it really need to target 10.5 or could we bump it to, say, 10.7?

> It seems that for gcapi, Chromium sets -mmacosx-version-min=10.5 which is lower than what's used for the browser in general: https://cs.chromium.org/chromium/src/chrome/installer/gcapi_mac/BUILD.gn?l=9
>
> r339807 removed the ability to target that version, saying the Xcode 10 SDK won't include a required library anyway.
>
>
> thakis: What is gcapi, does it really need to target 10.5 or could we bump it to, say, 10.7?

Actually +thakis this time

gcapi is a library that the installers of other products bundle when they want to offer to optionally install chrome too. Since other products potentially run on older macOS versions than Chrome, we made it run on older systems.

10.5 is very old and bumping it to 10.7 is probably fine (+mark who might have opinions, and +ellyjones, who's soft-maintaining rlz/mac for more other opinions).

But has that change in behavior been intentional? My guess it might not have been, so maybe first talk to cbieneman?

Thanks for the input!

> But has that change in behavior been intentional? My guess it might not have been, so maybe first talk to cbieneman?

Based on the commit message, it looks intentional. I've replied on the commit thread to make cbieneman aware it affected us, but I suspect bumping to 10.7 is the right way to handle this (I'll prepare a CL in the meantime).

>  Removes some legacy functionality for supporting macOS versions before 10.6.

This sounds like bumping to 10.6 might be enough?

> This sounds like bumping to 10.6 might be enough?

Yes, should be.

CL: https://chromium-review.googlesource.com/c/chromium/src/+/1179748

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/046d539b96f85568341629d6a7166f62817de501

commit 046d539b96f85568341629d6a7166f62817de501
Author: Hans Wennborg <hans@chromium.org>
Date: Fri Aug 17 12:29:49 2018

gcapi: Bump -mmacosx-version-min= from 10.5 to 10.6

Clang and the macOS SDK are starting to drop support for targeting versions
before 10.6.

Bug:  874997 
Change-Id: Id3ac6b42f49e0cdb28f85f127fb41305ba1af4cb
Reviewed-on: https://chromium-review.googlesource.com/1179748
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Hans Wennborg <hans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#584031}
[modify] https://crrev.com/046d539b96f85568341629d6a7166f62817de501/chrome/installer/gcapi_mac/BUILD.gn

The bot has made it past gcapi_example: https://ci.chromium.org/buildbot/chromium.clang/ToTMac/2017

Dropping 10.5 is fine. If any downstream consumers still install on 10.5 (doubtful), they can do their own build or, if using our example, hard-code their own 10.5 check or deal with failure to launch on 10.5 as an indication of an unsupported system. But I doubt anyone will notice or care.