Right now, if .boto is not correctly configured, mac_toolchain.py emits an emit message that gets lost in the output of "gclient sync", which still appear to succeed. 

To address c#1, my in-progress CL already addresses that problem: https://codereview.chromium.org/2445993004/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7778e931dabc1fa85d0b77321e3fdef166e0b6a0

commit 7778e931dabc1fa85d0b77321e3fdef166e0b6a0
Author: erikchen <erikchen@chromium.org>
Date: Mon Nov 07 21:12:28 2016

Turn on hermetic toolchain for all corp machines.

BUG= 659726 

Review-Url: https://codereview.chromium.org/2445993004
Cr-Commit-Position: refs/heads/master@{#430376}

[modify] https://crrev.com/7778e931dabc1fa85d0b77321e3fdef166e0b6a0/.gn
[add] https://crrev.com/7778e931dabc1fa85d0b77321e3fdef166e0b6a0/build/mac/should_use_hermetic_xcode.py
[modify] https://crrev.com/7778e931dabc1fa85d0b77321e3fdef166e0b6a0/build/mac_toolchain.py
[modify] https://crrev.com/7778e931dabc1fa85d0b77321e3fdef166e0b6a0/build/toolchain/toolchain.gni

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e62f1a5d8886fc1086a180f4da94e0a3d1ca03dc

commit e62f1a5d8886fc1086a180f4da94e0a3d1ca03dc
Author: sdefresne <sdefresne@chromium.org>
Date: Tue Nov 08 09:10:50 2016

Revert of Turn on hermetic toolchain for all corp machines. (patchset #2 id:20001 of https://codereview.chromium.org/2445993004/ )

Reason for revert:
This break developer workflow for Googlers (as build/mac/should_use_hermetic_xcode.py returns 1) as the hermetic build does not contains support for iOS SDK, thus "gn gen" fails with the following error:

$ gn gen --args='target_os="ios"' out/default
ERROR at //build/config/ios/ios_sdk.gni:98:21: Script returned non-zero exit code.
  _ios_sdk_result = exec_script(script_name, ios_sdk_info_args, "scope")
                    ^----------
Current dir: /Users/sdefresne/Developer/chromium/src/out/default/
Command: python -- /Users/sdefresne/Developer/chromium/src/build/config/mac/sdk_info.py --developer_dir /Users/sdefresne/Developer/chromium/src/build/mac_files/Xcode.app iphonesimulator
Returned 1.
stderr:

xcodebuild: error: SDK "iphonesimulator" cannot be located.
xcodebuild: error: SDK "iphonesimulator" cannot be located.
xcrun: error: unable to lookup item 'Path' in SDK 'iphonesimulator'

The was not caught by the bots because they do not set FORCE_MAC_TOOLCHAIN and are not corporate machine.

Original issue's description:
> Turn on hermetic toolchain for all corp machines.
>
> BUG= 659726 
>
> Committed: https://crrev.com/7778e931dabc1fa85d0b77321e3fdef166e0b6a0
> Cr-Commit-Position: refs/heads/master@{#430376}

TBR=thakis@chromium.org,dpranke@chromium.org,brettw@chromium.org,justincohen@chromium.org,erikchen@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 659726 

Review-Url: https://codereview.chromium.org/2487643002
Cr-Commit-Position: refs/heads/master@{#430553}

[modify] https://crrev.com/e62f1a5d8886fc1086a180f4da94e0a3d1ca03dc/.gn
[delete] https://crrev.com/2168ba09b54ace301266f28072ce962982a4541d/build/mac/should_use_hermetic_xcode.py
[modify] https://crrev.com/e62f1a5d8886fc1086a180f4da94e0a3d1ca03dc/build/mac_toolchain.py
[modify] https://crrev.com/e62f1a5d8886fc1086a180f4da94e0a3d1ca03dc/build/toolchain/toolchain.gni

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/da01626aa3c2e1533c53c325de9bd579f0aa54d2

commit da01626aa3c2e1533c53c325de9bd579f0aa54d2
Author: erikchen <erikchen@chromium.org>
Date: Wed Nov 09 04:32:13 2016

Turn on hermetic toolchain for all corp machines.

BUG= 659726 

Committed: https://crrev.com/7778e931dabc1fa85d0b77321e3fdef166e0b6a0
Review-Url: https://codereview.chromium.org/2445993004
Cr-Original-Commit-Position: refs/heads/master@{#430376}
Cr-Commit-Position: refs/heads/master@{#430863}

[modify] https://crrev.com/da01626aa3c2e1533c53c325de9bd579f0aa54d2/.gn
[add] https://crrev.com/da01626aa3c2e1533c53c325de9bd579f0aa54d2/build/mac/should_use_hermetic_xcode.py
[modify] https://crrev.com/da01626aa3c2e1533c53c325de9bd579f0aa54d2/build/mac_toolchain.py
[modify] https://crrev.com/da01626aa3c2e1533c53c325de9bd579f0aa54d2/build/toolchain/toolchain.gni

This breaks WebRTC build on mac for Googlers. Can't we just turn this on when FORCE_MAC_TOOLCHAIN is set?

If this breaks webrtc, we should revert, fix, reland. What's the error?

Re #7: See https://bugs.chromium.org/p/webrtc/issues/detail?id=6700#c1

We need 10.12 SDK for WebRTC, but the hermetic toolchain only provides 10.10 and 10.12.
I'll revert https://codereview.chromium.org/2445993004 then.

I found a better way than reverting: to make it possible for projects reusing the Chromium toolchain to decide if they want to use the system Xcode instead. 
That needs a change like https://codereview.chromium.org/2494103002/ to land though.
Then a simple change like https://codereview.webrtc.org/2494113002/ solves the problem for WebRTC.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f1e2718a3ff89c80691a50f8ea2503cbb9aa97ee

commit f1e2718a3ff89c80691a50f8ea2503cbb9aa97ee
Author: kjellander <kjellander@chromium.org>
Date: Mon Nov 14 06:52:00 2016

Move use_system_xcode to //build_overrides/build.gni

This makes it possible for projects that reuse Chromium's build
toolchain to use the system Xcode by setting use_system_xcode=true
in //build_overrides/build.gni

BUG= webrtc:6700 , chromium:659726 
TBR=brettw@chromium.org

Review-Url: https://codereview.chromium.org/2494103002
Cr-Commit-Position: refs/heads/master@{#431838}

[modify] https://crrev.com/f1e2718a3ff89c80691a50f8ea2503cbb9aa97ee/.gn
[modify] https://crrev.com/f1e2718a3ff89c80691a50f8ea2503cbb9aa97ee/build/toolchain/toolchain.gni
[modify] https://crrev.com/f1e2718a3ff89c80691a50f8ea2503cbb9aa97ee/build_overrides/build.gni

With the CL in #10 above + a roll in WebRTC (https://codereview.webrtc.org/2496113002/) this is no longer a problem for WebRTC Googlers. We'd be happy to switch to the hermetic toolchain at some point in the future when it's providing the 10.12 SDK.

`gn gen out/Release` fails for me. :-/

git checkout origin/master
git rev-parse HEAD # 67c1fca10c8c50e94b77fb510e2709acd1bf8d6b
gclient sync

> gn gen out/Release
ERROR at //build/config/mac/mac_sdk.gni:61:5: Script returned non-zero exit code.
    exec_script("//build/mac/find_sdk.py", find_sdk_args, "list lines")
    ^----------
Current dir: /Users/lgarron/chromium/src/out/Release/
Command: python -- /Users/lgarron/chromium/src/build/mac/find_sdk.py --print_sdk_path --developer_dir /Users/lgarron/chromium/src/build/mac_files/Xcode.app 10.10
Returned 1.
stderr:

2016-11-15 17:36:53.129 xcodebuild[9453:120474] [MT] DVTAssertions: ASSERTION FAILURE in /SourceCache/DVTFrameworks/DVTFrameworks-5074/DVTFoundation/Utilities/DVTFoundationUtilities_MRR.m:398
Details:  Assertion failed: [aClass instanceMethodForSelector:observationInfoSEL] != [NSObject instanceMethodForSelector:observationInfoSEL]
Function: __DVTSetupKVODeallocAssertions_block_invoke158
Thread:   <NSThread: 0x7fae5dc0eec0>{number = 1, name = main}
Hints: None
Backtrace:
  0  0x0000000107ea3e09 -[DVTAssertionHandler handleFailureInFunction:fileName:lineNumber:assertionSignature:messageFormat:arguments:] (in DVTFoundation)
  1  0x0000000107ea359f _DVTAssertionHandler (in DVTFoundation)
  2  0x0000000107ea37ff _DVTAssertionFailureHandler (in DVTFoundation)
  3  0x0000000107ea3761 _DVTAssertionFailureHandler (in DVTFoundation)
  4  0x0000000107e3ea53 __DVTSetupKVODeallocAssertions_block_invoke158 (in DVTFoundation)
  5  0x0000000107e3f21d __DVTSetupKVODeallocAssertions_block_invoke_2196 (in DVTFoundation)
  6  0x00007fff9bc5f128 _dispatch_client_callout (in libdispatch.dylib)
  7  0x00007fff9bc5fd62 _dispatch_barrier_sync_f_invoke (in libdispatch.dylib)
  8  0x0000000107e3f12e __DVTSetupKVODeallocAssertions_block_invoke193 (in DVTFoundation)
  9  0x00007fff885f53ce -[__NSObserver _doit:] (in Foundation)
 10  0x00007fff86be059c __CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__ (in CoreFoundation)
 11  0x00007fff86be049b _CFXRegistrationPost (in CoreFoundation)
 12  0x00007fff86be0202 ___CFXNotificationPost_block_invoke (in CoreFoundation)
 13  0x00007fff86b9d903 -[_CFXNotificationRegistrar find:object:observer:enumerator:] (in CoreFoundation)
 14  0x00007fff86b9c93c _CFXNotificationPost (in CoreFoundation)
 15  0x00007fff885cbc8d -[NSBundle loadAndReturnError:] (in Foundation)
 16  0x000000010977bc10 -[XCPluginManager loadPluginBundle:] (in DevToolsCore)
 17  0x000000010977b617 -[XCPluginManager findAndLoadPluginsInDomain:] (in DevToolsCore)
 18  0x000000010973ea88 XCInitializeCoreIfNeeded (in DevToolsCore)
 19  0x000000010896f270 +[Xcode3CoreInitializer ide_initializeWithOptions:error:] (in Xcode3Core)
 20  0x00000001083f2989 _IDEInitializeOnePlugInAndPrerequisites (in IDEFoundation)
 21  0x00000001083f2066 _IDEInitializePlugIns (in IDEFoundation)
 22  0x00000001083f1799 IDEInitialize (in IDEFoundation)
 23  0x0000000108988b9f -[Xcode3CommandLineBuildTool run] (in Xcode3Core)
 24  0x0000000107d0d98a (in xcodebuild)
 25  0x00007fff9bc95255 start (in libdyld.dylib)
2016-11-15 17:36:53.306 xcodebuild[9454:120490] [MT] DVTAssertions: ASSERTION FAILURE in /SourceCache/DVTFrameworks/DVTFrameworks-5074/DVTFoundation/Utilities/DVTFoundationUtilities_MRR.m:398
Details:  Assertion failed: [aClass instanceMethodForSelector:observationInfoSEL] != [NSObject instanceMethodForSelector:observationInfoSEL]
Function: __DVTSetupKVODeallocAssertions_block_invoke158
Thread:   <NSThread: 0x7ff96b50ec30>{number = 1, name = main}
Hints: None
Backtrace:
  0  0x0000000108b8fe09 -[DVTAssertionHandler handleFailureInFunction:fileName:lineNumber:assertionSignature:messageFormat:arguments:] (in DVTFoundation)
  1  0x0000000108b8f59f _DVTAssertionHandler (in DVTFoundation)
  2  0x0000000108b8f7ff _DVTAssertionFailureHandler (in DVTFoundation)
  3  0x0000000108b8f761 _DVTAssertionFailureHandler (in DVTFoundation)
  4  0x0000000108b2aa53 __DVTSetupKVODeallocAssertions_block_invoke158 (in DVTFoundation)
  5  0x0000000108b2b21d __DVTSetupKVODeallocAssertions_block_invoke_2196 (in DVTFoundation)
  6  0x00007fff9bc5f128 _dispatch_client_callout (in libdispatch.dylib)
  7  0x00007fff9bc5fd62 _dispatch_barrier_sync_f_invoke (in libdispatch.dylib)
  8  0x0000000108b2b12e __DVTSetupKVODeallocAssertions_block_invoke193 (in DVTFoundation)
  9  0x00007fff885f53ce -[__NSObserver _doit:] (in Foundation)
 10  0x00007fff86be059c __CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__ (in CoreFoundation)
 11  0x00007fff86be049b _CFXRegistrationPost (in CoreFoundation)
 12  0x00007fff86be0202 ___CFXNotificationPost_block_invoke (in CoreFoundation)
 13  0x00007fff86b9d903 -[_CFXNotificationRegistrar find:object:observer:enumerator:] (in CoreFoundation)
 14  0x00007fff86b9c93c _CFXNotificationPost (in CoreFoundation)
 15  0x00007fff885cbc8d -[NSBundle loadAndReturnError:] (in Foundation)
 16  0x000000010a46fc10 -[XCPluginManager loadPluginBundle:] (in DevToolsCore)
 17  0x000000010a46f617 -[XCPluginManager findAndLoadPluginsInDomain:] (in DevToolsCore)
 18  0x000000010a432a88 XCInitializeCoreIfNeeded (in DevToolsCore)
 19  0x0000000109660270 +[Xcode3CoreInitializer ide_initializeWithOptions:error:] (in Xcode3Core)
 20  0x00000001090de989 _IDEInitializeOnePlugInAndPrerequisites (in IDEFoundation)
 21  0x00000001090de066 _IDEInitializePlugIns (in IDEFoundation)
 22  0x00000001090dd799 IDEInitialize (in IDEFoundation)
 23  0x0000000109679b9f -[Xcode3CommandLineBuildTool run] (in Xcode3Core)
 24  0x00000001089f298a (in xcodebuild)
 25  0x00007fff9bc95255 start (in libdyld.dylib)
 26  0x0000000000000005
xcrun: error: unable to lookup item 'Path' in SDK 'macosx10.10'
Traceback (most recent call last):
  File "/Users/lgarron/chromium/src/build/mac/find_sdk.py", line 89, in <module>
    print main()
  File "/Users/lgarron/chromium/src/build/mac/find_sdk.py", line 81, in main
    ['xcrun', '-sdk', 'macosx' + best_sdk, '--show-sdk-path']).strip()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['xcrun', '-sdk', 'macosx10.10', '--show-sdk-path']' returned non-zero exit status 1

See //build/toolchain/mac/BUILD.gn:14:1: whence it was imported.
import("//build/config/mac/mac_sdk.gni")
^--------------------------------------
See //BUILD.gn:74:1: which caused the file to be included.
group("gn_all") {
^----------------

I'm on macOS Sierra 10.12.1 (16B2555).

I also can't turn off hermetic builds using $FORCE_MAC_TOOLCHAIN because _IsCorpMachine() is true for me.

https://cs.chromium.org/chromium/src/build/mac/should_use_hermetic_xcode.py?q=FORCE_MAC_TOOLCHAIN&sq=package:chromium&l=18&dr=C

For now, I'm hacking around it by forcing should_use_hermetic_xcode.py to return the value 0.

Normally, I'd revert first and ask questions later, but tha twould force a full rebuild on all developers, which I'd like to avoid. Investigating...

No worries. If you don't find an easy fix, would you mind allowing an environment variable to override `_IsCorpMachine()` so that I can work around it without having to modify should_use_hermetic_xcode.py in my checkout?

On lgarron's machine:
Tried running the same xcrun command on its own - same error.
Tried running the same xcrun command with DEVELOPER_DIR set to the hermetic toolchain. 
error:
"""
xcrun: error: unable to find Xcode installation from active developer path "...src/build/mac_files/Xcode.app", use xcode-select to change
"""

Tried running the same xcrun command with DEVELOPER_DIR set to the hermetic toolchain and using a direct path to xcrun [from hermetic toolchain]. Same error.

Observations: 
1) DEVELOPER_DIR doesn't affect xcrun, which uses the system install at /usr/bin/xcrun!
2) The current DEVELOPER_DIR path is slightly inaccurate. It should be: ..../build/mac_files/Xcode.app/Contents/Developer


Searching on the internet for sourcecache DVTFrameworks shows people with similar looking errors in totally unrelated contexts. This suggests that the problem is not strictly with hermetic build itself. The oldest couple of frames look interesting:
"""
 21  0x00000001083f2066 _IDEInitializePlugIns (in IDEFoundation)
 22  0x00000001083f1799 IDEInitialize (in IDEFoundation)
 23  0x0000000108988b9f -[Xcode3CommandLineBuildTool run] (in Xcode3Core)
 24  0x0000000107d0d98a (in xcodebuild)
 25  0x00007fff9bc95255 start (in libdyld.dylib)
"""
Somehow...xcrun calls xcodebuild, which then tries to initialize plugins?

lgarron@, do you have anything in the Xcode plugins directory, e.g.:
  ~/Library/Application\ Support/Developer/Shared/Xcode/Plug-ins

If so, I wonder if this problem will go away when we roll hermetic to the latest Xcode, which does plugins differently.

Yes, I do. Just one:
~/Library/Application Support/Developer/Shared/Xcode/Plug-ins/ClangFormat.xcplugin

Removing the plugin works!
I don't know if I ever needed it for anything, but I can keep it deleted for the time being.

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/868abcb08592e5fc1ac0d1a1dcc74fb6d000519f

commit 868abcb08592e5fc1ac0d1a1dcc74fb6d000519f
Author: machenbach <machenbach@chromium.org>
Date: Tue Nov 29 11:16:41 2016

[build] Roll build a3b623a:11a223f

This also ports some build overrides including:
https://codereview.chromium.org/2512043002/

BUG= chromium:659726 

Review-Url: https://codereview.chromium.org/2532043002
Cr-Commit-Position: refs/heads/master@{#41347}

[modify] https://crrev.com/868abcb08592e5fc1ac0d1a1dcc74fb6d000519f/.gn
[modify] https://crrev.com/868abcb08592e5fc1ac0d1a1dcc74fb6d000519f/DEPS
[modify] https://crrev.com/868abcb08592e5fc1ac0d1a1dcc74fb6d000519f/build_overrides/build.gni

The following revision refers to this bug:
  https://chromium.googlesource.com/external/webrtc.git/+/ec2c93e9479c3c6b8abffa2252a15475ab75f6dd

commit ec2c93e9479c3c6b8abffa2252a15475ab75f6dd
Author: kjellander <kjellander@webrtc.org>
Date: Fri Sep 08 17:52:17 2017

Use bundled SDK 10.12 toolchain for Mac and iOS.

BUG= webrtc:6608 ,  webrtc:8099 ,  chromium:659726 ,  chromium:763440 
TBR=mbonadei@webrtc.org
NOTRY=True

Review-Url: https://codereview.webrtc.org/2995033002
Cr-Commit-Position: refs/heads/master@{#19753}

[modify] https://crrev.com/ec2c93e9479c3c6b8abffa2252a15475ab75f6dd/.gn
[modify] https://crrev.com/ec2c93e9479c3c6b8abffa2252a15475ab75f6dd/DEPS
[modify] https://crrev.com/ec2c93e9479c3c6b8abffa2252a15475ab75f6dd/build_overrides/build.gni