[mac triage]

dba: Please let me know if you need my assistance configuring this bot. As a heads up, once this is working on the fyi waterfall, I'm going to want to image the main waterfall and official builders with this configuration.

Okay, getting pretty close to the point where I want to test hermetic builds on the waterfall. We already have a bot:

master:chromium.fyi
Builder: Chromium Mac 10.11 Force Mac Toolchain
vm: build9-m1

Preferred configuration: 
OS: macOS 10.12
installed binaries: git, svn
binaries definitely not installed: ld, bison, gperf, otool, dsymutil, strip, ibtool, libtool, lipo and strings

Trying to invoke an uninstalled binary should trigger the warning [which comes from the shims in /usr/bin/]: 
"""
xcode-select: note: no developer tools were found at '/Applications/Xcode.app', requesting install. Choose an option in the dialog to download the command line developer tools.
"""

Ideally, rather than have a binary blacklist, we'd have a binary whitelist. Pull in only the binaries need to deploy and manage bots. Maybe we can try deleting CommandLineTools after the bot has been configured and see if everything still works? At the least we'd need to manually install git/svn.

If we can't get macOS 10.12 because of issues with Go, then use macOS 10.11.6.

In the meanwhile, I've removed all the tools mentioned in c#3 from build9-m1. They've all been renamed to have a "_bak" suffix". 

This allows me to temporarily continue to work on mac hermetic toolchain. 

I'm assuming I need to redeploy build9-m1 now that I have a working 10.12 image?

As stated over VC previously, removing CommandLineTools will stop the buildbot process from starting, as it starts via a Makefile (it does this on all platforms IIRC), so the better bet is moving those binaries listed in #3 after deployment of the machine as 'make' isn't one of the binaries in the way.

Yup, please redeploy as 10.12. As per https://bugs.chromium.org/p/chromium/issues/detail?id=659213#c5, please redeploy the other bots on the FYI waterfall.

So I updated the bot, apparently the build slave started while i was working on it, and it killed 3 builds, but now it's just sitting there with pending build but connected and I have no clue why:

https://build.chromium.org/p/chromium.fyi/builders/Chromium%20Mac%2010.11%20Force%20Mac%20Toolchain

from twistd.log:

2016-11-15 13:51:07-0800 [Broker,client] SlaveBuilder.remote_print(Chromium Mac 10.11 Force Mac Toolchain): message from master: attached
2016-11-15 13:51:07-0800 [Broker,client] Connected to master1.golo.chromium.org:8111; slave is ready
2016-11-15 13:51:07-0800 [Broker,client] sending application-level keepalives every 300 seconds
2016-11-15 13:56:07-0800 [-] sending app-level keepalive

Maybe the master is now confused and needs a restart?

It finally picked up a job and is running a compile.

Tests look good. There are two errors:
https://build.chromium.org/p/chromium.fyi/builders/Chromium%20Mac%2010.11%20Force%20Mac%20Toolchain/builds/14681

nacl_integration is no longer run on Mac [but is still run on fyi waterfall], and browser_test failures look unrelated to hermetic build, though they might be related to 10.12.1.

sshed into the vm (build9-m1), and confirmed that it doesn't have any of the binaries from c#3 in CommandLineTools [other than bison. bison was a late addition to the binaries to remove, so perhaps it got missed?]

Could we get the default 10.12.1 image to not have bison as well? 

Otherwise the image looks good. I need to investigate the 10.12 related errors. https://bugs.chromium.org/p/chromium/issues/detail?id=665691

bison surely was too on the list (as I took the list from #3), and looks to be moved as expected:

chrome-bot@build9-m1:(Mac 10.12.1):~$ bison
bison: error: unable to find utility "bison", not a developer tool or in PATH
[72] chrome-bot@build9-m1:(Mac 10.12.1):~$ ls -l /Library/Developer/CommandLineTools/usr/bin/bison*
-rwxr-xr-x  1 root  admin  223744 Aug 28 19:33 /Library/Developer/CommandLineTools/usr/bin/bison.clt

You're right. There's also a directory called "bison" in CLT, which confused me.

"""
chrome-bot@build9-m1:(Mac 10.12.1):~$ find /Library/Developer/CommandLineTools -name bison
/Library/Developer/CommandLineTools/usr/share/bison
chrome-bot@build9-m1:(Mac 10.12.1):~$ /Library/Developer/CommandLineTools/usr/share/bison
-bash: /Library/Developer/CommandLineTools/usr/share/bison: is a directory
"""

ah ok.

I'm guessing we need the test failures worked out on this bot before we continue work with the FYI bots in bug 659213

10.12 images don't have Xcode, and the bootstrap process to provision a bot without Xcode exists with HERMETIC=1 env variable being exported prior to executing the labs mac bootstrap script.

10.12 images don't have Xcode, and the bootstrap process to provision a bot without Xcode exists with HERMETIC=1 env variable being exported prior to executing the labs mac bootstrap script.