New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 6 users

Issue metadata

Status: Assigned
Owner:
Cc:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 2
Type: Feature

Blocking:
issue 314403



Sign in to add a comment

Make OSX ninja build deterministic

Project Member Reported by mar...@chromium.org, Dec 20 2013

Issue description

1. ninja -C out/Release browser_tests
2. ren out out2
3. gclient runhooks
2. ninja -C out/Release browser_tests

Expected:
Content of out/ and out2/ is byte-for-byte identical.

Actual:
They aren't.

This is specific about OSX/ninja/clang
 

Comment 1 by mar...@chromium.org, Dec 20 2013

Labels: -Type-Bug Type-Feature

Comment 2 by mar...@chromium.org, Dec 20 2013

Blocking: chromium:314403

Comment 3 by mar...@chromium.org, Dec 20 2013

Labels: -Build-Toolchain

Comment 4 by mar...@chromium.org, Jun 11 2014

Summary: Make OSX ninja build deterministic (was: Mac OSX ninja build deterministic)
Project Member

Comment 5 by bugdroid1@chromium.org, Nov 5 2014

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/050a43d0b4a5da906b1484d63dd3d4aa1f4952d0

commit 050a43d0b4a5da906b1484d63dd3d4aa1f4952d0
Author: maruel <maruel@chromium.org>
Date: Wed Nov 05 22:23:50 2014

Roll gyp 1994->1998.

$ git log -4 --format="%ad %s" --date=short
2014-11-05 Set ZERO_AR_DATE=1 when running libtool.
2014-11-04 msvs: Fix rule dependencies for VS2013
2014-10-30 Switch to 2013 now that it's what's on the bots
2014-10-28 Add indentation offset guess for gyp emacs mode.

R=thakis@chromium.org,scottmg@chromium.org
BUG=330262

Review URL: https://codereview.chromium.org/705793002

Cr-Commit-Position: refs/heads/master@{#302887}

[modify] https://chromium.googlesource.com/chromium/src.git/+/050a43d0b4a5da906b1484d63dd3d4aa1f4952d0/DEPS

Since that patch, the timestamp of the libs is 0 when building for iOS.
This means every execution of ninja rebuild 326 targets even when no change is made.

My bug : 
https://code.google.com/p/chromium/issues/detail?id=432483
Project Member

Comment 7 by bugdroid1@chromium.org, Nov 12 2014

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fc35df0d4e8e32cb02206fa6640a327f4867edb2

commit fc35df0d4e8e32cb02206fa6640a327f4867edb2
Author: thakis <thakis@chromium.org>
Date: Wed Nov 12 18:26:54 2014

Revert of Roll gyp 1994->1998. (patchset #3 id:40001 of https://codereview.chromium.org/705793002/)

Reason for revert:
Caused http://crbug.com/432483

Original issue's description:
> Roll gyp 1994->1998.
>
> $ git log -4 --format="%ad %s" --date=short
> 2014-11-05 Set ZERO_AR_DATE=1 when running libtool.
> 2014-11-04 msvs: Fix rule dependencies for VS2013
> 2014-10-30 Switch to 2013 now that it's what's on the bots
> 2014-10-28 Add indentation offset guess for gyp emacs mode.
>
> R=thakis@chromium.org,scottmg@chromium.org
> BUG=330262
>
> Committed: https://crrev.com/050a43d0b4a5da906b1484d63dd3d4aa1f4952d0
> Cr-Commit-Position: refs/heads/master@{#302887}

TBR=scottmg@chromium.org,maruel@chromium.org
NOTREECHECKS=true
NOTRY=true
BUG=330262

Review URL: https://codereview.chromium.org/717243003

Cr-Commit-Position: refs/heads/master@{#303854}

[modify] https://chromium.googlesource.com/chromium/src.git/+/fc35df0d4e8e32cb02206fa6640a327f4867edb2/DEPS

Project Member

Comment 8 by bugdroid1@chromium.org, Jan 22 2015

------------------------------------------------------------------
r293756 | maruel@chromium.org | 2015-01-22T20:46:31.813565Z

Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/tools/build/scripts/slave/recipes/swarming/deterministic_build.py?r1=293756&r2=293755&pathrev=293756
   M http://src.chromium.org/viewvc/chrome/trunk/tools/build/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic_build.json?r1=293756&r2=293755&pathrev=293756

Reenable goma on OSX now that pnacl issues are fixed.

There are still known issue with remove_build_metadata being executed after
isolate.py check is being run within the compile step, this will be addressed in
a follow up.

R=sebmarchand@chromium.org
BUG=330262

Review URL: https://codereview.chromium.org/871603002
-----------------------------------------------------------------
Project Member

Comment 9 by bugdroid1@chromium.org, Mar 31 2015

------------------------------------------------------------------
r294581 | maruel@chromium.org | 2015-03-26T20:58:37.819093Z

Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/tools/build/scripts/slave/recipe_modules/isolate/resources/compare_build_artifacts.py?r1=294581&r2=294580&pathrev=294581

Create whitelist for 'Mac deterministic build'.

If there's no typo, this should make this builder green.

R=sebmarchand@chromium.org
BUG=330262

Review URL: https://codereview.chromium.org/1038103002
-----------------------------------------------------------------
Project Member

Comment 10 by bugdroid1@chromium.org, Oct 27 2015

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/3c62e8cc01d4fd6c54a892d3ce897e0bb304fe61

commit 3c62e8cc01d4fd6c54a892d3ce897e0bb304fe61
Author: maruel@chromium.org <maruel@chromium.org>
Date: Tue Oct 27 20:15:13 2015

Update regressions on OSX deterministic build.

R=sebmarchand@chromium.org
BUG=330262

Review URL: https://codereview.chromium.org/1419953004

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/build@297338 0039d316-1c4b-4281-b951-d872f2087c98

[modify] http://crrev.com/3c62e8cc01d4fd6c54a892d3ce897e0bb304fe61/scripts/slave/recipe_modules/isolate/resources/compare_build_artifacts.py

Project Member

Comment 11 by sheriffbot@chromium.org, Jul 6 2016

Labels: Hotlist-OpenBugWithCL
A change has landed for this issue, but it's been open for over 6 months. Please review and close it if applicable. If this issue should remain open, remove the "Hotlist-OpenBugWithCL" label. If no action is taken, it will be archived in 30 days.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Labels: -Hotlist-OpenBugWithCL
Cc: yyanagisawa@chromium.org
I sometimes see non-deterministic build in:
https://uberchromegw.corp.google.com/i/chromium.fyi/builders/Mac%20deterministic/builds/672/steps/compare_build_artifacts/logs/stdio

Since all object files seems to be the same even if executables are different, linker might do something bad?

ld64 (the mac linker) is known-knowndeterministic. It uses threads internally but doesn't synchronize them much, so whatever thread is done first writes output first as far as I understand. I guess a first step would be to file a rdar on bugreporter.apple.com.
Re: #14
filed with 28332953.
Note: how to reproduce non-deterministic build #15.
% gn gen --args="is_debug=false is_component_build=false" out/Default
% ninja -C out/Default base_unittests
% mv out/Default/base_unittests{,.0}
% ninja -C out/Default base_unittests
% mv out/Default/base_unittests{,.1}
% diff out/Default/base_unittests.{0,1}
yyanagisawa: did you get an update for apple bug 28332953?
No, no updates since last September.

16th Sep: filed the issue.
24th Sep: I was asked to give samples.
26th Sep: I have submitted the samples.
No updates since then.  Let me ping.
I just ping'ed yesterday, and got reply saying the issue should be fixed in the latest iOS beta.  I will try it later but just FYI.
You mean Xcode beta (?)
The original reply says iOS beta.  However, it sounds strange.  Let me update the issue to ask Xcode or iOS.
Project Member

Comment 22 by sheriffbot@chromium.org, May 7

Labels: Hotlist-Recharge-Cold
Status: Untriaged (was: Available)
This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Status: Available (was: Untriaged)
[mac triage]
yyanagisawa@, any update on this?
No more updates but Mac deterministic still shows non-deterministic.
https://uberchromegw.corp.google.com/i/chromium.fyi/builders/Mac%20deterministic

Is that because xcode used there is not updated?  Or, the issue is not fixed yet?
Comment from thakis:
"""
on mac ld used to be nondeterministic at some point. if that's still the case we might need lld on mac to make mac work (but there's interest on making that happen for other reasons anyway)
"""

Regardless, this needs further investigation.
Cc: mar...@chromium.org tikuta@chromium.org
Owner: erikc...@chromium.org
Status: Assigned (was: Available)
I've started looking into this. I've started with looking at libbase.dylib [component, debug build]. The build outputs appear semi-deterministic [there are a small number of possible outputs, but the chosen one is random].

I've attached two possible outputs. Using hexedit, we see that there are three differences:

1) LC_UUID load command [no surprise, this is probably a hash of the rest of the file]
2) sectname __stub_helper 
3) sectname __la_symbol_ptr  

Sections (2) and (3) are used for symbol stubs [la = lazy]. See https://stackoverflow.com/questions/8825537/mach-o-symbol-stubs-ios/8836580#8836580

There are ~30 stubs that have a different ordering in (2). The symbols in (3) are sorted alphabetically, but since (2) is in a different order, the pointers in (3) which point to (2) also get reordered.

Continuing to investigate.
libbase.dylib.zip
2.4 MB Download
libbase.dylib.zip
2.4 MB Download
Functionally, the two outputs are identical. To summarize:

* There's a stream of dyld commands located at LC_DYLD_INFO. This is the same for both dylibs. 
* Each symbol in (3) contains a pointer to a function in (2), which contains an index into LC_DYLD_INFO. 

The relative ordering of symbols in (2) does not affect behavior. And the only difference in outputs is caused by a difference of ordering in (2). 
I don't know the exact issue inside ld64, but it uses multi-threaded input file parsing, see: https://opensource.apple.com/source/ld64/ld64-274.2/src/ld/InputFiles.cpp.auto.html. There's no reasonable way to disable this, so who knows what kind of nondeterminism this causes. This leaves us with:

1) Find a way to get sysctl(ncpus) to return 1, without modifying the binary.
2) Fork ld64 and rebuild it ourselves with multi-threaded input disabled.
3) Use lld.

I'll investigate (3) first.
Slowing down linking by #1 and #2 seem also undesirable.

Could we file a feature request to have the parsed input files be sorted before linking them to the image?
Robert, would it in practice? I mean we generally are quite CPU / I/O bound during a build. I'm not sure the difference would be material with careful tuning, this would be worth measuring.
I think it'd depend a lot on the variables like hardware (particularly the disk type/speed) and number of input files. Agree it's something we should measure if possible.
I'm not sure that lld is a viable option in the short term. This llvm thread claims it's not actively maintained [as of a year ago]: https://reviews.llvm.org/D38290

Attempting to use it for Chromium gives these warnings:

ld64.lld: warning: ignoring unknown argument: -segprot
ld64.lld: warning: ignoring unknown argument: -ObjC
ld64.lld: warning: -sdk_version is required when emitting min version load command.  Setting sdk version to match provided min version
ld64.lld: error: Unable to find framework for -framework AppKit

Looking at the framework search code in MachOLinkingContext.cpp:
"""
 570 llvm::Optional<StringRef>                                                       
 571 MachOLinkingContext::findPathForFramework(StringRef fwName) const{              
 572   SmallString<256> fullPath;                                                    
 573   for (StringRef dir : frameworkDirs()) {                                       
 574     fullPath.assign(dir);                                                       
 575     llvm::sys::path::append(fullPath, Twine(fwName) + ".framework", fwName);                                                                                                                         
 576     if (fileExists(fullPath))                                                   
 577       return fullPath.str().copy(_allocator);                                   
 578   }                                                                             
 579                                                                                 
 580   return llvm::None;                                                            
 581 }      
"""

It's searching for sysroot/fwName.framework/fwName. In the 10.12 SDK that we link against, the relevant file that lld should be searching for is "MacOSX10.12.sdk/System/Library/Frameworks/ApplicationServices.framework/ApplicationServices.tbd". Note the ".tbd" extension, which reflects a new format for forward declaring symbols. Searching the lld source for "tbd" gives no hits. That's a pretty bad sign, and suggests we probably shouldn't be using lld yet for Mach-O binaries.

That matches with what I remember thakis@ saying when we last chatted about it. IIRC, lld is missing several features for linking Mach-O images, and he thought we should stick with ld64 as that is what Apple seem to be doing.
I tried to build ld64 from the open source tarball, but that was a mess. Couldn't get the dependencies configured correctly. Eventually I gave up and used the same port that Firefox uses: https://github.com/tpoechtrager/cctools-port.

I confirmed that turning off multi-threaded file input parsing does not fix this problem. Still investigating.
The ordering of the symbols in the __stub_helper section are determined by Layout::Comparer, see https://opensource.apple.com/source/ld64/ld64-264.3.102/src/ld/passes/order.cpp.auto.html.

They are first sorted by the dylib they come from, and then by name. Each dylib is given a monotonically increasing ordinal. The assignment of ordinals to dylibs is non-deterministic, which causes the bug.

The root problem is in InputFiles::createIndirectDylibs, see https://opensource.apple.com/source/ld64/ld64-264.3.102/src/ld/InputFiles.cpp.auto.html. The contents of the unprocessedDylibs vector are non-deterministic, as they rely on std::set iteration. This in turn influences the ordinals given to each dylib, which in turn affects the final symbol ordering.

A similar issue was reported in http://www.openradar.me/15002251. The solution there was to add a std::sort to InputFiles::dylibs. But that occurs too late for us -- the ordinals have already been set. 

Workarounds:
1) If we want to build ld64 ourselves, adding a std::sort to InputFiles::createIndirectDylibs fixes this issue.
2) Barring that, I believe that crawling the dylib dependency graph and explicitly specifying all dylib dependencies [which also gives the ordering] will also fix this problem.
(2) doesn't work for all use cases, since we need to specify the ordinals for all dylibs, but the interface for ld64 does not allow us to explicitly link against nested frameworks, e.g. /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/CommonPanels.framework/Versions/A/CommonPanels

So I think the best approach is to build ld64 with the fix ourselves.
I chatted with mark@, who strongly prefers that we try not to build ld64 on our own. Current plan is:

1) File a radar with patch.
2) Ping Apple to see if we can get them to push a fix. 
3) If they don't manage to get something going in a few weeks, reconsider rolling our own ld64.
> I chatted with mark@, who strongly prefers that we try not to build ld64 on our own.

Mostly because ld64 source releases lag what ships with Xcode, sometimes by a very long period. Xcode 9.4.1 is current, which ships with ld64-351.8 (this version first appeared in Xcode 9.3). But the latest ld64 source release is ld64-274.2 from Xcode 8.2, vintage 2016-12-12. This is over a year and a half old!
Given that we currently build with and against Xcode 8.3.2, it would be weird if we were linking with ld64 from Xcode 9.4.1. I suspect that this happens on dev devices mostly by accident, but our bots should be hermetic and using the older version of ld64.
> Given that we currently build with and against Xcode 8.3.2,
> it would be weird if we were linking with ld64 from Xcode
> 9.4.1

That's not the point. The point is that building our own ld64 would artificially hold us back on toolchain upgrades. Mix-and-match toolchains have proven themselves to be problems in the past.
We don't have the source code for Xcode 8.3.2's ld64-278.4 either. What linker bug fixes would we be giving up by downgrading to ld64-274.2? Nobody knows, because we haven't seen the source.
> The point is that building our own ld64 would artificially hold us back on toolchain upgrades.

1) If Apple fixes the issue in two weeks, this is a moot point.
2) If Apple takes 3 months to fix the issue, then using a custom build of ld64 for the interim 3 months allows us to make progress on deterministic builds, and then when we next roll the toolchain [which we don't do more than once a year anyways], we can switch back to their ld64. The difference between ld64-278.4 vs ld64-274.2 is likely minimal compared to the difference between ld64-278.4 and ld64-351.8.
3) If Apple never fixes the issue, then we need to think more carefully about how to best proceed. I think this option is unlikely.


I've started looking at next steps assuming this issue is fixed. For debug builds, the next issue is that ld64 embeds the file last-touched timestamp for OSOs in the LC_SYMTAB segment. This gets picked up in Options.cpp -- FileInfo::checkFileExists, and outputted in OutputFile::synthesizeDebugNotes -- objStab.value  = atomObjFile->debugInfoModificationTime();. 

Options to fix this are:
1) Change ld64. Easiest fix, but I believe mark would prefer we not do this.
2) Have our linker driver:
  2a) grab a timestamp of all relevant files prior to linking
  2b) override all timestamps with a dummy
  2c) link
  2d) replace timestamp
3) Write a utility to post-process the linker output to zero out the relevant entries in the LC_SYMTAB section.


Is that timestamp used for anything? ld probably doesn't just put it there for fun.

(On Windows for example, obj files contain mtimes used for link.exe's /incremental mode and not writing them makes incremental linking fail. ld might have some good reason too)
> Is that timestamp used for anything? 

Only by lldb.

1) The LC_SYMTAB segment isn't used to generate the LC_UUID, and it can be stripped via strip, so this definitely does not affect the final binary.
2) I don't see any code in xnu that uses the timestamp in N_OSO.
3) I don't see any code in ld64 that uses this timestamp [maybe I'm missing it?]
4) I confirmed that a custom version of ld64 that always writes "0" for the timestamp builds a valid binary that functions properly.
5) Loading this binary in lldb causes some error messages to appear, e.g.
"""
error: Chromium debug map object file '/Users/erikchen/projects/chromium/src/out/gn/obj/chrome/chrome_app_executable/chrome_exe_main_mac.o' has changed (actual time is 2018-07-30 11:16:46.000000000, debug map time is 1969-12-31 19:00:00.000000000) since this executable was linked, file will be ignored
"""

but this does not appear to affect basic functionality of lldb. backtrace still works. frame analysis still works, e.g.
"""
frame #12: 0x00000001043b4f88 libchrome_dll.dylib`::-[BrowserCrApplication nextEventMatchingMask:untilDate:inMode:dequeue:]() at chrome_browser_application_mac.mm:232
   229 	                           inMode:(NSString*)mode
   230 	                          dequeue:(BOOL)dequeue {
   231 	  __block NSEvent* event = nil;
-> 232 	  base::mac::CallWithEHFrame(^{
   233 	      event = [super nextEventMatchingMask:mask
   234 	                                 untilDate:expiration
   235 	                                    inMode:mode
"""

Looking at the source for lldb, the only place that oso_mod_time is used is in SymbolFileDWARFDebugMap::GetModuleByCompUnitInfo [source/Plugins/SymbolFile/DWARF/SymbolFileDWARFDebugMap.cpp]. A mismatch causes an early return:
"""
 428         if (oso_mod_time != comp_unit_info->oso_mod_time) {                                                                                                                     
 429           obj_file->GetModule()->ReportError(                                   
 430               "debug map object file '%s' has changed (actual time is "         
 431               "%s, debug map time is %s"                                        
 432               ") since this executable was linked, file will be ignored",       
 433               oso_file.GetPath().c_str(), llvm::to_string(oso_mod_time).c_str(),
 434               llvm::to_string(comp_unit_info->oso_mod_time).c_str());           
 435           return NULL;                                                          
 436         }  
"""

I do not know enough about lldb to know what changes this causes.
N_OSO is just debugger junk. This would only possibly be picked up by a debugger.

These timestamps are just there so that the debugger can kinda-sorta know whether an .o used for linking matches the .o that it’d load debug info from, if debug info is loaded from object files and not .dSYMs.
i think the Right Fix (tm) is to file another rdar for giving ld a "write hash in timestamp" flag. lldb already only compares equality not ordering, so lldb will be happy with that. Microsoft also made their timestamp fields mean "hash, really" for reproducible build reasons.

The short-term workaround is change timestamp in linker wrapper, link, change back (suggested by Erik)


I've achieved the MVP for local build determinism using the following steps and configuration:

Two changes to ld64:
  1) The fix listed in c#40
  2) Always write 0 for N_OSO timestamp

gn args:
"""
  3 use_goma = true                                                                 
  4 mac_views_browser = true                                                        
  5 strip_absolute_paths_from_debug_symbols = true                                  
  6 is_component_build = true                                                       
  7 is_debug = true                                                                 
  8 symbol_level = 1    
"""

I modified compare_build_artifacts.py to:
  * Also check .o, .a files
  * Also check non-executable files

Repro Steps:
1) Build all targets in out/gn
2) Move out/gn to out.11/gn
3) Build all targets in out/gn
4) Run compare_build_artifacts.py

Next steps:
* I've pinged the author of ld64. We'll see how quickly we can get traction on the fix in c#40
* I'll add a linker wrapper option to make deterministic timestamps for N_OSO. 
  * I'll file a feature request to put the hash in the N_OSO field.
* We'll want some stronger deterministic guarantees [e.g. checking that an empty commit doesn't change the output].
c#47: I filed "Add flag to ld64 to use hash instead of timestamp for N_OSO entries in LC_SYMTAB section."
  - https://bugreport.apple.com/web/?problemID=42737051

> I've pinged the author of ld64.
Patch has been accepted and committed. Hopefully the next Xcode roll will contain the fix. 
> The short-term workaround is change timestamp in linker wrapper, link, change back (suggested by Erik)

This doesn't work. There may be multiple simultaneous invocations of the linker wrapper. This causes a race condition:
1) linker_wrapper 1 sets mtime to 0
2) linker_wrapper 2 resets mtime to <original>
3) linker_wrapper 1 runs ld64.
Only reset timestamp if it's newer than the current one, that should work around that issue.

(It's really only a problem in component builds, right?)
Hm I suppose it fixes it for setting the right mtime back, but it doesn't help with ld reading an mtime of 0 off disk.
Make linker_wrapper modify the ld output directly, rather than messing around with the .o files on the filesystem?

Messing with the .o file timestamps sounds like it can get messy with parallelism and object file freshness. Seems fragile.
I implemented a python based mach-o parser to clear out the OSO, but it's way too slow.  It significantly increases the time needed to build all artifacts. 

At some point in the future, there will be a release of ld64 that both has the bug fixed, and has support for the environment variable ZERO_AR_DATE, which zeroes the mtime in OSO fields.

Here is what I'd like to do. 

Preference 1:
In the short term, use a custom build of ld64 based on https://github.com/tpoechtrager/cctools-port. This will have the bug fixed and have support for ZERO_AR_DATE, which requires a 6 line diff. 

This is the linker the Mozilla uses for Firefox, so it's seen some testing in the field. It's based on ld64 274.2, which is only slightly behind the current linker we're using -- ld64 305. c#42 appears inaccurate?

Update linker_driver to support the env variable ZERO_AR_DATE. When a new version of ld64 is released, we can switch straight to that.

Preference 2:
If we don't want to build ld64 ourselves, we won't have non-determinism no matter what we do. So we might as well wait for the new version of ld64.
For posterity, a link to the python Mach-o parser:
https://chromium-review.googlesource.com/c/chromium/src/+/1155660

I've confirmed that along with a build of ld64 with the non-determinism bug fixed, this creates deterministic builds as per c#48.
it looks like that reads the whole file into memory, modifies a bunch of headers at the very front, and writes it again. maybe it's faster if you mmap the file instead?
> This is the linker the Mozilla uses for Firefox, so it's seen some testing in the
> field. It's based on ld64 274.2, which is only slightly behind the current linker
> we're using -- ld64 305. c#42 appears inaccurate?

In that it’s further behind than I had thought?

You told me we were using Xcode 8.3.2 (#c39), which should be ld64-278.4. If we’re actually using ld64-305, that’s Xcode 9.2. So, to be clear, you’re proposing using the Xcode 8 linker with Xcode 9 tools and SDK? That’s folly, even more than I originally believed.

It’d be worth consideration if we were dealing with a stop-ship ld bug or severe developer unhappiness, but we’re talking about a low-priority bug that’s been on file for five years. There’s not a compelling enough case to take on the risk. Let’s wait for a totally-fixed ld64 binary release, or a compatible ld64 source release to patch.

> I implemented a python based mach-o parser to clear out the OSO, but it's way too
> slow.

This should be fast: http://crrev.com/c/1156626. Untested!
> In that it’s further behind than I had thought?

Whoops, my mistake. There are two ld binaries in each Xcode release. I was checking the version of ./Contents/Developer/usr/bin/ld, but that just redirects to the system install of ld. Checking ./Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld of Xcode 8.3.2 shows that it is indeed ld64-278.4. 

I've checked that ld64-305 [xcode 9] does not support 'ZERO_AR_DATE', but ld64-409.10 [released today in xcode 10 beta 5] does. ld64-409.10 does not have the non-determinism fix. 

mark: How do you feel about using a newer version of ld [e.g. with xcode 10 final release] with an older tools/SDK pairing [still xcode 8.3.2]
> mark: How do you feel about using a newer version of ld [e.g. with xcode 10 final
> release] with an older tools/SDK pairing [still xcode 8.3.2]

I feel generally better about that, but why wouldn’t we just use all of the Xcode 10 tools once there’s a final release, assuming it fixes all of our bugs? We were told that this kind of thing would be easy with the hermetic toolchain.
Rolling the toolchain is easy. But the toolchain has OS dependencies and rolling OSes on all relevant builders is still relatively difficult. There is a magical "handwavy" future world when Luci has been deployed everywhere where rolling OSes should also be easy [since we would have decoupled builders from specific machines].
Apple’s not going to promise that a tool that ships with Xcode 10 will run on anything older than 10.13. Will it? Maybe?
Given that Xcode 10 Beta 5 requires 10.13.6, and Xcode 9.3+ require 10.13.2, it seems unlikely that Xcode 10 final release will run on anything older than 10.13.

Sign in to add a comment