Another example: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/174397

FAILED: components_perftests 
export DEVELOPER_DIR=/b/s/w/ir/cache/builder/src/build/mac_files/Xcode.app;  TOOL_VERSION=1540420510 ../../build/toolchain/mac/linker_driver.py ../../third_party/llvm-build/Release+Asserts/bin/clang++  -stdlib=libc++ -arch x86_64 -segprot PROTECTED_MEMORY rw r -Werror -isysroot ../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -mmacosx-version-min=10.9.0 -Wl,-ObjC -L../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/lib -o "./components_perftests" -Wl,-filelist,"./components_perftests.rsp"  -framework Cocoa -framework Foundation -framework IOKit -framework Security -framework SystemConfiguration -framework ApplicationServices -framework AppKit -lbsm -framework CoreFoundation -framework CoreGraphics -framework CoreText -framework CFNetwork -framework CoreServices -lresolv -framework Accelerate -framework QuartzCore -framework AudioUnit -framework Carbon -framework CoreVideo -framework IOSurface -framework DiskArbitration -framework OpenGL -framework Quartz -framework AVFoundation -framework CoreMedia -framework AudioToolbox -framework CoreAudio -framework VideoToolbox -lsandbox -framework IOBluetooth -framework CoreMIDI -framework LocalAuthentication -framework ForceFeedback -framework GameController -framework CoreWLAN -lcups
Undefined symbols for architecture x86_64:
  "content::GuestMode::IsCrossProcessFrameGuest(content::WebContents*)", referenced from:
      content::WebContentsViewGuest::WebContentsViewGuest(content::WebContentsImpl*, content::BrowserPluginGuest*, std::__1::unique_ptr<content::WebContentsView, std::__1::default_delete<content::WebContentsView> >, content::RenderViewHostDelegateView**) in web_contents_view_guest.o
      content::WebContentsViewGuest::WebContentsViewGuest(content::WebContentsImpl*, content::BrowserPluginGuest*, std::__1::unique_ptr<content::WebContentsView, std::__1::default_delete<content::WebContentsView> >, content::RenderViewHostDelegateView**) in web_contents_view_guest.o
ld: symbol(s) not found for architecture x86_64

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/174095 is the earliest build with linking error that I could find.

Speculative revert in progress: https://chromium-review.googlesource.com/c/chromium/src/+/1309224

Chatted a bit with dpranke@:
https://chromium-swarm.appspot.com/task?id=40d7027870869d10&refresh=10 is the first build post-ICU-roll on vm645-m1, and it has the linking failure. This corresponds to https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/172603.

The error message in https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/172603. 

Show

[19760/21732] SOLINK 'obj/chrome/chrome_framework_shared_library/Chromium Framework' 'obj/chrome/chrome_framework_shared_library/Chromium Framework.TOC'
FAILED: obj/chrome/chrome_framework_shared_library/Chromium Framework obj/chrome/chrome_framework_shared_library/Chromium Framework.TOC 
export DEVELOPER_DIR=/b/s/w/ir/cache/builder/src/build/mac_files/Xcode.app;  if [ ! -e "obj/chrome/chrome_framework_shared_library/Chromium Framework" -o ! -e "obj/chrome/chrome_framework_shared_library/Chromium Framework.TOC" ] || otool -l "obj/chrome/chrome_framework_shared_library/Chromium Framework" | grep -q LC_REEXPORT_DYLIB ; then TOOL_VERSION=1540419883 ../../build/toolchain/mac/linker_driver.py ../../third_party/llvm-build/Release+Asserts/bin/clang++ -shared -Wl,-install_name,@executable_path/../Versions/72.0.3596.0/Chromium\ Framework.framework/Chromium\ Framework -compatibility_version 3596.0.0 -current_version 3596.0.0 -Wl,-order_file,../../chrome/app/framework.order -stdlib=libc++ -arch x86_64 -segprot PROTECTED_MEMORY rw r -Werror -isysroot ../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -mmacosx-version-min=10.9.0 -Wl,-ObjC -L../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/lib -o "obj/chrome/chrome_framework_shared_library/Chromium Framework" -Wl,-filelist,"obj/chrome/chrome_framework_shared_library/Chromium Framework.rsp" -framework Cocoa -framework Foundation -framework IOKit -framework Security -framework SystemConfiguration -framework Accelerate -framework AudioUnit -framework DiskArbitration -framework ImageCaptureCore -framework OpenGL -framework QuartzCore -framework SecurityInterface -lcups -framework ApplicationServices -framework AppKit -lbsm -framework CoreFoundation -framework Carbon -framework CoreVideo -framework CoreGraphics -framework CoreText -framework IOSurface -framework CFNetwork -framework CoreServices -lresolv -framework IOBluetooth -framework AudioToolbox -framework CoreAudio -framework Quartz -framework AVFoundation -framework CoreMedia -framework VideoToolbox -lsandbox -framework CoreMIDI -framework OpenDirectory -framework LocalAuthentication -framework ForceFeedback -framework GameController -framework CoreWLAN -lbz2  && { otool -l "obj/chrome/chrome_framework_shared_library/Chromium Framework" | grep LC_ID_DYLIB -A 5; nm -gP "obj/chrome/chrome_framework_shared_library/Chromium Framework" | cut -f1-2 -d' ' | grep -v U$$; true; } > "obj/chrome/chrome_framework_shared_library/Chromium Framework.TOC"; else TOOL_VERSION=1540419883 ../../build/toolchain/mac/linker_driver.py ../../third_party/llvm-build/Release+Asserts/bin/clang++ -shared -Wl,-install_name,@executable_path/../Versions/72.0.3596.0/Chromium\ Framework.framework/Chromium\ Framework -compatibility_version 3596.0.0 -current_version 3596.0.0 -Wl,-order_file,../../chrome/app/framework.order -stdlib=libc++ -arch x86_64 -segprot PROTECTED_MEMORY rw r -Werror -isysroot ../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -mmacosx-version-min=10.9.0 -Wl,-ObjC -L../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/lib -o "obj/chrome/chrome_framework_shared_library/Chromium Framework" -Wl,-filelist,"obj/chrome/chrome_framework_shared_library/Chromium Framework.rsp" -framework Cocoa -framework Foundation -framework IOKit -framework Security -framework SystemConfiguration -framework Accelerate -framework AudioUnit -framework DiskArbitration -framework ImageCaptureCore -framework OpenGL -framework QuartzCore -framework SecurityInterface -lcups -framework ApplicationServices -framework AppKit -lbsm -framework CoreFoundation -framework Carbon -framework CoreVideo -framework CoreGraphics -framework CoreText -framework IOSurface -framework CFNetwork -framework CoreServices -lresolv -framework IOBluetooth -framework AudioToolbox -framework CoreAudio -framework Quartz -framework AVFoundation -framework CoreMedia -framework VideoToolbox -lsandbox -framework CoreMIDI -framework OpenDirectory -framework LocalAuthentication -framework ForceFeedback -framework GameController -framework CoreWLAN -lbz2  && { otool -l "obj/chrome/chrome_framework_shared_library/Chromium Framework" | grep LC_ID_DYLIB -A 5; nm -gP "obj/chrome/chrome_framework_shared_library/Chromium Framework" | cut -f1-2 -d' ' | grep -v U$$; true; } > "obj/chrome/chrome_framework_shared_library/Chromium Framework.tmp" && if ! cmp -s "obj/chrome/chrome_framework_shared_library/Chromium Framework.tmp" "obj/chrome/chrome_framework_shared_library/Chromium Framework.TOC"; then mv "obj/chrome/chrome_framework_shared_library/Chromium Framework.tmp" "obj/chrome/chrome_framework_shared_library/Chromium Framework.TOC" ; fi; fi
Undefined symbols for architecture x86_64:
  "icu_62::Locale::Locale(char const*, char const*, char const*, char const*)", referenced from:
      media_router::MediaRouterUIBase::InitCommon(content::WebContents*) in libui.a(media_router_ui_base.o)
  "icu_62::Locale::~Locale()", referenced from:
      media_router::MediaRouterUIBase::InitCommon(content::WebContents*) in libui.a(media_router_ui_base.o)
  "icu_62::Collator::createInstance(icu_62::Locale const&, UErrorCode&)", referenced from:
      media_router::MediaRouterUIBase::InitCommon(content::WebContents*) in libui.a(media_router_ui_base.o)
  "media_router::MediaSink::CompareUsingCollator(media_router::MediaSink const&, icu_62::Collator const*) const", referenced from:
      void std::__1::__sort<media_router::MediaRouterUIBase::OnResultsUpdated(std::__1::vector<media_router::MediaSinkWithCastModes, std::__1::allocator<media_router::MediaSinkWithCastModes> > const&)::$_1&, media_router::MediaSinkWithCastModes*>(media_router::MediaSinkWithCastModes*, media_router::MediaSinkWithCastModes*, media_router::MediaRouterUIBase::OnResultsUpdated(std::__1::vector<media_router::MediaSinkWithCastModes, std::__1::allocator<media_router::MediaSinkWithCastModes> > const&)::$_1&) in libui.a(media_router_ui_base.o)
      unsigned int std::__1::__sort3<media_router::MediaRouterUIBase::OnResultsUpdated(std::__1::vector<media_router::MediaSinkWithCastModes, std::__1::allocator<media_router::MediaSinkWithCastModes> > const&)::$_1&, media_router::MediaSinkWithCastModes*>(media_router::MediaSinkWithCastModes*, media_router::MediaSinkWithCastModes*, media_router::MediaSinkWithCastModes*, media_router::MediaRouterUIBase::OnResultsUpdated(std::__1::vector<media_router::MediaSinkWithCastModes, std::__1::allocator<media_router::MediaSinkWithCastModes> > const&)::$_1&) in libui.a(media_router_ui_base.o)
      unsigned int std::__1::__sort4<media_router::MediaRouterUIBase::OnResultsUpdated(std::__1::vector<media_router::MediaSinkWithCastModes, std::__1::allocator<media_router::MediaSinkWithCastModes> > const&)::$_1&, media_router::MediaSinkWithCastModes*>(media_router::MediaSinkWithCastModes*, media_router::MediaSinkWithCastModes*, media_router::MediaSinkWithCastModes*, media_router::MediaSinkWithCastModes*, media_router::MediaRouterUIBase::OnResultsUpdated(std::__1::vector<media_router::MediaSinkWithCastModes, std::__1::allocator<media_router::MediaSinkWithCastModes> > const&)::$_1&) in libui.a(media_router_ui_base.o)
      unsigned int std::__1::__sort5<media_router::MediaRouterUIBase::OnResultsUpdated(std::__1::vector<media_router::MediaSinkWithCastModes, std::__1::allocator<media_router::MediaSinkWithCastModes> > const&)::$_1&, media_router::MediaSinkWithCastModes*>(media_router::MediaSinkWithCastModes*, media_router::MediaSinkWithCastModes*, media_router::MediaSinkWithCastModes*, media_router::MediaSinkWithCastModes*, media_router::MediaSinkWithCastModes*, media_router::MediaRouterUIBase::OnResultsUpdated(std::__1::vector<media_router::MediaSinkWithCastModes, std::__1::allocator<media_router::MediaSinkWithCastModes> > const&)::$_1&) in libui.a(media_router_ui_base.o)
      bool std::__1::__insertion_sort_incomplete<media_router::MediaRouterUIBase::OnResultsUpdated(std::__1::vector<media_router::MediaSinkWithCastModes, std::__1::allocator<media_router::MediaSinkWithCastModes> > const&)::$_1&, media_router::MediaSinkWithCastModes*>(media_router::MediaSinkWithCastModes*, media_router::MediaSinkWithCastModes*, media_router::MediaRouterUIBase::OnResultsUpdated(std::__1::vector<media_router::MediaSinkWithCastModes, std::__1::allocator<media_router::MediaSinkWithCastModes> > const&)::$_1&) in libui.a(media_router_ui_base.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Traceback (most recent call last):
  File "../../build/toolchain/mac/linker_driver.py", line 229, in <module>
    Main(sys.argv)
  File "../../build/toolchain/mac/linker_driver.py", line 79, in Main
    subprocess.check_call(compiler_driver_args)
  File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['../../third_party/llvm-build/Release+Asserts/bin/clang++', '-shared', '-Wl,-install_name,@executable_path/../Versions/72.0.3596.0/Chromium Framework.framework/Chromium Framework', '-compatibility_version', '3596.0.0', '-current_version', '3596.0.0', '-Wl,-order_file,../../chrome/app/framework.order', '-stdlib=libc++', '-arch', 'x86_64', '-segprot', 'PROTECTED_MEMORY', 'rw', 'r', '-Werror', '-isysroot', '../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk', '-mmacosx-version-min=10.9.0', '-Wl,-ObjC', '-L../../build/mac_files/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/lib', '-o', 'obj/chrome/chrome_framework_shared_library/Chromium Framework', '-Wl,-filelist,obj/chrome/chrome_framework_shared_library/Chromium Framework.rsp', '-framework', 'Cocoa', '-framework', 'Foundation', '-framework', 'IOKit', '-framework', 'Security', '-framework', 'SystemConfiguration', '-framework', 'Accelerate', '-framework', 'AudioUnit', '-framework', 'DiskArbitration', '-framework', 'ImageCaptureCore', '-framework', 'OpenGL', '-framework', 'QuartzCore', '-framework', 'SecurityInterface', '-lcups', '-framework', 'ApplicationServices', '-framework', 'AppKit', '-lbsm', '-framework', 'CoreFoundation', '-framework', 'Carbon', '-framework', 'CoreVideo', '-framework', 'CoreGraphics', '-framework', 'CoreText', '-framework', 'IOSurface', '-framework', 'CFNetwork', '-framework', 'CoreServices', '-lresolv', '-framework', 'IOBluetooth', '-framework', 'AudioToolbox', '-framework', 'CoreAudio', '-framework', 'Quartz', '-framework', 'AVFoundation', '-framework', 'CoreMedia', '-framework', 'VideoToolbox', '-lsandbox', '-framework', 'CoreMIDI', '-framework', 'OpenDirectory', '-framework', 'LocalAuthentication', '-framework', 'ForceFeedback', '-framework', 'GameController', '-framework', 'CoreWLAN', '-lbz2']' returned non-zero exit status 1
[19761/21732] LINK ./components_perftests
[19762/21732] SOLINK 'obj/content/shell/content_shell_framework_shared_library/Content Shell Framework' 'obj/content/shell/content_shell_framework_shared_library/Content Shell Framework.TOC'
[19763/21732] LINK newlib_pnacl/remoting_client_plugin_newlib.pexe
ninja: build stopped: subcommand failed.
step returned non-zero exit code: 1


Notice it is looking for a icu62 symbol (not icu63) so it is possible that there is a dep issue, that the compiled object is compiled under icu62, and then somehow didn't got clean up and link against icu63.

This isn’t running on Buildbot

Per  conversation at  https://chromium-review.googlesource.com/c/chromium/src/+/1309224,  ICU 63 roll is not responsible for this failure. 

I spent quite a bit of time poking into this, along with help from dcheng@, tbansal, horo@, and thakis@.

The investigation of the audio_worklet_object_proxy.o failure led me to ssh'ing into vm257-m4 (where the failure was happening), and from looking at things on that machine, my best guess is that the .ninja_deps file was corrupted, leading ninja to not rebuild the file when it should've. I do not know how it would've gotten into that state. After  hand-rebuilding things, it appears to have started working again.

The icu failures look similar, as do a couple of other classes of compile failure we were seeing repeatedly.

This led me to hand-clobber the following bots:

vm257-m4 (sort of, didn't really clobber it, but I think it may have at least been partially repaired)
vm645-m1 (build failure in 174707)
vm696-m1 (build failure in 174706)
vm1098-m4 (build failure in 174695)

The other machines all appeared to have recently compiled successfully. If we see any build failures after build 174707, we may need to do more repair work.

This did not determine a root cause, so whatever happened will likely happen again.

A quick glance at the most recent 200 failures suggests that I repaired the machines that were broken, and we're in the clear now.

Seeing some occurrences of this again:
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/176235
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/176289
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/176280

I guess I'll start clobbering.

Very strange. Given that we haven't changed ninja or clang recently, I'm at a bit of a loss for why we'd be seeing this now.

Found a couple bad apples:
https://chromium-swarm.appspot.com/bot?id=vm690-m4
https://chromium-swarm.appspot.com/bot?id=vm279-m4

I'll conjure up a `swarming.py trigger` invocation that we can use to clear out the builder caches on a given bot. It's a bit more targeted than a landmine, and easier to use than ssh'ing.

I'll try leaving a bot around for debugging, but I doubt I'll be able to uncover anything new.

Since this is a linker error and we use Xcode's linker: Is someone updating Xcode atm? Is it possible that different bots use different Xcodes? I don't know how we deploy Xcode to bots, does it use the hermetic downloader, or cipd, or, ...?

To clobber a bot:
`
swarming.py trigger --priority=1 --swarming https://chromium-swarm.appspot.com --dimension pool $POOL --dimension id $BOT --named-cache $NAMED_CACHE cache --raw-cmd -- /bin/bash -c 'rm -rfv cache/src/out/*'
`
Where $POOL and $BOT is the bot's pool and bot id, and $NAMED_CACHE is the name of the builder cache that shows up in its dimensions (should look like "builder_511547cfb341341b90e7c0a77ea9dbd71f29f8bf0284645aa58086da56a15ab4_v2")

> Is someone updating Xcode atm?

I can't say for sure, but it doesn't appear so:
https://chromium.googlesource.com/chromium/src/+log/HEAD/build/mac_toolchain.py

If I read that link right the the last commit mostly deletes all that code and make it forward to a cipd go binary whose source is elsewhere. (Maybe the xcode hash is still in the python file, didn't check.)

I was assuming it was controlled via the MAC_TOOLCHAIN_VERSION global var at the top of that script, but I'll need someone more familiar w/ it to confirm.

I have not seen this problem in the last 200 builds of mac_chromium_rel_ng so I'm removing it from the sheriff queue. Feel free to re-add it if that is a mistake, I've not followed the discussions.

Re #c19-21: Xcode is indeed installed by the hermetic script, and its version is controlled by MAC_TOOLCHAIN_VERSION. The Go binary is somewhat irrelevant, since it only does what the script asks it to do. So, no, the Xcode version hasn't changed lately.

Haven't seen the errors recently. I'll check back in a week and close this if they haven't reemerged since.

The NextAction date has arrived: 2018-11-14

Don't think this has reoccurred since.