I currently can't ssh to my Mac for some reason, so I can't investigate at the moment.

First bad 316206, last good 316203

$ svn log -r316204:316206 https://nico@llvm.org/svn/llvm-project/
------------------------------------------------------------------------
r316204 | compnerd | 2017-10-20 00:11:28 -0400 (Fri, 20 Oct 2017) | 6 lines

Basic: restore {,u}intptr_t on NetBSD/ARM

NetBSD uses `long int` for `intptr_t` on ARM.  This was changed in SVN
r316046, referenced against other compilers.  However, NetBSD's
reference was incorrect as the current clang behaviour is more
up-to-date.  Restore the original behaviour for that target.
------------------------------------------------------------------------
r316205 | dylanmckay | 2017-10-20 00:17:14 -0400 (Fri, 20 Oct 2017) | 1 line

[AVR] Fix the select-mbb-placement-bug.ll
------------------------------------------------------------------------
r316206 | vitalybuka | 2017-10-20 01:44:12 -0400 (Fri, 20 Oct 2017) | 1 line

[zorg] Try to combine fuzzer and autoconf bots on faster hardware
------------------------------------------------------------------------


None of that looks very related.

Maybe due to  https://chromium-review.googlesource.com/725080 -- we might now build more code, and some of that doesn't actually build? (IIRC only our bots build 'all', most other bots don't)

With pinned clang:

$ time ninja -C out/gnios ios_clang_arm64/obj/components/cronet/ios/cronet_consumer/arm64/cronet_consumer 
ninja: Entering directory `out/gnios'
[8724/8724] LINK ios_clang_arm64/obj/components/cronet/ios/cronet_consumer/arm64/cronet_consumer
ld: warning: ignoring file obj/third_party/boringssl/boringssl_asm/chacha-armv4.o, file was built for armv7 which is not the architecture being linked (arm64): obj/third_party/boringssl/boringssl_asm/chacha-armv4.o
ld: warning: ld: warning: ignoring file obj/components/grpc_support/grpc_support/bidirectional_stream_c.o, file was built for armv7 which is not the architecture being linked (arm64): obj/components/grpc_support/grpc_support/bidirectional_stream_c.oignoring file obj/components/grpc_support/grpc_support/bidirectional_stream.o, file was built for armv7 which is not the architecture being linked (arm64): obj/components/grpc_support/grpc_support/bidirectional_stream.o

ld: warning: ignoring file obj/third_party/boringssl/boringssl_asm/aes-armv4.o, file was built for armv7 which is not the architecture being linked (arm64): obj/third_party/boringssl/boringssl_asm/aes-armv4.o
ld: warning: ignoring file obj/third_party/boringssl/boringssl_asm/aesv8-armx32.o, file was built for armv7 which is not the architecture being linked (arm64): obj/third_party/boringssl/boringssl_asm/aesv8-armx32.o
ld: warning: ignoring file obj/third_party/boringssl/boringssl_asm/bsaes-armv7.o, file was built for armv7 which is not the architecture being linked (arm64): obj/third_party/boringssl/boringssl_asm/bsaes-armv7.o
ld: warning: ignoring file obj/third_party/boringssl/boringssl_asm/armv4-mont.o, file was built for armv7 which is not the architecture being linked (arm64): obj/third_party/boringssl/boringssl_asm/armv4-mont.o
ld: warning: ignoring file obj/third_party/boringssl/boringssl_asm/ghash-armv4.o, file was built for armv7 which is not the architecture being linked (arm64): obj/third_party/boringssl/boringssl_asm/ghash-armv4.o
ld: warning: ignoring file obj/third_party/boringssl/boringssl_asm/ghashv8-armx32.o, file was built for armv7 which is not the architecture being linked (arm64): obj/third_party/boringssl/boringssl_asm/ghashv8-armx32.o
ld: warning: ignoring file obj/third_party/boringssl/boringssl_asm/sha1-armv4-large.o, file was built for armv7 which is not the architecture being linked (arm64): obj/third_party/boringssl/boringssl_asm/sha1-armv4-large.o
ld: warning: ignoring file obj/third_party/boringssl/boringssl_asm/sha256-armv4.o, file was built for armv7 which is not the architecture being linked (arm64): obj/third_party/boringssl/boringssl_asm/sha256-armv4.o
ld: warning: ignoring file obj/net/constants/trace_constants.o, file was built for armv7 which is not the architecture being linked (arm64): obj/net/constants/trace_constants.o
ld: warning: ignoring file obj/third_party/boringssl/boringssl_asm/sha512-armv4.o, file was built for armv7 which is not the architecture being linked (arm64): obj/third_party/boringssl/boringssl_asm/sha512-armv4.o
ld: warning: ignoring file obj/third_party/zlib/zlib_adler32_simd/adler32_simd.o, file was built for armv7 which is not the architecture being linked (arm64): obj/third_party/zlib/zlib_adler32_simd/adler32_simd.o
ld: warning: ld: warning: ld: warning: ignoring file obj/base/third_party/dynamic_annotations/libdynamic_annotations.a, file was built for archive which is not the architecture being linked (arm64): obj/base/third_party/dynamic_annotations/libdynamic_annotations.aignoring file obj/third_party/modp_b64/libmodp_b64.a, file was built for archive which is not the architecture being linked (arm64): obj/third_party/modp_b64/libmodp_b64.ald: warning: ld: warning: ignoring file obj/base/libbase_static.a, file was built for archive which is not the architecture being linked (arm64): obj/base/libbase_static.ald: warning: 

ignoring file obj/base/third_party/libevent/libevent.a, file was built for archive which is not the architecture being linked (arm64): obj/base/third_party/libevent/libevent.aignoring file obj/base/libbase.a, file was built for archive which is not the architecture being linked (arm64): obj/base/libbase.ald: warning: 
ignoring file obj/net/libnet_quic_proto.a, file was built for archive which is not the architecture being linked (arm64): obj/net/libnet_quic_proto.a

ignoring file obj/net/libnet.a, file was built for archive which is not the architecture being linked (arm64): obj/net/libnet.a

ld: warning: ignoring file obj/crypto/libcrcrypto.a, file was built for archive which is not the architecture being linked (arm64): obj/crypto/libcrcrypto.a
ld: warning: ignoring file obj/third_party/zlib/libchrome_zlib.a, file was built for archive which is not the architecture being linked (arm64): obj/third_party/zlib/libchrome_zlib.a
ld: warning: ignoring file obj/third_party/protobuf/libprotobuf_lite.a, file was built for archive which is not the architecture being linked (arm64): obj/third_party/protobuf/libprotobuf_lite.a
ld: warning: ignoring file obj/third_party/zlib/libzlib_x86_simd.a, file was built for archive which is not the architecture being linked (arm64): obj/third_party/zlib/libzlib_x86_simd.a
ld: warning: ignoring file obj/third_party/icu/libicui18n.a, file was built for archive which is not the architecture being linked (arm64): obj/third_party/icu/libicui18n.a
ld: warning: ignoring file obj/third_party/boringssl/libboringssl.a, file was built for archive which is not the architecture being linked (arm64): obj/third_party/boringssl/libboringssl.a
ld: warning: ignoring file obj/url/liburl.a, file was built for archive which is not the architecture being linked (arm64): obj/url/liburl.a
ld: warning: ignoring file obj/base/libbase_i18n.a, file was built for archive which is not the architecture being linked (arm64): obj/base/libbase_i18n.a
ld: warning: ignoring file obj/third_party/icu/libicuuc.a, file was built for archive which is not the architecture being linked (arm64): obj/third_party/icu/libicuuc.a
ld: warning: ignoring file obj/third_party/ced/libced.a, file was built for archive which is not the architecture being linked (arm64): obj/third_party/ced/libced.ald: warning: 
ignoring file obj/third_party/brotli/libdec.a, file was built for archive which is not the architecture being linked (arm64): obj/third_party/brotli/libdec.a
ld: warning: ignoring file obj/third_party/brotli/libcommon.a, file was built for archive which is not the architecture being linked (arm64): obj/third_party/brotli/libcommon.a

real	43m54.157s
user	312m5.549s
sys	18m56.065s

So it works there. But I bet it's still triggered by that CL, and some earlier change on tot added the now-fatal diagnostic. Which means we don't have a clean regression window :-/

We're currently at 315613, bad build was at 316206, so our range is 315614:316206

...but all these warnings in any case mean that the build is in a pretty bad state, and fixing that might fix the error too. +justincohen, sdefresne, see warnings in comment 4. I used these args.gn:

additional_target_cpus = [ "arm64", "x64", "x86" ]
ios_enable_code_signing = false
is_component_build = false
is_debug = false
target_cpu = "arm"
target_os = "ios"

The error string in comment 0 belongs to object_error::bitcode_section_not_found, but I don't see any recent commits touching that.

Does the bad build at r316206 repro locally?

Maybe after https://chromium-review.googlesource.com/725080 we build more, and then the bot started breaking due to some resource problem perhaps?

need to pass tools/clang/clang.order-DCOMPILER_RT_ENABLE_IOS=ON to cmake to build libclang_rt.profile_iossim.a

Hm cmake caches make this setting inconvenient to toggle. I went with `cp $(find ~/src/chrome/src/third_party/llvm-build -name libclang_rt.ios.a) lib/clang/6.0.0/lib/darwin/` instead.

Repros locally with my `clang version 6.0.0 (trunk 316334)` build with the instructions above. In particular, comment 10/11. I wonder if the missing bitcode is in libclang_rt.ios.a?

https://reviews.llvm.org/rL218078 suggests pcc is somewhat familiar with the error; +pcc in case he knows why we started seeing this.

Error string is emitted here (rebuild libLTO.dylib, not clang, when changing things):

Unchecked Expected<T> contained error:
0   libc++.1.dylib                      0x00007fff9dcdeea4 std::__1::error_code::message() const + 24
1   libLTO.dylib                        0x000000010f022e59 llvm::ECError::log(llvm::raw_ostream&) const + 25
2   libLTO.dylib                        0x000000010ef83a72 llvm::Expected<llvm::MemoryBufferRef>::assertIsChecked() + 434
3   libLTO.dylib                        0x000000010ef80b32 llvm::LTOModule::isBitcodeForTarget(llvm::MemoryBuffer*, llvm::StringRef) + 434
4   libLTO.dylib                        0x000000010e102251 lto_module_is_object_file_in_memory_for_target + 97
5   ld                                  0x0000000108f9b4fe lto::Parser::validFile(unsigned char const*, unsigned long long, int, int) + 68
6   ld                                  0x0000000108f9feac lto::isObjectFile(unsigned char const*, unsigned long long, int, int) + 56
7   ld                                  0x0000000108f9a22f archive::File<arm64>::validFile(unsigned char const*, unsigned long long, mach_o::relocatable::ParserOptions const&) + 333
8   ld                                  0x0000000108f95bbb archive::parse(unsigned char const*, unsigned long long, char const*, long, ld::File::Ordinal, archive::ParserOptions const&) + 248
9   ld                                  0x0000000108faecec ld::tool::InputFiles::makeFile(Options::FileInfo const&, bool) + 1350
10  ld                                  0x0000000108fb146d ld::tool::InputFiles::parseWorkerThread() + 497
11  libsystem_pthread.dylib             0x00007fff9f31b93b _pthread_body + 180
12  libsystem_pthread.dylib             0x00007fff9f31b887 _pthread_body + 0
13  libsystem_pthread.dylib             0x00007fff9f31b08d thread_start + 13

Hm, we don't ship libLTO.dylib in the package, so we don't get any signal from the pinned build working (this also means this doesn't block rolls). So probably due to r315483 (and just exposes an existing badness)?

We should probably just delete libLTO.dylib as part of update.py then.

Probably. But I think this is a bug from https://reviews.llvm.org/rL315483. Expected<> calls assertIsChecked() in its dtor, and operator bool() only calls setChecked() if there's no error. I think the bool-returning functions in LTOModule.cpp need to do something like

  if (!myExpected) {
    consumeError(BCData.takeError());
    return false:
  }
  return true;

with Expected<> if I'm reading things right.

This fixes the assert:

$ svn diff lib/LTO/
Index: lib/LTO/LTOModule.cpp
===================================================================
--- lib/LTO/LTOModule.cpp	(revision 316334)
+++ lib/LTO/LTOModule.cpp	(working copy)
@@ -89,8 +89,10 @@
                                    StringRef TriplePrefix) {
   Expected<MemoryBufferRef> BCOrErr =
       IRObjectFile::findBitcodeInMemBuffer(Buffer->getMemBufferRef());
-  if (!BCOrErr)
+  if (!BCOrErr) {
+    consumeError(BCOrErr.takeError());
     return false;
+  }
   LLVMContext Context;
   ErrorOr<std::string> TripleOrErr =
       expectedToErrorOrAndEmitErrors(Context, getBitcodeTargetTriple(*BCOrErr));


There are several other callers that look like they need this treatment. I wonder if respindola did this mass-replace elsewhere and missed this nuance.

Sent respindola an email.

Also sent https://reviews.llvm.org/D39437

https://reviews.llvm.org/rL317010

Will hopefully stick and the bot will hopefully cycle green.

It turned green: https://build.chromium.org/p/chromium.clang/builders/ToTiOS/builds/179