https://ci.chromium.org/buildbot/chromium.clang/ToTLinux/ was also red in compile and is now red in tests, so maybe this is already resolved at head and the win bots are just behind.

Still broken.

First instance of this failure:
https://ci.chromium.org/buildbot/chromium.clang/ToTWin64%28dbg%29/1150

Assuming this is caused by upstream changes, our LLVM rev range would be:
bad: r342111
good: r342102

Pretty small. I tried to get a reproducer, but I must've gotten my gn args wrong, because it didn't crash.

This bug looks related to PCH. It looks like we auto-enable PCH without goma, and auto-disable it with goma. Normally to repro problems quickly, I build with goma -j900 to get to the crashing TU, and then rebuild just that TU with ToT clang-cl. That won't work for PCH-related issues, I guess. I suppose it makes it more important to finish deploying the crash uploader.

Should we be continuously testing PCH builds on the ToT waterfall? We definitely don't want to ship a clang with broken PCH, since we want users to be able to build Chromium without goma. I imagine that PCH builds are inherently more fragile than non-PCH builds, but is it possible that by testing PCH builds only, we will miss some goma-specific failures? I suppose we will inevitably catch those types of non-PCH or goma-only failures when we run try jobs on our clang roll change after uploading the new package, but should we have more continuous monitoring?

There are no clang changes in the LLVM revision range, so this must have been caused by some Chromium side change. It's possible that it will reproduce with pinned clang without goma. I'll check that next. If so, would it be reasonable to revert the chromium cl that caused this while we fix the bug?

> It's possible that it will reproduce with pinned clang without goma. I'll check that next.

I believe I was running into this when trying to build Chromium; see my report in  Issue 885008 .

I'm not aware of any issues we missed due to our only testing pchs on the tot bots.

Someone one irc claimed that they're seeing this crash with pinned clang on irc. I wondered how that could happen given all the public bots are happy -- but that's because none of them use pchs.

So yes, this is likely broken with pinned clang too.

 Issue 885008  has been merged into this issue.

 Issue 885324  has been merged into this issue.

From the dupe, a stack:

1.      ../../third_party/blink/renderer/modules/accessibility/ax_object.h:1088:2: current parser token ';'
2.      ../../third_party/blink/renderer/modules/accessibility/ax_object.h:164:1: parsing namespace 'blink'
3.      ../../third_party/blink/renderer/modules/accessibility/ax_object.h:166:1: parsing struct/union/class body 'blink::AXObject'
4.      ../../third_party/blink/renderer/modules/accessibility/ax_object.h:295:32: LLVM IR generation of inline function 'blink::swap'
5.      ../../third_party/blink/renderer/modules/accessibility/ax_object.h:295:32: Generating code for declaration 'blink::swap'
6.      /usr/local/google/home/tmathmeyer/chromium/src/out/Windows/../../third_party/blink/renderer/platform/heap/heap_page.h:717:3 <Spelling=../../third_part
y/blink/renderer/platform/wtf/allocator.h:110:9>: LLVM IR generation of declaration 'blink::BaseArena::operator new'
7.      ../../third_party/blink/renderer/modules/accessibility/ax_object.h:243:5: Generating code for declaration 'blink::AXObject::InOrderTraversalIterator::
InOrderTraversalIterator'
#0 0x00000000020d5c84 (../../third_party/llvm-build/Release+Asserts/bin/clang-cl+0x20d5c84)
#1 0x00000000020d3b1e llvm::sys::RunSignalHandlers() (../../third_party/llvm-build/Release+Asserts/bin/clang-cl+0x20d3b1e)
#2 0x00000000020d5e42 (../../third_party/llvm-build/Release+Asserts/bin/clang-cl+0x20d5e42)
#3 0x00007fb9c09370c0 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x110c0)
#4 0x00007fb9bf6f2fcf gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x32fcf)
#5 0x00007fb9bf6f43fa abort (/lib/x86_64-linux-gnu/libc.so.6+0x343fa)
#6 0x00007fb9bf6ebe37 (/lib/x86_64-linux-gnu/libc.so.6+0x2be37)
#7 0x00007fb9bf6ebee2 (/lib/x86_64-linux-gnu/libc.so.6+0x2bee2)
#8 0x00000000023bf34c clang::CodeGen::CodeGenTypes::getCGRecordLayout(clang::RecordDecl const*) (../../third_party/llvm-build/Release+Asserts/bin/clang-cl+0x2
3bf34c)
...
#98 0x000000000335c666 clang::ParseAST(clang::Sema&, bool, bool) (../../third_party/llvm-build/Release+Asserts/bin/clang-cl+0x335c666)
#99 0x00000000027751c0 clang::FrontendAction::Execute() (../../third_party/llvm-build/Release+Asserts/bin/clang-cl+0x27751c0)
#100 0x000000000271d6b1 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (../../third_party/llvm-build/Release+Asserts/bin/clang-cl+0x271d6b1)
#101 0x0000000002819399 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (../../third_party/llvm-build/Release+Asserts/bin/clang-cl+0x2819399)
#102 0x00000000008d8694 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (../../third_party/llvm-build/Release+Asserts/bin/clang-cl+0x8d8694)
#103 0x00000000008d5e95 main (../../third_party/llvm-build/Release+Asserts/bin/clang-cl+0x8d5e95)
#104 0x00007f078c0232b1 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b1)
#105 0x00000000008d2e3d _start (../../third_party/llvm-build/Release+Asserts/bin/clang-cl+0x8d2e3d)
clang-cl: error: unable to execute command: Aborted
clang-cl: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 8.0.0 (trunk 340925)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: ../../third_party/llvm-build/Release+Asserts/bin
clang-cl: note: diagnostic msg: PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run
script.
clang-cl: note: diagnostic msg:
********************

(The dupe makes the point that this affects everyone doing cross builds and has args.gn that repro)

For people blocked on this, from the first dupe:

"""Modifying build/config/pch.gni to add "&& !is_clang" to "enable_precompiled_headers" has allowed me to build."""

(This makes the build slower, so don't forget to undo once we have fixed this)

I wasn't able to come up with a good test case for this, but it looks like this bug was caused by some special-case code for dllexport inline friend function definitions added back in r264841. I simplified the code in r342516, and that seems to fix the issue.

We rolled in a fixed clang in #592531, if you're synced past that you should be good (assuming the roll sticks).

> Should we be continuously testing PCH builds on the ToT waterfall? We definitely don't want to ship a clang with broken PCH, since we want users to be able to build Chromium without goma. I imagine that PCH builds are inherently more fragile than non-PCH builds, but is it possible that by testing PCH builds only, we will miss some goma-specific failures? I suppose we will inevitably catch those types of non-PCH or goma-only failures when we run try jobs on our clang roll change after uploading the new package, but should we have more continuous monitoring?

I think the current setup is probably fine: as you say, the PCH build is inherently more fragile so having that enabled gives us more coverage. It does mean the non-PCH case doesn't get tested as well, but probably the diff in problem surface isn't that big, and we'll see such failures when running tryjobs.