I patched in the 2017 toolchain CL and ran 'gclient runhooks', and I wasn't able to reproduce this crash. =/

That this didn't fire on CrWinClang64 could suggest that it's only a problem with ToT Clang, but it could also suggest it's a problem with Clang when boostrapped with VS2017.

My money is on a compiler bug in VS 2017. Assign to me if you want me to investigate.

Note to self: set LLVM_FORCE_HEAD_REVISION=1 and then gclient runhooks to build ToT Chrome.

I can repro the crash locally, however the TOT builds of clang default to having symbols disabled. What is the recommended way to change that?

I have newer VC++ 2017 builds which I want to try once I repro the crash with symbols.

If you use the `and args.bootstrap` bit here https://cs.chromium.org/chromium/src/tools/clang/scripts/update.py?q=update.py&sq=package:chromium&dr=C&l=591 then you'll probably get a build with symbols.

(…or you could also pass --bootstrap, but then the build will take twice as long.)

Thanks. That will help.

I think that we should always build the compiler with symbols, so that we can always debug and profile.

At the very least we should always do this part:

    ldflags += ['/DEBUG', '/OPT:REF', '/OPT:ICF']

That's equivalent to symbol_level = 1. That gives us symbolized call stacks, sufficient for first-level crash triage and for profiling, and the extra build cost is O(0). I'll create a CL to propose that.

I tried with the latest version of VC++ 2017 - Update 3 Preview, version 4.0. Testing with that took a bit of hackery (tools\gyp\pylib\gyp\MSVSVersion.py won't find locally installed preview versions) but ultimately led to the same failure. Symbolized partial call stack is here:

#0 0x00007ff78b776406 `anonymous namespace'::StmtProfiler::VisitStmt d:\src\chromium\src\third_party\llvm\tools\clang\lib\ast\stmtprofile.cpp:199:0
#1 0x00007ff78b773f23 clang::StmtVisitorBase<clang::make_const_ptr,`anonymous namespace'::StmtProfiler,void>::Visit d:\src\chromium\src\third_party\llvm-build\release+asserts\tools\clang\include\clang\ast\stmtnodes.inc:489:0
#2 0x00007ff78b7735c3 clang::StmtVisitorBase<clang::make_const_ptr,`anonymous namespace'::StmtProfiler,void>::Visit d:\src\chromium\src\third_party\llvm-build\release+asserts\tools\clang\include\clang\ast\stmtnodes.inc:345:0
#3 0x00007ff78b772cac clang::Stmt::Profile(class llvm::FoldingSetNodeID &,class clang::ASTContext const &,bool)const  d:\src\chromium\src\third_party\llvm\tools\clang\lib\ast\stmtprofile.cpp:1885:0
#4 0x00007ff78b5dc2de clang::TemplateSpecializationType::Profile(class llvm::FoldingSetNodeID &,class clang::TemplateName,class llvm::ArrayRef<class clang::TemplateArgument>,class clang::ASTContext const &) d:\src\chromium\src\third_party\llvm\tools\clang\lib\ast\type.cpp:3194:0
#5 0x00007ff78b67d600 clang::ASTContext::getCanonicalTemplateSpecializationType(class clang::TemplateName,class llvm::ArrayRef<class clang::TemplateArgument>)const  d:\src\chromium\src\third_party\llvm\tools\clang\lib\ast\astcontext.cpp:3714:0
#6 0x00007ff78afe6751 clang::Sema::CheckTemplateIdType(class clang::TemplateName,class clang::SourceLocation,class clang::TemplateArgumentListInfo &) d:\src\chromium\src\third_party\llvm\tools\clang\lib\sema\sematemplate.cpp:3014:0
#7 0x00007ff78b192f33 clang::TreeTransform<`anonymous namespace'::TemplateInstantiator>::TransformTemplateSpecializationType d:\src\chromium\src\third_party\llvm\tools\clang\lib\sema\treetransform.h:5780:0
#8 0x00007ff78b194771 clang::TreeTransform<`anonymous namespace'::TemplateInstantiator>::TransformType d:\src\chromium\src\third_party\llvm\tools\clang\include\clang\ast\typenodes.def:98:0
#9 0x00007ff78b194f63 clang::TreeTransform<`anonymous namespace'::TemplateInstantiator>::TransformType d:\src\chromium\src\third_party\llvm\tools\clang\lib\sema\treetransform.h:4108:0
#10 0x00007ff78b171876 clang::Sema::SubstType(class clang::QualType,class clang::MultiLevelTemplateArgumentList const &,class clang::SourceLocation,class clang::DeclarationName) d:\src\chromium\src\third_party\llvm\tools\clang\lib\sema\sematemplateinstantiate.cpp:1624:0

The crash happens because the StmtProfiler this pointer is bogus. It should (as far as I can tell) be pointing to a local variable a few levels up the stack so it seems hard to imagine that this could be anything other than a code-gen bug.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4c3d6a648ac28bcb24ab3537cc393f5467679b98

commit 4c3d6a648ac28bcb24ab3537cc393f5467679b98
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Sat Jul 15 01:51:41 2017

Always build clang with symbols on Windows

If we always build clang with symbols on Windows then it makes debugging
and profiling easier. The build time costs shouldn't be too bad, and the
generated code will be identical. This would have saved a bit of time in
the investigation of  crbug.com/727447 .

BUG= 727447 

Change-Id: Ib065be626db5d6b0b9c27792a0141c5fb696e208
Reviewed-on: https://chromium-review.googlesource.com/572660
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#486962}
[modify] https://crrev.com/4c3d6a648ac28bcb24ab3537cc393f5467679b98/tools/clang/scripts/update.py

I just installed VS 2017 Update 3 preview as well, so I might be able to reproduce this soon.

One other gn arg you can use to debug these kinds of problems is 'clang_base_path', which you can point at a local build of clang that you update and control yourself.

I wasn't able to reproduce this on the first try. I installed VS 2017 update 3 preview from Microsoft and used that to build LLVM. Is that the right version? This is the version string that compiler prints:
$ cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.10.25019 for x64

Is that right?

19.10 means that you are getting Update 2 or before. The very latest compiler should print Microsoft (R) C/C++ Optimizing Compiler Version 19.11.25505 for x64.

I believe that the bug is new in Update 3 so it makes sense that you would not see it.

Building clang with a locally installed version of VS 2017 is finicky, to say the least. Your best bet is to use the depot_tools toolchain. I updated that to the latest Update 3 version on Friday, so ideally you should sync past that point.

To request building with the VS 2017 depot_tools toolchain just set this and then run "gclient runhooks"

set GYP_MSVS_VERSION=2017

Then make sure that your ninja files are regenerated so that the change is picked up - gn gen does the trick.

If you want to use your locally built compiler then I think the trick is to do something like this:

set vs2017_install=C:\Program Files (x86)\Microsoft Visual Studio\Preview\Professional

I also had to hack src\tools\gyp\pylib\gyp\MSVSVersion.py to replace VC/vcvarsall.bat with VC/Auxiliary/Build/vcvarsall.bat.

I rolled tools\gyp so that local installs of VC++ 2017 should be found (vs2017_install might still need to be set to use preview versions). Here is my analysis up to date - I'm passing it off to Microsoft now because the problem seems to be that the compiler gets confused about where the 'this' pointer can be loaded from, and there are no signs in the code of why it does this. Here's my latest (hopefully complete) explanation:

The code-gen bug/crash only shows up when we build clang-cl with VC++ 2017 Update 3. When this happens we get a corrupt this pointer that leads to crash on what appears to be every compile. Typical crashes are like this:

mov     rcx,qword ptr [rbx+8] ds:00007ff6`00000014=????????????????

The bad "this" pointer is fairly consistent in that it is always the first 32-bits of the load address of clang.exe, but the bottom 32 bits are 0xC. Here's the load module information from the same debugging session that caused the crash shown above:

lm m clang
Browse full module list
start             end                 module name
00007ff6`0fd70000 00007ff6`140ce000   clang      (private pdb symbols)  C:\src\chromium3\src\third_party\llvm-build\Release+Asserts\bin\clang.pdb

I've investigated it by sprinkling code like this throughout the clang code-base:
    size_t THIS = (size_t)this;
    if ((THIS & 0xFFFFFFF0FFFF0000) == 0x7ff000000000)
      __debugbreak();

I just spent a while looking at one of the times when it hit a breakpoint and it looks to me like the VC++ compiler decides that the this pointer can be found somewhere that it simply isn't. For instance, I rearranged one of our functions to add my checks, like this:

    ID.AddInteger(SC);
    unsigned N = S->getNumArgs();
    size_t THIS = (size_t)this;
    if ((THIS & 0xFFFFFFF0FFFF0000) == 0x7ff000000000)
      __debugbreak();
    for (unsigned I = 0; I != N; ++I) {
      const auto* foo = S->getArg(I);
      THIS = (size_t)this;
      if ((THIS & 0xFFFFFFF0FFFF0000) == 0x7ff000000000)
        __debugbreak();
      Visit(foo);
    }

In the disassembly we can see that rbx initially contains this, but is destroyed. Then it reloads this and does the second masking operation - oddly enough, before it calls getArg(). And that reloaded this value is wrong.

00007ff6`1291231f 488d4b08        lea     rcx,[rbx+8] - apparently 'this' is in rbx at this point
00007ff6`12912323 8945c8          mov     dword ptr [rbp-38h],eax
00007ff6`12912326 48894dd0        mov     qword ptr [rbp-30h],rcx
00007ff6`1291232a 8bd0            mov     edx,eax
00007ff6`1291232c 488b09          mov     rcx,qword ptr [rcx]
00007ff6`1291232f e8ac5a90fe      call    clang!llvm::FoldingSetNodeID::AddInteger (00007ff6`11217de0)
00007ff6`12912334 8b5718          mov     edx,dword ptr [rdi+18h]
00007ff6`12912337 48b90000fffff0ffffff mov rcx,0FFFFFFF0FFFF0000h
00007ff6`12912341 4823d9          and     rbx,rcx - rbx gets destroyed
00007ff6`12912344 895540          mov     dword ptr [rbp+40h],edx
00007ff6`12912347 48b800000000f07f0000 mov rax,7FF000000000h
00007ff6`12912351 483bd8          cmp     rbx,rax
00007ff6`12912354 7501            jne     clang!clang::StmtVisitorBase<clang::make_const_ptr,`anonymous namespace'::StmtProfiler,void>::Visit+0x1c7 (00007ff6`12912357)
00007ff6`12912356 cc              int     3 - it doesn't hit this one
00007ff6`12912357 85d2            test    edx,edx
00007ff6`12912359 7476            je      clang!clang::StmtVisitorBase<clang::make_const_ptr,`anonymous namespace'::StmtProfiler,void>::Visit+0x241 (00007ff6`129123d1)
00007ff6`1291235b 488b4530        mov     rax,qword ptr [rbp+30h] - this is the start of the loop and it appears to be loading 'this' from rbp+30h and I am reasonably certain that that is wrong
00007ff6`1291235f 48bb00000000f07f0000 mov rbx,7FF000000000h
00007ff6`12912369 488bd0          mov     rdx,rax
00007ff6`1291236c 4823d1          and     rdx,rcx
00007ff6`1291236f 4883c008        add     rax,8
00007ff6`12912373 488955d8        mov     qword ptr [rbp-28h],rdx - here it stores away the *second* masked of copy of this
00007ff6`12912377 488945e0        mov     qword ptr [rbp-20h],rax
00007ff6`1291237b 0f1f440000      nop     dword ptr [rax+rax]
00007ff6`12912380 3b7718          cmp     esi,dword ptr [rdi+18h]
00007ff6`12912383 7219            jb      clang!clang::StmtVisitorBase<clang::make_const_ptr,`anonymous namespace'::StmtProfiler,void>::Visit+0x20e (00007ff6`1291239e)
00007ff6`12912385 41b8e8080000    mov     r8d,8E8h
00007ff6`1291238b 488d151ecb2600  lea     rdx,[clang!`string' (00007ff6`12b7eeb0)]
00007ff6`12912392 488d0d67e82600  lea     rcx,[clang!`string' (00007ff6`12b80c00)]
00007ff6`12912399 e82eb90f00      call    clang!_wassert (00007ff6`12a0dccc)
00007ff6`1291239e 8b07            mov     eax,dword ptr [rdi]
00007ff6`129123a0 488b4f10        mov     rcx,qword ptr [rdi+10h]
00007ff6`129123a4 c1e811          shr     eax,11h
00007ff6`129123a7 83e001          and     eax,1
00007ff6`129123aa ffc0            inc     eax
00007ff6`129123ac 03c6            add     eax,esi
00007ff6`129123ae 488b0cc1        mov     rcx,qword ptr [rcx+rax*8]
00007ff6`129123b2 e829a84dfd      call    clang!llvm::cast_or_null<clang::Expr,clang::Stmt> (00007ff6`0fdecbe0)
00007ff6`129123b7 48395dd8        cmp     qword ptr [rbp-28h],rbx - here it checks the stored copy of this, and it is corrupt
00007ff6`129123bb 7501            jne     clang!clang::StmtVisitorBase<clang::make_const_ptr,`anonymous namespace'::StmtProfiler,void>::Visit+0x22e (00007ff6`129123be)
00007ff6`129123bd cc              int     3 - it hits this breakpoint, and rbp+30h contains a bad value


I've found that if I add an iteration counter (static int64_t s_counter = 0; ++s_counter;) to the beginning of the function then the bug goes away, but that's not a very satisfying workaround. Our bug for this is  crbug.com/727447 . In order to reproduce this my steps (needs a tree from this afternoon or later) are:


set LLVM_FORCE_HEAD_REVISION=1
set GYP_MSVS_VERSION=2017
@rem This just has to be run one
python tools\clang\scripts\update.py
On subsequent syncs it is enough to "gclient runhooks" to build ToT clang
After clang has been built (which takes quite a while) I used these gn args. Most are probably not necessary, but the first three presumably are.

>type out\clang_tot\args.gn
clang_use_chrome_plugins = false
is_clang = true
llvm_force_head_revision = true
is_component_build = false
is_debug = false
is_official_build = true
symbol_level = 1

I then go:
> ninja -C clang_tot d keeprsp obj/base/base/message_pump_win.obj

and I get a compiler crash. I use -d keeprsp to keep the .rsp files so that I can run clang under the debugger. My debugger command-line is:

> cd out\clang_tot
> windbg64 -g -G -o "..\..\third_party\llvm-build\Release+Asserts\bin\clang-cl.exe" /nologo @obj/base/base/message_pump_win.obj.rsp /c ../../base/message_loop/message_pump_win.cc /Foobj/base/base/message_pump_win.obj /Fd"obj/base/base_cc.pdb"

It doesn't matter what source file you build, but this one works for me.

In order to iterate on this - rebuilding clang after making a single change to its code - I run this once:

"C:\Program Files (x86)\Microsoft Visual Studio\Preview\Professional\VC\Auxiliary\Build\vcvarsall.bat" amd64

And then run this batch file to incrementally rebuild clang.

@setlocal
pushd C:\src\chromium3\src\third_party\llvm-build\Release+Asserts
ninja.exe -k 0"
popd

When experimenting my modifications look like this - this catches the corrupt this pointer in before/after state. This is in src\third_party\llvm\tools\clang\lib\AST\StmtProfile.cpp, which will exist after running the previous steps.

void StmtProfiler::VisitCXXOperatorCallExpr(const CXXOperatorCallExpr *S) {
  //static int64_t s_callcount = 0; // Uncommenting this 'fixes' the bug
  //++s_callcount;
  if (S->isTypeDependent()) {
    // Type-dependent operator calls are profiled like their underlying
    // syntactic operator.
    //
    // An operator call to operator-> is always implicit, so just skip it. The
    // enclosing MemberExpr will profile the actual member access.
    if (S->getOperator() == OO_Arrow)
      return Visit(S->getArg(0));

    UnaryOperatorKind UnaryOp = UO_Extension;
    BinaryOperatorKind BinaryOp = BO_Comma;
    Stmt::StmtClass SC = DecodeOperatorCall(S, UnaryOp, BinaryOp);

    ID.AddInteger(SC);
    unsigned N = S->getNumArgs();
    size_t THIS = (size_t)this;
    if ((THIS & 0xFFFFFFF0FFFF0000) == 0x7ff000000000)
      __debugbreak();
    for (unsigned I = 0; I != N; ++I) {
      const auto* foo = S->getArg(I);
      THIS = (size_t)this;
      if ((THIS & 0xFFFFFFF0FFFF0000) == 0x7ff000000000)
        __debugbreak();
      Visit(foo);
    }
    if (SC == Stmt::UnaryOperatorClass)
      ID.AddInteger(UnaryOp);
    else if (SC == Stmt::BinaryOperatorClass || 
             SC == Stmt::CompoundAssignOperatorClass)
      ID.AddInteger(BinaryOp);
    else
      assert(SC == Stmt::ArraySubscriptExprClass);

    return;
  }

  VisitCallExpr(S);
  ID.AddInteger(S->getOperator());
}

This modification to llvm\tools\clang\lib\AST\StmtProfile.cpp is sufficient to avoid the crash. The crash happens in the loop because VC++ loses track of where the 'this' pointer can be found somewhere around the start of the loop. There may be code rearrangements that would avoid the bug but disabling optimizations for the problematic version seems more stable. A VS bug was filed here:

https://developercommunity.visualstudio.com/content/problem/84002/clang-cl-when-built-with-vc-2017-crashes-cause-vc.html

#if defined(_MSC_VER) && _MSC_VER == 1911
// Work around https://bugs.chromium.org/p/chromium/issues/detail?id=727447
// It appears to first show up in VS 2017 Update 3 and one hopes will be fixed
// in subsequent versions.
#pragma optimize("", off)
#endif

void StmtProfiler::VisitCXXOperatorCallExpr(const CXXOperatorCallExpr *S) {
  if (S->isTypeDependent()) {
    // Type-dependent operator calls are profiled like their underlying
    // syntactic operator.
    //
    // An operator call to operator-> is always implicit, so just skip it. The
    // enclosing MemberExpr will profile the actual member access.
    if (S->getOperator() == OO_Arrow)
      return Visit(S->getArg(0));

    UnaryOperatorKind UnaryOp = UO_Extension;
    BinaryOperatorKind BinaryOp = BO_Comma;
    Stmt::StmtClass SC = DecodeOperatorCall(S, UnaryOp, BinaryOp);

    ID.AddInteger(SC);
    for (unsigned I = 0, N = S->getNumArgs(); I != N; ++I)
      Visit(S->getArg(I));
    if (SC == Stmt::UnaryOperatorClass)
      ID.AddInteger(UnaryOp);
    else if (SC == Stmt::BinaryOperatorClass || 
             SC == Stmt::CompoundAssignOperatorClass)
      ID.AddInteger(BinaryOp);
    else
      assert(SC == Stmt::ArraySubscriptExprClass);

    return;
  }

  VisitCallExpr(S);
  ID.AddInteger(S->getOperator());
}

#if defined(_MSC_VER) && _MSC_VER == 1911
// Work around https://bugs.chromium.org/p/chromium/issues/detail?id=727447
#pragma optimize("", on)
#endif

Nice sleuthing! Seems reasonable to land that patch upstream.

I wish I'd found something more definitive and precisely actionable. Oh well - not all compiler bugs are amenable to "thinking like a compiler" making them obvious.

> Nice sleuthing! Seems reasonable to land that patch upstream.

Bruce or Nico, do you want to push this upstream? It would be nice to get into the upcoming LLVM 5 release if so.

Check your inbox for "r308897 - Work around an MSVC2017 update 3 codegen bug." for the 5.0 merge request :-)

Also, Microsoft has reproduced the issue internally and is investigating.

Disabling inlining is also sufficient to avoid this code-gen bug. Microsoft was having trouble reproducing it so I sent them a preprocessed file and while I was validating that this reproduces the bug I learned a bit more about how the bug works. The generated code basically goes:

mov DWORD PTR [rbp+Offset1],12 - store some value to a 32-bit integer local
mov EAX, [rbp+Offset2] - load the this pointer into a register

Unfortunately Offset1 == Offset2, so the bottom 32-bits of the 'this' pointer were trashed. And, it turns out that the top 32-bits were bogus as well - I don't think that stack location was ever actually initialized with the 'this' pointer, although I'm not sure.

Anyway, the current workaround is sufficient and we can remove it if we get a fixed compiler.

This bug isn't actually fixed (disabling optimizations for that function is apparently not sufficient) but the switch to clang may have rendered it irrelevant.

Either way, I'm attaching the preprocessed StmtProfile.cpp that makes it easy to reproduce the bug. Below I have the instructions for how I created the preprocessed file and how I manually verified that it was producing the bad code-gen through inspection of the .cod file it generates. The summary is that the compiler writes 12 to UnaryOp$17[rbp-88] and later reads THIS from tv13670[rbp-88]. These two locations are the same which explains why the lower 32-bits of THIS contain 0x0000000C instead of the correct value. VS bug is filed at https://developercommunity.visualstudio.com/content/problem/84002/clang-cl-when-built-with-vc-2017-crashes-cause-vc.html. Here are the boring details:


I ran ninja -v -d keeprsp to get the command-line for compiling StmtProfile.cpp and then modified it to remove /showIncludes and add /P, thusly:

cl /nologo /TP -DGTEST_HAS_RTTI=0 -DLLVM_BUILD_GLOBAL_ISEL -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_GNU_SOURCE -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools\clang\lib\AST -IC:\src\chromium3\src\third_party\llvm\tools\clang\lib\AST -IC:\src\chromium3\src\third_party\llvm\tools\clang\include -Itools\clang\include -Iinclude -IC:\src\chromium3\src\third_party\llvm\include -DLLVM_FORCE_HEAD_REVISION /Zi /Zc:inline /Zc:strictStrings /Oi /Zc:rvalueCast /W4 -wd4141 -wd4146 -wd4180 -wd4244 -wd4258 -wd4267 -wd4291 -wd4345 -wd4351 -wd4355 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4800 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4324 -w14062 -we4238 /MT /O2 /Ob2   -UNDEBUG  /EHs-c- /GR- /Fotools\clang\lib\AST\CMakeFiles\clangAST.dir\StmtProfile.cpp.obj /Fdtools\clang\lib\AST\CMakeFiles\clangAST.dir\ /FS -c C:\src\chromium3\src\third_party\llvm\tools\clang\lib\AST\StmtProfile.cpp /P

I then renamed the .i file and compiled it (with preprocessor/include directives removed) and /FAcs added - this you can trivially do:

cl /TP /Zi /Zc:inline /Zc:strictStrings /Oi /Zc:rvalueCast /W4 -wd4141 -wd4146 -wd4180 -wd4244 -wd4258 -wd4267 -wd4291 -wd4345 -wd4351 -wd4355 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4800 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4324 -w14062 -we4238 /MT /O2 /Ob2   -UNDEBUG  /EHs-c- /GR- /Fotools\clang\lib\AST\CMakeFiles\clangAST.dir\StmtProfile.cpp.obj /Fdtools\clang\lib\AST\CMakeFiles\clangAST.dir\ /FS -c StmtProfile.cpp /FAcs

If you search the .cod file for "; 1417" (without the quotes) then the first instance is the non-inlined function (which is correct) and the second instance is the inlined function (which is incorrect). Here is an excerpt from the .cod file:

	00 00		 mov	 DWORD PTR UnaryOp$17[rbp-88], 12 - store 12 to the stack
  00188	48 8d 55 30	 lea	 rdx, QWORD PTR UnaryOp$17[rbp-88]
  0018c	c7 45 38 1f 00
	00 00		 mov	 DWORD PTR BinaryOp$20[rbp-88], 31
  00193	e8 00 00 00 00	 call	 ?DecodeOperatorCall@@YA?AW4StmtClass@Stmt@clang@@PEBVCXXOperatorCallExpr@3@AEAW4UnaryOperatorKind@3@AEAW4BinaryOperatorKind@3@@Z ; DecodeOperatorCall

; 1407 : 
; 1408 :     ID.AddInteger(SC);

  00198	48 8d 4b 08	 lea	 rcx, QWORD PTR [rbx+8]
  0019c	89 45 c8	 mov	 DWORD PTR SC$1$[rbp-88], eax
  0019f	48 89 4d d0	 mov	 QWORD PTR tv13814[rbp-88], rcx
  001a3	8b d0		 mov	 edx, eax
  001a5	48 8b 09	 mov	 rcx, QWORD PTR [rcx]
  001a8	e8 00 00 00 00	 call	 ?AddInteger@FoldingSetNodeID@llvm@@QEAAXH@Z ; llvm::FoldingSetNodeID::AddInteger
; File c:\src\chromium3\src\third_party\llvm\tools\clang\include\clang\ast\expr.h

; 2263 :   unsigned getNumArgs() const { return NumArgs; }

  001ad	8b 47 18	 mov	 eax, DWORD PTR [rdi+24]
; File c:\src\chromium3\src\third_party\llvm\tools\clang\lib\ast\stmtprofile.cpp

; 1411 :     if ((THIS & 0xFFFFFFF0FFFF0000) == 0x7ff000000000)

  001b0	48 ba 00 00 ff
	ff f0 ff ff ff	 mov	 rdx, -64424574976	; fffffff0ffff0000H
; File c:\src\chromium3\src\third_party\llvm\tools\clang\include\clang\ast\expr.h

; 2263 :   unsigned getNumArgs() const { return NumArgs; }

  001ba	89 45 40	 mov	 DWORD PTR N$1$[rbp-88], eax
; File c:\src\chromium3\src\third_party\llvm\tools\clang\lib\ast\stmtprofile.cpp

; 1411 :     if ((THIS & 0xFFFFFFF0FFFF0000) == 0x7ff000000000)

  001bd	48 23 da	 and	 rbx, rdx
  001c0	48 b8 00 00 00
	00 f0 7f 00 00	 mov	 rax, 140668768878592	; 00007ff000000000H
  001ca	48 3b d8	 cmp	 rbx, rax
  001cd	75 01		 jne	 SHORT $LN2226@Visit

; 1412 :       __debugbreak();

  001cf	cc		 int	 3
$LN2226@Visit:

; 1413 :     for (unsigned I = 0; I != N; ++I) {

  001d0	8b 5d 40	 mov	 ebx, DWORD PTR N$1$[rbp-88] - loop start
  001d3	85 db		 test	 ebx, ebx
  001d5	74 73		 je	 SHORT $LN2221@Visit
  001d7	48 8b 45 30	 mov	 rax, QWORD PTR tv14635[rbp-88] - load 'this' pointer, but this is the same address used for UnaryOp$17 so it is bad
  001db	48 8b c8	 mov	 rcx, rax
  001de	48 23 ca	 and	 rcx, rdx
  001e1	48 83 c0 08	 add	 rax, 8
  001e5	48 89 4d 40	 mov	 QWORD PTR tv13670[rbp-88], rcx
  001e9	48 89 45 d8	 mov	 QWORD PTR tv13669[rbp-88], rax
  001ed	0f 1f 00	 npad	 3
$LL2222@Visit:
; File c:\src\chromium3\src\third_party\llvm\tools\clang\include\clang\ast\expr.h

; 2280 :     assert(Arg < NumArgs && "Arg access out of range!");

  001f0	3b 77 18	 cmp	 esi, DWORD PTR [rdi+24]
  001f3	72 19		 jb	 SHORT $LN2244@Visit
  001f5	41 b8 e8 08 00
	00		 mov	 r8d, 2280		; 000008e8H
  001fb	48 8d 15 00 00
	00 00		 lea	 rdx, OFFSET FLAT:??_C@_1JG@PHHIMMIO@?$AAC?$AA?3?$AA?2?$AAs?$AAr?$AAc?$AA?2?$AAc?$AAh?$AAr?$AAo?$AAm?$AAi?$AAu?$AAm?$AA3?$AA?2?$AAs?$AAr?$AAc?$AA?2?$AAt?$AAh?$AAi?$AAr?$AAd?$AA_?$AAp?$AAa?$AAr?$AAt?$AAy@
  00202	48 8d 0d 00 00
	00 00		 lea	 rcx, OFFSET FLAT:??_C@_1FI@GHKDCCJK@?$AAA?$AAr?$AAg?$AA?5?$AA?$DM?$AA?5?$AAN?$AAu?$AAm?$AAA?$AAr?$AAg?$AAs?$AA?5?$AA?$CG?$AA?$CG?$AA?5?$AA?$CC?$AAA?$AAr?$AAg?$AA?5?$AAa?$AAc?$AAc?$AAe?$AAs?$AAs?$AA?5?$AAo?$AAu?$AAt@
  00209	e8 00 00 00 00	 call	 _wassert
$LN2244@Visit:

; 2237 :   unsigned getNumPreArgs() const { return CallExprBits.NumPreArgs; }

  0020e	8b 07		 mov	 eax, DWORD PTR [rdi]

; 2281 :     return cast_or_null<Expr>(SubExprs[Arg + getNumPreArgs() + PREARGS_START]);

  00210	48 8b 4f 10	 mov	 rcx, QWORD PTR [rdi+16]

; 2237 :   unsigned getNumPreArgs() const { return CallExprBits.NumPreArgs; }

  00214	c1 e8 11	 shr	 eax, 17
  00217	83 e0 01	 and	 eax, 1

; 2281 :     return cast_or_null<Expr>(SubExprs[Arg + getNumPreArgs() + PREARGS_START]);

  0021a	ff c0		 inc	 eax
  0021c	03 c6		 add	 eax, esi
  0021e	48 8b 0c c1	 mov	 rcx, QWORD PTR [rcx+rax*8]
  00222	e8 00 00 00 00	 call	 ??$cast_or_null@VExpr@clang@@VStmt@2@@llvm@@YAPEAVExpr@clang@@PEAVStmt@2@@Z ; llvm::cast_or_null<clang::Expr,clang::Stmt>
; File c:\src\chromium3\src\third_party\llvm\tools\clang\lib\ast\stmtprofile.cpp

; 1416 :       if ((THIS & 0xFFFFFFF0FFFF0000) == 0x7ff000000000)

  00227	48 b9 00 00 00
	00 f0 7f 00 00	 mov	 rcx, 140668768878592	; 00007ff000000000H
  00231	48 39 4d 40	 cmp	 QWORD PTR tv13670[rbp-88], rcx - load and check the guaranteed-bad masked 'this' pointer (THIS)
  00235	75 01		 jne	 SHORT $LN2227@Visit

; 1417 :         __debugbreak();

  00237	cc		 int	 3


And here are the variable offsets that show that those two variables overlap:

tv14635 = 136
tv13858 = 136
tv13737 = 136
tv13736 = 136
tv13720 = 136
tv13718 = 136
tv13717 = 136
tv13715 = 136
tv13714 = 136
tv13700 = 136
tv13699 = 136
tv13695 = 136
$T15 = 136
$T16 = 136
UnaryOp$17 = 136

Deleted: StmtProfile.zip 1.1 MB

StmtProfile.zip 1.1 MB Download

Good news/bad news:

Good: Microsoft shared a fixed c2.dll. After reverting the workaround (and verifying that the crash was still happening) I copied over the new amd64\c2.dll and rebuilt clang (del *.obj /s & ninja, is there a better way?) and the old failure is gone. But there's a new failure, that is hit after about 5,500 build steps of chrome:

Can anyone tell if this is possibly a known clang issue? It could be a new VC++ bug, it could mean that the old VC++ bug is not fully fixed, or it could be a clang bug. I can investigate but I want to make sure this isn't a known issue. Here's the assertion failure output:

ninja -v -d keeprsp -C out\clang_tot obj/third_party/angle/libANGLE/Caps.obj
ninja.exe -v -d keeprsp -C out\clang_tot obj/third_party/angle/libANGLE/Caps.obj -j 46 -l 48
ninja: Entering directory `out\clang_tot'
[1 processes, 1/1 @ 0.2/s : 4.732s ] ../../third_party/llvm-build/Release+Asserts/bin/clang-cl.exe /nologo /showIncludes  @obj/third_party/angle/libANGLE/Caps.obj.rsp /c ../../third_party/angle/src/libANGLE/Caps.cpp /Foobj/third_party/angle/libANGLE/Caps.obj /Fd"obj/third_party/angle/libANGLE_cc.pdb"
FAILED: obj/third_party/angle/libANGLE/Caps.obj
../../third_party/llvm-build/Release+Asserts/bin/clang-cl.exe /nologo /showIncludes  @obj/third_party/angle/libANGLE/Caps.obj.rsp /c ../../third_party/angle/src/libANGLE/Caps.cpp /Foobj/third_party/angle/libANGLE/Caps.obj /Fd"obj/third_party/angle/libANGLE_cc.pdb"
Assertion failed: Val && "isa<> used on a null pointer", file C:\src\chromium3\src\third_party\llvm\include\llvm/Support/Casting.h, line 106
Wrote crash dump file "c:\src\Temp\clang-cl.exe-d4113f.dmp"
#0 0x00007ff6f6338775 HandleAbort c:\src\chromium3\src\third_party\llvm\lib\support\windows\signals.inc:411:0
#1 0x00007ff6f7b99769 raise minkernel\crts\ucrt\src\appcrt\misc\signal.cpp:516:0
#2 0x00007ff6f7b86df8 abort minkernel\crts\ucrt\src\appcrt\startup\abort.cpp:71:0
#3 0x00007ff6f7b86532 common_assert_to_stderr<wchar_t> minkernel\crts\ucrt\src\appcrt\startup\assert.cpp:149:0
#4 0x00007ff6f7b865ce _wassert minkernel\crts\ucrt\src\appcrt\startup\assert.cpp:404:0
#5 0x00007ff6f5ed6de3 llvm::GEPOperator::hasAllZeroIndices(void)const  c:\src\chromium3\src\third_party\llvm\include\llvm\ir\operator.h:456:0
#6 0x00007ff6f5ed32fe `anonymous namespace'::stripPointerCastsAndOffsets<1> c:\src\chromium3\src\third_party\llvm\lib\ir\value.cpp:483:0
#7 0x00007ff6f61bd100 isOverwrite c:\src\chromium3\src\third_party\llvm\lib\transforms\scalar\deadstoreelimination.cpp:326:0
#8 0x00007ff6f61b9bd6 eliminateDeadStores c:\src\chromium3\src\third_party\llvm\lib\transforms\scalar\deadstoreelimination.cpp:1131:0
#9 0x00007ff6f61ba7d7 eliminateDeadStores c:\src\chromium3\src\third_party\llvm\lib\transforms\scalar\deadstoreelimination.cpp:1265:0
#10 0x00007ff6f61bf55c `anonymous namespace'::DSELegacyPass::runOnFunction c:\src\chromium3\src\third_party\llvm\lib\transforms\scalar\deadstoreelimination.cpp:1308:0
#11 0x00007ff6f5ee35fd llvm::FPPassManager::runOnFunction(class llvm::Function &) c:\src\chromium3\src\third_party\llvm\lib\ir\legacypassmanager.cpp:1514:0
#12 0x00007ff6f5bdae07 `anonymous namespace'::CGPassManager::RunPassOnSCC c:\src\chromium3\src\third_party\llvm\lib\analysis\callgraphsccpass.cpp:149:0
#13 0x00007ff6f5bdab93 `anonymous namespace'::CGPassManager::RunAllPassesOnSCC c:\src\chromium3\src\third_party\llvm\lib\analysis\callgraphsccpass.cpp:418:0
#14 0x00007ff6f5bdba59 `anonymous namespace'::CGPassManager::runOnModule c:\src\chromium3\src\third_party\llvm\lib\analysis\callgraphsccpass.cpp:474:0
#15 0x00007ff6f5ee3a3e `anonymous namespace'::MPPassManager::runOnModule c:\src\chromium3\src\third_party\llvm\lib\ir\legacypassmanager.cpp:1591:0
#16 0x00007ff6f5ee2ef1 llvm::legacy::PassManagerImpl::run(class llvm::Module &) c:\src\chromium3\src\third_party\llvm\lib\ir\legacypassmanager.cpp:1695:0
#17 0x00007ff6f653810c `anonymous namespace'::EmitAssemblyHelper::EmitAssembly c:\src\chromium3\src\third_party\llvm\tools\clang\lib\codegen\backendutil.cpp:788:0
#18 0x00007ff6f653a140 clang::EmitBackendOutput(class clang::DiagnosticsEngine &,class clang::HeaderSearchOptions const &,class clang::CodeGenOptions const &,class clang::TargetOptions const &,class clang::LangOptions const &,class llvm::DataLayout const &,class llvm::Module *,enum clang::BackendAction,class std::unique_ptr<class llvm::raw_pwrite_stream,struct std::default_delete<class llvm::raw_pwrite_stream> >) c:\src\chromium3\src\third_party\llvm\tools\clang\lib\codegen\backendutil.cpp:1148:0
#19 0x00007ff6f7cc3a31 clang::BackendConsumer::HandleTranslationUnit(class clang::ASTContext &) c:\src\chromium3\src\third_party\llvm\tools\clang\lib\codegen\codegenaction.cpp:265:0
#20 0x00007ff6f68c6cbc clang::MultiplexConsumer::HandleTranslationUnit(class clang::ASTContext &) c:\src\chromium3\src\third_party\llvm\tools\clang\lib\frontend\multiplexconsumer.cpp:305:0
#21 0x00007ff6f6f224d6 clang::ParseAST(class clang::Sema &,bool,bool) c:\src\chromium3\src\third_party\llvm\tools\clang\lib\parse\parseast.cpp:162:0
#22 0x00007ff6f688654d clang::ASTFrontendAction::ExecuteAction(void) c:\src\chromium3\src\third_party\llvm\tools\clang\lib\frontend\frontendaction.cpp:1005:0
#23 0x00007ff6f7cc30fb clang::CodeGenAction::ExecuteAction(void) c:\src\chromium3\src\third_party\llvm\tools\clang\lib\codegen\codegenaction.cpp:992:0
#24 0x00007ff6f68863db clang::FrontendAction::Execute(void) c:\src\chromium3\src\third_party\llvm\tools\clang\lib\frontend\frontendaction.cpp:906:0
#25 0x00007ff6f6873b65 clang::CompilerInstance::ExecuteAction(class clang::FrontendAction &) c:\src\chromium3\src\third_party\llvm\tools\clang\lib\frontend\compilerinstance.cpp:981:0
#26 0x00007ff6f68f7bfd clang::ExecuteCompilerInvocation(class clang::CompilerInstance *) c:\src\chromium3\src\third_party\llvm\tools\clang\lib\frontendtool\executecompilerinvocation.cpp:252:0
#27 0x00007ff6f4fc5ff4 cc1_main(class llvm::ArrayRef<char const *>,char const *,void *) c:\src\chromium3\src\third_party\llvm\tools\clang\tools\driver\cc1_main.cpp:221:0
#28 0x00007ff6f4fc1d40 ExecuteCC1Tool c:\src\chromium3\src\third_party\llvm\tools\clang\tools\driver\driver.cpp:313:0
#29 0x00007ff6f4fc3b3a main c:\src\chromium3\src\third_party\llvm\tools\clang\tools\driver\driver.cpp:387:0
#30 0x00007ff6f7b6349d __scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283:0
#31 0x00007ff855298364 (C:\WINDOWS\System32\KERNEL32.DLL+0x8364)
#32 0x00007ff855e870d1 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x670d1)
clang-cl.exe: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 6.0.0 (trunk 310083)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\src\chromium3\src\third_party\llvm-build\Release+Asserts\bin
clang-cl.exe: note: diagnostic msg: PLEASE submit a bug report to http://llvm.org/bugs/ and include the crash backtrace, preprocessed source, and associated run script.
clang-cl.exe: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-cl.exe: note: diagnostic msg: c:\src\Temp\Caps-3874c6.sh
clang-cl.exe: note: diagnostic msg:

********************
ninja: build stopped: subcommand failed.
C:\src\chromium3\src>type out\clang_tot\args.gn
clang_use_chrome_plugins = false
is_clang = true
llvm_force_head_revision = true
is_component_build = false
is_debug = false
is_official_build = true
symbol_level = 1

And the relevant environment variables:
set GYP_MSVS_VERSION=2017
set LLVM_FORCE_HEAD_REVISION=1
set vs2017_install=C:\Program Files (x86)\Microsoft Visual Studio\Preview\Professional
set DEPOT_TOOLS_WIN_TOOLCHAIN=0

I don't see any issues filed about this yet, but it looks like a null deref in some new code:
https://reviews.llvm.org/D30703

Your version is "trunk 310083", and that landed at r310055, so that's where my money is.

That crash might be https://bugs.llvm.org/show_bug.cgi?id=34074

Reverted 310055 in 310123, you could try syncing again. But chances are things are good now.

Thanks for the pointers and the quick revert. I synced to 310124, and then built tip-of-tree chrome with tip-of-tree clang built with tip-of-tree VC++. It worked.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1a56a637268fb862ea7c4fbfd8fc2d89c4b9eff4

commit 1a56a637268fb862ea7c4fbfd8fc2d89c4b9eff4
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Fri Aug 25 18:56:35 2017

VS 2017 Update 3.2

This change switches the VS 2017 package to use the 10.0.15063.468 SDK
and VS 2017 Update 3.2 (includes fix to code-gen bug hit when building
clang with VC++ 2017).

Packaging was done on a Windows Server 2016 VM, cleanly created for this
purpose.

Compiler was packaged up by downloading VS 2017 Update 3.2, from
https://www.visualstudio.com/vs/, and then running this
command (executable name was different):

    vs_professional.exe --add Microsoft.VisualStudio.Workload.NativeDesktop
    --add Microsoft.VisualStudio.Component.VC.ATLMFC --includeRecommended
    --passive

Then the Windows 10.0.15063.468 SDK installer was used to install the
Debuggers package.

Then the packaging script was run like this:

    python depot_tools\win_toolchain\package_from_installed.py 2017 -w 10.0.15063.0

VS 2015 builds are still the default so this makes no immediate difference.

R=scottmg@chromium.org
BUG= 683729 , 727447 

Change-Id: Ie615737503ff3a502c6c4df3651c7511db901ffc
Reviewed-on: https://chromium-review.googlesource.com/628243
Reviewed-by: Scott Graham <scottmg@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#497477}
[modify] https://crrev.com/1a56a637268fb862ea7c4fbfd8fc2d89c4b9eff4/build/vs_toolchain.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ec9a1e6ce2708f93ea08f82dd76a1f556310af91

commit ec9a1e6ce2708f93ea08f82dd76a1f556310af91
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Fri Aug 25 23:55:03 2017

Change default VC++ compiler from 2015 to 2017

Now that VC++ 2017 15.3.2 is packaged up we can try again to have VC++
2017 as the default compiler. This will be reverted at the end of the
weekend. If the clang workaround for 727447 is reverted then this will
also test whether that code-gen bug is fixed.

R=thakis@chromium.org
BUG= 683729 , 727447 

Change-Id: I74b070a0f8b5c58348309e1bea553136499c6922
Reviewed-on: https://chromium-review.googlesource.com/636325
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#497599}
[modify] https://crrev.com/ec9a1e6ce2708f93ea08f82dd76a1f556310af91/build/vs_toolchain.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/43b6c849b3c49f71261a71fcb40455d0e263c49d

commit 43b6c849b3c49f71261a71fcb40455d0e263c49d
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Mon Aug 28 01:12:50 2017

Revert "Change default VC++ compiler from 2015 to 2017"

This reverts commit ec9a1e6ce2708f93ea08f82dd76a1f556310af91.

Reason for revert: Weekend testing of VC++ 2017 is complete. A canary
was produced (62.0.3197.0) and a bug was found ( crbug.com/759402 ), so
it was a worthwhile experiment.

Original change's description:
> Change default VC++ compiler from 2015 to 2017
> 
> Now that VC++ 2017 15.3.2 is packaged up we can try again to have VC++
> 2017 as the default compiler. This will be reverted at the end of the
> weekend. If the clang workaround for 727447 is reverted then this will
> also test whether that code-gen bug is fixed.
> 
> R=​thakis@chromium.org
> BUG= 683729 , 727447 
> 
> Change-Id: I74b070a0f8b5c58348309e1bea553136499c6922
> Reviewed-on: https://chromium-review.googlesource.com/636325
> Reviewed-by: Nico Weber <thakis@chromium.org>
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#497599}

TBR=thakis@chromium.org,brucedawson@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  683729 ,  727447 
Change-Id: I9bef79f3197597fea6009692ec96809ff7ecf560
Reviewed-on: https://chromium-review.googlesource.com/637046
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#497684}
[modify] https://crrev.com/43b6c849b3c49f71261a71fcb40455d0e263c49d/build/vs_toolchain.py