I was using component build.

This should only happen when /guard and incremental linking are both in use, which means only in release component builds. nektar@ was using such a build and chengx@ may have been. If there are any counter examples please let me know.

The other reporter thinks it quite likely that they were also doing a release component build. So, I am going to fix this bug by disabling /guard:cf (CFG) on component builds (crrev.com/c/920761).

I also filed a bug report:

https://developercommunity.visualstudio.com/content/problem/198903/guardcf-and-incremental-leads-to-random-failures.html

The recent resurgence of these bugs is because of restoring incremental linking in crrev.com/c/894448, which apparently was working around a different bug from its intended target.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dd23fadffd3a979017c07fda50828edf9c5583ae

commit dd23fadffd3a979017c07fda50828edf9c5583ae
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Thu Feb 15 15:48:51 2018

Disable CFG (/guard:cf) for component builds

There appears to be a bug in Microsoft's linker when using /guard:cf
with incremental linking. The table of functions occasionally gets
corrupted or not update which leads to a CFG violation when the OS
thinks that an invalid indirect branch is being taken. The stack shows:

    ntdll.dll!RtlFailFast2()
    ntdll.dll!RtlpHandleInvalidUserCallTarget()

The error code returned is 0xC0000409 which is
STATUS_STACK_BUFFER_OVERRUN which is quite non-obvious and confusing.

A bug in the linker with incremental linking and CFG seems quite
plausible, and that combination is quite worthless, so the fix is to not
use CFG in component builds.

Note that future occurrences of this bug, if any, will show an error
code of 0xC0000409 where they used to show -1073741819. This is due to
a separate change that alters how we print these error codes.

Bug:  812421 
Change-Id: I8042d4363ea93084ca56e0634124799183c4153c
Reviewed-on: https://chromium-review.googlesource.com/920761
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#537027}
[modify] https://crrev.com/dd23fadffd3a979017c07fda50828edf9c5583ae/build/config/win/BUILD.gn

For the record, since you asked, Bruce, here is my gn config where the issue showed up:

is_debug = false
is_component_build = true
use_goma = true
goma_dir = "c:\\src\\goma\\goma-win64"
proprietary_codecs = true
ffmpeg_branding = "Chrome"

This was reproducing consistently even after gn clean out/Release.  But after I removed out/Release and rebuilt, the issue went away.

Thanks, all of the evidence is consistent with this being a bug that shows up in release component builds. I *think* that my change should resolve this, but please let me know if it does not. If I don't hear any objections I'll wait a few days and then close this as fixed.

Do you think this might be due to a clang-cl bug somehow? The cfg and incremental flags are both link.exe flags, so it doesn't feel super likely, but the thought occurred to me.

It's possible that clang-cl is creating an object file that is unexpected by the linker. My brief attempts at reproing this incremental linking issue with cl.exe failed but I didn't retry with clang-cl.

I think the repro "should" be as simple as making an executable with some indirect calls (virtual function calls?), linking with /incremental and /cfg, and then modifying one of the target functions and relinking. But that didn't trigger anything and I didn't feel motivated to investigate why not.

Incremental and /cfg together is not an important combination so I'm okay with hiding the bug. I'm going to close this as fixed and we can reopen if more occurrences show up.

Note that these failures will now print as 0xC0000409 instead of -1073741819.

Woot! Thanks for fixing this.