Detailed report: https://cluster-fuzz.appspot.com/testcase?key=6634625733754880

Job Type: windows_asan_chrome
Platform Id: windows

Crash Type: 
Crash Address: 
Crash State:
  

Unminimized Testcase: https://cluster-fuzz.appspot.com/download/AMIfv971xcyY-z7rFTHjnm2JK-4jvZzQzB5vH3OyUIFYz9lOvwXrH9nwy6W8XX0ohiogL_SccN6tRrVICpdfgiSNbUd3JpwMx5gU7jPhT8ZJVjOsHKUlB5mtKAjdL_zn31ohGFNSLrP7FCBux3stCGI7uocohrZg0w?testcase_id=6634625733754880


See https://dev.chromium.org/Home/chromium-security/bugs/reproducing-clusterfuzz-bugs for more information.

I don't see any security crashes at all as well in last 7 days - https://cluster-fuzz.appspot.com/crashstats?fuzzer_name=All&job_type=windows_asan_chrome&security_flag=Both&date_start=2016-10-24&date_end=2016-11-01. So, it could be Win Clang ASAN is completely broken. 

I can look into this next Monday after the LLVM dev meeting.

Thanks Reid.

There's a bunch of low-hanging issues on the asan coverage bot that I'm going to look into first:
https://build.chromium.org/p/chromium.fyi/builders/CrWinAsanCov%20tester/builds/696

Essentially, that bot is red because dynamic-tools-team makes changes to sanitizer coverage and I don't have the bandwidth to keep it green by myself.

 Issue 637048  has been merged into this issue.

A (probably unrelated) bug was making it really hard to debug this. I fixed that in LLVM r286290. I'd recommend building chromium with that version of the asan runtime if anyone else is trying to debug this.

LLVM r286615 should fix a crash on startup in osmesa.dll. That should fix a lot of the test failures on the CrWinAsanCov bot.

I'm seeing a crash on startup in chrome.dll here:
  static void WriteJumpInstruction(uptr from, uptr target) {
    if (!DistanceIsWithin2Gig(from + kJumpInstructionLength, target))
      InterceptionFailed(__LINE__);
    ptrdiff_t offset = target - from - kJumpInstructionLength;
    *(u8*)from = 0xE9;
    *(u32*)(from + 1) = offset;
  }

Chrome is getting loaded at 0x7FFF0000, and the code we are patching is way up around 0x9780e7e6, and chrome.exe and the asan runtime is down at 0x00330000-0x010d8000.

chrome.dll itself is about 468MB. It's possible this is just too much code to work in a 32-bit address space.

The osmesa bug fix significantly cleaned up the waterfall:
https://build.chromium.org/p/chromium.fyi/builders/CrWinAsanCov%20tester/builds/714

Now there's only one remaining chrome_elf test failure, which is not coverage related, and is already filed here: https://bugs.chromium.org/p/chromium/issues/detail?id=646414

Waterfall aside, the actual chrome crash on startup still needs investigation.

Latest result from today's uploads.

1) Chrome's exception handler is kicking in first, so we dont get sanitizer stack - https://cluster-fuzz.appspot.com/v2/testcase-detail/5963159136632832.
2) Content shell job type is working fine - https://cluster-fuzz.appspot.com/v2/testcase-detail/5633286237061120
Haven't verified with a use-after-free or buffer overflow testcase. Don't even know why we are not crashing on startup on CF for Chrome.

This looks like it's because of crashpad. There's 3 calls to SetUnhandledExceptionFilter on process startup now:
1. CRT
2. ASan
3. crashpad

Crashpad's exception filter signals the crashpad process to take a crash dump and then waits indefinitely. It does not delegate the ASan's exception handler like the //base stack dumper does.

What should happen here? The easy thing to fix CF would be to disable crashpad in LLVM ASan builds. However, if we ship LLVM ASan to users, we need to wire the crash report through crashpad the way SyzyASan does.

Normally, Crashpad tries to avoid subsequent callers here: https://cs.chromium.org/chromium/src/chrome_elf/chrome_elf_main.cc?rcl=0&l=35. But it's not doing that at the moment due to https://bugs.chromium.org/p/chromium/issues/detail?id=655788, so I unfortunately can't say confidently who gets to last SetUEF, depending on what revision we're talking about. (I'm also not sure where asan tries to SetUEF.)

For dev builds using the ADDRESS_SANITIZER #define seems like the simplest way. I'm not sure about shipping to users. I feel like chrome_elf DllMain() is probably where that decision should happen though.

We are not shipping ASAN builds to users yet. Till we decide how to properly solve this, can we please disable this for ASAN builds to unblock fuzz testing on windows.

Maybe we should arrange for crashpad's exception filter to call ASan's crash handler. Then we can remove that ifdef I added in https://codereview.chromium.org/2220583002/.

I wanted to add a test this time around. I can see lots of examples of how to write a test that crashes a renderer by searching for kChromeUICrashURL, but I don't see any examples of capturing renderer logs. Are there any tools I should use to capture the renderer's stderr?

Maybe something like this one? https://cs.chromium.org/chromium/src/content/test/content_browser_test_test.cc?q=MANUAL_RendererCrash&sq=package:chromium&l=61

It's disabled for some silly reason, Dr. Memory bots or something (? I forget), but basically just a test that re-runs a MANUAL_ test capturing output.

John added base::GetAppOutputAndError to do that one; you might need a slight modification if you only want stderr.

It looks like content_browsertests doesn't use crashpad, so this wouldn't test the fix. Maybe just a unit test in crashpad would be enough. The crashpad multiprocess testing code looks nice, but I don't see an existing way to capture stderr, which is where the asan report is going to come from. Should I go ahead and add that functionality?

Well, anyway, this fixes the issue, but it could use a real test:
https://codereview.chromium.org/2504773002/

Also needs a clang roll, which is  bug 665063 .

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c9175dab5210cb05151ee27dc3e9bd54439650e2

commit c9175dab5210cb05151ee27dc3e9bd54439650e2
Author: rnk <rnk@chromium.org>
Date: Wed Nov 30 07:57:26 2016

Have crashpad call ASan's crash handler if present

This ensures that ClusterFuzz will see an ASan report for null
dereferences.

We can also remove chrome_elf's ifdefs after this change, since
we don't need ASan's call to SetUnhandledExceptionFilter to succeed.

R=mark@chromium.org
TBR=pennymac@chromium.org
BUG= 661209 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win10_chromium_x64_rel_ng

Review-Url: https://codereview.chromium.org/2504773002
Cr-Commit-Position: refs/heads/master@{#435147}

[modify] https://crrev.com/c9175dab5210cb05151ee27dc3e9bd54439650e2/chrome_elf/chrome_elf_main.cc
[modify] https://crrev.com/c9175dab5210cb05151ee27dc3e9bd54439650e2/third_party/crashpad/README.chromium
[modify] https://crrev.com/c9175dab5210cb05151ee27dc3e9bd54439650e2/third_party/crashpad/crashpad/client/crashpad_client_win.cc