Detailed report: https://cluster-fuzz.appspot.com/testcase?key=5181685140553728

Fuzzer: ochang_image_mutator
Job Type: windows_asan_chrome
Platform Id: windows

Crash Type: UNKNOWN
Crash Address: 0x8e37383b
Crash State:
  C:\windows\SYSTEM32\ntdll.dll
  C:\windows\SYSTEM32\SHELL32.dll
  C:\windows\SYSTEM32\SHELL32.dll
  
Recommended Security Severity: Medium


Unminimized Testcase: https://cluster-fuzz.appspot.com/download/AMIfv94sz6oE57gzM5P5jM7-KifZ4v6xcUz74DyRw_RCeAkJmFtoxgvriC2e5O3CMz5SM9AoTXAnVYJO1k6hq-6Zd57UH5p5CV1J-7hMOkoX098W0-q2tCig6bZ16rQT43jdkAHgnWqUsPPN9Y88Z0FIto1InNuPyg?testcase_id=5181685140553728


Issue manually filed by: inferno

See https://dev.chromium.org/Home/chromium-security/bugs/reproducing-clusterfuzz-bugs for more information.

I'm guessing this is because the version number got bumped to 4.0.0 and something still expects the asan runtime at 3.9.0 (the version number is part of the path). Is that possible?

The TOT bots are happy.

If I download the build linked from the report and run chrome.exe locally, the output is


C:\src\chrome\src>"c:\Users\thakis\Downloads\media%2Fwin32-release%2Fasan-win32-release-410400\asan-win32-release-410400
\chrome.exe"
=================================================================
==15160==ERROR: AddressSanitizer: global-buffer-overflow on address 0x01496d4b at pc 0x012a8358 bp 0x03c3e9e8 sp 0x03c3e
9d4
READ of size 13 at 0x01496d4b thread T0
    #0 0x12a8372 in __asan_wrap_memcmp e:\b\build\slave\win_upload_clang\build\src\third_party\llvm\projects\compiler-rt
\lib\sanitizer_common\sanitizer_common_interceptors.inc:629
    #1 0x10aa60 in crashpad::getopt+0x9d0 (c:\Users\thakis\Downloads\media%2Fwin32-release%2Fasan-win32-release-410400\a
san-win32-release-410400\chrome.exe+0x44aa60)
    #2 0x10b25a in crashpad::getopt_long+0x18 (c:\Users\thakis\Downloads\media%2Fwin32-release%2Fasan-win32-release-4104
00\asan-win32-release-410400\chrome.exe+0x44b25a)
    #3 0x56e73c in crashpad::HandlerMain+0x62c (c:\Users\thakis\Downloads\media%2Fwin32-release%2Fasan-win32-release-410
400\asan-win32-release-410400\chrome.exe+0x8ae73c)
    #4 0x4652f0 in crash_reporter::RunAsCrashpadHandler+0xb10 (c:\Users\thakis\Downloads\media%2Fwin32-release%2Fasan-wi
n32-release-410400\asan-win32-release-410400\chrome.exe+0x7a52f0)
    #5 0xc6310 in main+0x3a0 (c:\Users\thakis\Downloads\media%2Fwin32-release%2Fasan-win32-release-410400\asan-win32-rel
ease-410400\chrome.exe+0x406310)
    #6 0x12c9710 in __scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:255
    #7 0x75893389 in BaseThreadInitThunk+0x11 (C:\Windows\syswow64\kernel32.dll+0x7dd73389)
    #8 0x77c69901 in RtlInitializeExceptionChain+0x62 (C:\Windows\SysWOW64\ntdll.dll+0x7dea9901)
    #9 0x77c698d4 in RtlInitializeExceptionChain+0x35 (C:\Windows\SysWOW64\ntdll.dll+0x7dea98d4)

0x01496d4b is located 53 bytes to the left of global variable '<string literal>' defined in '../../third_party/crashpad/
crashpad/handler/handler_main.cc:175:8' (0x1496d80) of size 9
  '<string literal>' is ascii string 'database'
0x01496d4b is located 0 bytes to the right of global variable '<string literal>' defined in '../../third_party/crashpad/
crashpad/handler/handler_main.cc:174:8' (0x1496d40) of size 11
  '<string literal>' is ascii string 'annotation'
SUMMARY: AddressSanitizer: global-buffer-overflow e:\b\build\slave\win_upload_clang\build\src\third_party\llvm\projects\
compiler-rt\lib\sanitizer_common\sanitizer_common_interceptors.inc:629 in __asan_wrap_memcmp
Shadow bytes around the buggy address:
  0x30292d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x30292d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x30292d70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x30292d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x30292d90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x30292da0: 00 00 f9 f9 f9 f9 f9 f9 00[03]f9 f9 f9 f9 f9 f9
  0x30292db0: 00 01 f9 f9 f9 f9 f9 f9 00 00 01 f9 f9 f9 f9 f9
  0x30292dc0: 00 06 f9 f9 f9 f9 f9 f9 00 02 f9 f9 f9 f9 f9 f9
  0x30292dd0: 04 f9 f9 f9 f9 f9 f9 f9 05 f9 f9 f9 f9 f9 f9 f9
  0x30292de0: 00 f9 f9 f9 f9 f9 f9 f9 01 f9 f9 f9 f9 f9 f9 f9
  0x30292df0: 00 00 00 00 f9 f9 f9 f9 00 00 04 f9 f9 f9 f9 f9
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==15160==ABORTING
[0809/094944:ERROR:file_io.cc(30)] read: expected 4, observed 0
[0809/094945:ERROR:main_dll_loader_win.cc(125)] Failed to load Chrome DLL from c:\Users\thakis\Downloads\media%2Fwin32-r
elease%2Fasan-win32-release-410400\asan-win32-release-410400\chrome.dll: One or more arguments are invalid (0x80000003)
=================================================================
==12432==ERROR: AddressSanitizer: access-violation on unknown address 0x8e37383b (pc 0x8e37383b bp 0x03c2f640 sp 0x03c2f
628 T0)
    #0 0x8e37383a  (<unknown module>)
    #1 0x77c694c4 in RtlIsCurrentThreadAttachExempt+0x5e (C:\Windows\SysWOW64\ntdll.dll+0x7dea94c4)
    #2 0x77c88ae9 in LdrShutdownProcess+0x96 (C:\Windows\SysWOW64\ntdll.dll+0x7dec8ae9)
    #3 0x77c88a35 in RtlExitUserProcess+0x73 (C:\Windows\SysWOW64\ntdll.dll+0x7dec8a35)
    #4 0x758979ec in ExitProcess+0x14 (C:\Windows\syswow64\kernel32.dll+0x7dd779ec)
    #5 0x1310736 in exit_or_terminate_process d:\th\minkernel\crts\ucrt\src\appcrt\startup\exit.cpp:129
    #6 0x13106cb in common_exit d:\th\minkernel\crts\ucrt\src\appcrt\startup\exit.cpp:269
    #7 0x131084e in exit d:\th\minkernel\crts\ucrt\src\appcrt\startup\exit.cpp:282
    #8 0x12c972b in __scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:264
    #9 0x75893389 in BaseThreadInitThunk+0x11 (C:\Windows\syswow64\kernel32.dll+0x7dd73389)
    #10 0x77c69901 in RtlInitializeExceptionChain+0x62 (C:\Windows\SysWOW64\ntdll.dll+0x7dea9901)
    #11 0x77c698d4 in RtlInitializeExceptionChain+0x35 (C:\Windows\SysWOW64\ntdll.dll+0x7dea98d4)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: access-violation (<unknown module>)
==12432==ABORTING

+scottmg since crashpad is on the stack

You can ignore c#4, those stuff we usually ignore with ASAN_OPTIONS=strict_memcmp=0 in default configuration.

I did try with a local build and i don't see any crash in c#0 and c#2. So, this is something only happening on ClusterFuzz bots :( (Win Server 2012).

Oh wait, sorry, i didn't see the end part of stack in c#4, that is the same one on CF bot. The first part of stack can be disabled with strict_memcmp=0.

[0809/094945:ERROR:main_dll_loader_win.cc(125)] Failed to load Chrome DLL from c:\Users\thakis\Downloads\media%2Fwin32-r
elease%2Fasan-win32-release-410400\asan-win32-release-410400\chrome.dll: One or more arguments are invalid (0x80000003)
=================================================================
==12432==ERROR: AddressSanitizer: access-violation on unknown address 0x8e37383b (pc 0x8e37383b bp 0x03c2f640 sp 0x03c2f
628 T0)
    #0 0x8e37383a  (<unknown module>)
    #1 0x77c694c4 in RtlIsCurrentThreadAttachExempt+0x5e (C:\Windows\SysWOW64\ntdll.dll+0x7dea94c4)
    #2 0x77c88ae9 in LdrShutdownProcess+0x96 (C:\Windows\SysWOW64\ntdll.dll+0x7dec8ae9)

Is it possible to ssh there? Probably best if some clusterfuzz problem first tracks down what they problem is since you guys are a lot more familiar with that environment?

My local compiled build is ok, but archived builds are broken, e.g. tried with https://storage.cloud.google.com/chromium-browser-asan/win32-release/asan-win32-release-410615.zip and ASAN_OPTIONS=strict_memcmp=0, i can see the same CF stack.

[0809/072034:ERROR:main_dll_loader_win.cc(125)] Failed to load Chrome DLL from C:\Users\aarya\Downloads\win32-release%2Fasan-win32-release-410615\asan-win32-release-410615\chrome.dll: One or more arguments are invalid (0x80000003)
=================================================================
==7868==ERROR: AddressSanitizer: access-violation on unknown address 0x8e22ecd5 (pc 0x8e22ecd5 bp 0x0024fa74 sp 0x0024fa5c T0)
SCARINESS: 10 (signal)
    #0 0x8e22ecd4  (<unknown module>)
    #1 0x77319e04  (C:\Windows\SysWOW64\ntdll.dll+0x7de99e04)
    #2 0x77339429  (C:\Windows\SysWOW64\ntdll.dll+0x7deb9429)
    #3 0x77339375  (C:\Windows\SysWOW64\ntdll.dll+0x7deb9375)
    #4 0x750979ec  (C:\Windows\syswow64\kernel32.dll+0x7dd779ec)
    #5 0x2363142 in exit_or_terminate_process d:\th\minkernel\crts\ucrt\src\appcrt\startup\exit.cpp:129
    #6 0x23630d7 in common_exit d:\th\minkernel\crts\ucrt\src\appcrt\startup\exit.cpp:269
    #7 0x236329e in exit d:\th\minkernel\crts\ucrt\src\appcrt\startup\exit.cpp:282
    #8 0x231c21b in _scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:264
    #9 0x75093389  (C:\Windows\syswow64\kernel32.dll+0x7dd73389)
    #10 0x7731a241  (C:\Windows\SysWOW64\ntdll.dll+0x7de9a241)
    #11 0x7731a214  (C:\Windows\SysWOW64\ntdll.dll+0x7de9a214)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: access-violation (<unknown module>) 
==7868==ABORTING

I don't know if it's related, but compared to http://commondatastorage.googleapis.com/chromium-browser-clang/Win/clang-274369-7.tgz , the files in lib/clang/4.0.0/lib/windows are 2-5x as large now. clang_rt.asan-i386.lib grew from 3MB to 6MB.

Filed https://llvm.org/bugs/show_bug.cgi?id=28916 for the size increase. Probably not related though.

> are 2-5x as large now

If I'm right, we added debug information into it to be able to debug.
As we turned on code-view, this may increase the binary size.
The debug information is not in the PDB on the side.

inferno: Do the bots run with handle_sigill set to 1? I think it defaults to 0, and sigill handling on Windows is new in this roll.

etienneb: Thanks that makes sense.

https://cluster-fuzz.appspot.com/testcase?key=5181685140553728 prints the ASAN_OPTIONS used and it doesn't mention handle_sigill (and locally setting it to 0 does nothing), so that's probably not it.

locally my chrome.dll is 294 mb, whereas in archive it is 861 mb. I think this is some issue with large size of chrome.dll in archived builds. my local build gn flags are the same as lkgr builder and i clobbered twice just to make nothing was special in my local build.

Did you build with goma locally? Setting use_goma turns off symbols (https://cs.chromium.org/chromium/src/build/config/compiler/BUILD.gn?q=use_goma+file:%5C.gn&sq=package:chromium&l=1481&dr=C) so maybe that's related somehow? The bot uses goma.

This stack looks something like the issue I fixed in LLVM r277478, where ntdll was calling some function we hooked in the ASan runtime *after* we unloaded the asan runtime DLL.

This can't be the exact same issue because chrome statically links the asan runtime, but it could be some other kind of issue where one of our interceptor hooks has gone wrong during shutdown.

Anyway, I'm building chrome and I'll try to repro.

I can't repro this in my local build either, like inferno. It does repro with the build archived at https://cluster-fuzz.appspot.com/testcase?key=5181685140553728

(I'm now building with goma, on the off chance that it's caused by that. Other than that, my local gn config was identical to the bot's, and that didn't let it repro locally)

I looked at the build in the archive, and it's full of debug info that shouldn't be there:

$ dumpbin -summary chrome.exe
...
      539000 .data
       35000 .debug_a
      561000 .debug_i
      9F7000 .debug_l
        1000 .debug_m
      7AE000 .debug_r
      1F7000 .debug_s
        1000 .gfids
      468000 .rdata
      3BB000 .reloc
       1F000 .rsrc
     1274000 .text
        1000 .tls
        1000 CPADinfo

I think somehow clang on goma decided it was supposed to emit DWARF instead of CodeView. Those section names look like the usual DWARF section names (.debug_info, .debug_line, .debug_string, etc) were truncated to 8 characters, which is a limitation of the PE section table.

This would explain why the binaries are comically large and why I can't debug the archived build.

I _can_ repro this on my local build if I build with goma. So this is due to goma + asan somehow. (the tot bots don't use goma.)

As a workaround, we could disable goma on the lkgr bots until we understand what's going on and have a fix (possibly in goma).

As mentioned in comment 16, if goma is one we don't pass /Zi (https://cs.chromium.org/chromium/src/build/config/compiler/BUILD.gn?q=use_goma+file:%5C.gn&sq=package:chromium&l=1481&dr=C). Does clang-cl default to dwarf debug info?

I verified locally that 'clang-cl -gline-tables-only t.cpp -o t.obj' produces codeview.

What about `C:\goma\goma-win64/gomacc.exe ...clang-cl -gline-tables-only t.cpp -o t.obj` (use the clang binary in your chrome checkout, else goma doesn't have it)

Had to install goma, but as suspected, it uses DWARF:
$ dumpbin -summary t.obj
...
           0 .bss
           0 .data
          14 .debug_abbrev
          2A .debug_info
          5B .debug_line
           0 .debug_loc
           1 .debug_macinfo
           0 .debug_ranges
          3B .debug_str
          30 .drectve
          30 .pdata
          6C .text
          20 .xdata

I'm not going to do anything since it sounds like you're getting close, but let me know if I can help. (Sorry if Crashpad is doing something dumb. It might be that that function is pretty close to the first allocations after main() that happen in the process.)

@scottmg if you can run down that memcmp issue in crashpad's getopt, that'd be great, I keep forgetting to set ASAN_OPTIONS=strict_memcmp=0 in my environment when I run asanified chrome.

---

My local verification in c#22 was wrong. This is *not* goma related, Chromium's current `clang-cl -gline-tables-only` will generate DWARF. I had a local patch to fix this, which I am committing ASAP.

The DWARF thing is still kind of a red herring, though. It's not clear if it's directly related to the crash or not.

backfill from irc: clang-cl -gline-tables-only currently produces dwarf. rnk had a local patch that made that not the case he forgot about. so goma doesn't change that bit.

I'm currently trying to do a local build without /Zi, this will also give me dwarf debug info without goma. If that repros the crash, the crash is somehow due to the dwarf debug info. (Unlikely.)

Enabling goma also disables precompiled headers. I'll try a local build without pch later and see how that does.

Finally, someone could do a remote goma build with /Z7 passed together with -gline-tables-only. If that still repros the crash, they'd have real debug info then.

fyi +mark on Crashpad's getopt (see c#4).

I don't get the output from #4 with the build from the report, I get

[0809/101334:ERROR:main_dll_loader_win.cc(125)] Failed to load Chrome DLL from d:\src\getopt-asan\asan-win32-release-410400\chrome.dll: A
dynamic link library (DLL) initialization routine failed. (0x45A)

But I think crashpad_handler is crashing without a report. Is it possible to run an asan build in the debugger? I get a lot of int 3 in InterceptHooks() so I can't actually get to the real problem.

This is really common: InterceptHooks.

I recently patched the code so it's failing when the hooking is incorrect.
This was to bring asan on 64-bits, but some issue were found on 32-bits too.

This means, the library (CRT) is not the same. And, ASAN is having issue to hook. It's easy to figure out if we have an access (a debugger hooked).

And I wonder if we should not output the "bytes" when we can hook. This may ease debugging.

-> when we CAN'T hook.

Scratch that, I can get it in windbg. In GN is_asan=true is_clang=true enough to repro this build?

i.e.:

[4d87f4e...]d:\src\cr3\src>gn gen //out/RelClang "--args=is_debug=false is_chrome_branded=false is_component_build=false symbol_level=2 target_cpu=\"x86\" is_asan=true is_clang=true" --check
Done. Made 4917 targets from 984 files in 2887ms

[4d87f4e...]d:\src\cr3\src>ninja -C out\RelClang chrome

scottmg: Yes. (Haven't tried symbol_level=2, but I think it should be fine.)

(https://www.chromium.org/developers/testing/addresssanitizer#TOC-GYP-Windows-32-bit-build could use updating when things are stable.)

Ok I _can_ repro the crash in my local build that does not use goma but also does not pass /Zi [1]

So this seems to be triggered by dwarf debug info somehow, and goma is innocent (and just happens to imply "no /Zi"). The question why this only happens after the clang roll is still open.



1: 
C:\src\chrome\src>git diff
diff --git a/build/config/compiler/BUILD.gn b/build/config/compiler/BUILD.gn
index 588db2b..101217b 100644
--- a/build/config/compiler/BUILD.gn
+++ b/build/config/compiler/BUILD.gn
@@ -1484,7 +1484,7 @@ config("symbols") {
       # consumption and link times unsustainable ( crbug.com/630074 ).
       cflags = []
     } else {
-      cflags = [ "/Zi" ]  # Produce PDB file, no edit and continue.
+      cflags = []  # Produce PDB file, no edit and continue.
     }
     if (is_win_fastlink && visual_studio_version != "2013") {
       # Tell VS 2015+ to create a PDB that references debug

Thanks Nico.

Looking into getopt now for Crashpad() but there seems to be a bunch of asan UAFs in tools while building 'chrome', so I'm not sure whether it'll help to fix just this one.

Huh, asan should only be used for the default toolchain. Is your checkout up-to-date? asan/win works in gn as of last thursday (
https://codereview.chromium.org/2216183002/)

I get a lot of these, that's what you mean isn't expected?

[20962->9459/30454 ~33] ACTION //third_party/ffmpeg:ffmpeg_yasm_action(//build/toolchain/win:clang_x86)
=================================================================
==4188==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x00b64b48 in thread T0
    #0 0x108d18a in free e:\b\build\slave\win_upload_clang\build\src\third_party\llvm\projects\compiler-rt\lib\asan\asan_malloc_win.cc:45
    #1 0x74470ba1 in ltoa+0x1a1 (C:\WINDOWS\System32\ucrtbase.dll+0x10030ba1)
    #2 0x774a9db2 in RtlProcessFlsData+0x122 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2a9db2)
    #3 0x774aa032 in LdrShutdownProcess+0x82 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2aa032)
    #4 0x774a9c75 in RtlExitUserProcess+0x95 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2a9c75)
    #5 0x73feadc2 in ExitProcess+0x12 (C:\WINDOWS\System32\KERNEL32.DLL+0x6b82adc2)
    #6 0x10bf19e in exit_or_terminate_process d:\th\minkernel\crts\ucrt\src\appcrt\startup\exit.cpp:129
    #7 0x10bf134 in common_exit d:\th\minkernel\crts\ucrt\src\appcrt\startup\exit.cpp:269
    #8 0x10bf2fa in exit d:\th\minkernel\crts\ucrt\src\appcrt\startup\exit.cpp:282
    #9 0x10a8c56 in __scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:264
    #10 0x73fd62c3 in BaseThreadInitThunk+0x23 (C:\WINDOWS\System32\KERNEL32.DLL+0x6b8162c3)
    #11 0x774e0608 in RtlSubscribeWnfStateChangeNotification+0x438 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2e0608)
    #12 0x774e05d3 in RtlSubscribeWnfStateChangeNotification+0x403 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4b2e05d3)

AddressSanitizer can not describe address in more detail (wild memory access suspected).
SUMMARY: AddressSanitizer: bad-free e:\b\build\slave\win_upload_clang\build\src\third_party\llvm\projects\compiler-rt\lib\asan\asan_malloc_win.cc:45 in free

Maybe I need to sync?

[4d87f4e...]d:\src\cr3\src>python tools\clang\scripts\update.py
Updating Clang to 277962-1...
Clang is already up to date.

Let's move that discussion to bug 635990

Things work fine if I build with -gline-tables-only /Z7.

So for the asan bots I think the best workaround for now is to pass /Z7 with -gline-tables-only, which should cure the crashes.

We still don't understand what causes the crash, or why it only happens after the clang roll.

https://codereview.chromium.org/2224073003/ forces /Z7 with -gline-tables-only.

inferno, do you happen to have a build to a clusterfuzz build built with the old clang, so that I can pull that for comparison?

How strict is the strict memcmp()?

The getopt() that we use in Crashpad on Windows is third_party code. I can see how its uses of memcmp() might violate strict checks. If “strict” means that [s1, s1 + n) and [s2, s2 + n) each need to be initialized, then this is probably it.

ASan doesn't care about initialization, just bounds, so it's probably complaining about something like this:
  memcmp("abcd", "ac", 4)

This will reliably return -1, "abcd" comes before "ac", but it may do an OOB read on "ac".

For C#41, Nico, here is one - https://www.googleapis.com/download/storage/v1/b/chromium-browser-asan/o/win32-release%2Fasan-win32-release-410269.zip?generation=1470526560494000&alt=media (this is before clang roll r410289). All builds can be browsed at http://commondatastorage.googleapis.com/chromium-browser-asan/index.html?prefix=win32-release/

#43 Reid, yes, that’s what I see in Crashpad’s getopt(). Thanks.

Comparing https://www.googleapis.com/download/storage/v1/b/chromium-browser-asan/o/win32-release%2Fasan-win32-release-410269.zip?generation=1470526560494000&alt=media and https://storage.cloud.google.com/chrome-test-builds/media/win32-release/asan-win32-release-410400.zip , the binaries look pretty similar as far as I can see...

I guess worst case we could bisect, but the bisection is "1. build clang 2. build chrome" so it'd take a while.

I'm out for the next 10 days. https://codereview.chromium.org/2224073003/ should heal the bots (if it doesn't, disable goma on them). Hopefully, when I'm back someone has figured out what's going on here :-)

I think it broke between r408781 and r409964.

asan-win32-release-408692.zip	2016-07-29 22:06:20	1708.49MB	408692	3a796de6b43352fb756aa1c9c24a8bc5fc481553
[DIR]	asan-win32-release-408734.zip	2016-07-30 00:04:56	1707.06MB	408734	a8e4ad8678760fe8d08c24ad1788347990042b34
[DIR]	asan-win32-release-408781.zip	2016-07-30 02:01:04	1707.06MB	408781	59e7b40948030815a49af887b922b370ba048b8b
[DIR]	asan-win32-release-409964.zip	2016-08-05 04:34:26	2555.79MB	409964	3a6cbe4c1dab38c5e9094a3ce6164b902c319bf2
[DIR]	asan-win32-release-409973.zip	2016-08-05 06:22:34	2555.75MB	409973	4f70f4aa155228044960e6521266c22e9f4e5539

Thanks for https://codereview.chromium.org/2224073003/, that should bring us back to normal. I will take a look again at the bots to confirm if it worked.

That size jump might be the gn switch.

https://www.googleapis.com/download/storage/v1/b/chromium-browser-asan/o/win32-release%2Fasan-win32-release-410269.zip?generation=1470526560494000&alt=media which was before the clang roll and which works fine is already 2.5 GB.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/314120a8cfd5df331d825566d25db1ba1a1983fe

commit 314120a8cfd5df331d825566d25db1ba1a1983fe
Author: Nico Weber <thakis@chromium.org>
Date: Tue Aug 09 20:01:04 2016

On asan/win bots building with goma, don't accidentally emit DWARF

This happens to work around a bug we don't yet understand, but seems
like a good change in its own right too.

BUG= 635715 
R=rnk@chromium.org

Review URL: https://codereview.chromium.org/2224073003 .

Cr-Commit-Position: refs/heads/master@{#410785}

[modify] https://crrev.com/314120a8cfd5df331d825566d25db1ba1a1983fe/build/config/sanitizers/BUILD.gn

This is now fixed, we are seeing good reports, yay! thanks Nico and Reid for debugging help here.

bug 635990 will track memcmp issue in crashpad.
 bug 636212  will track the size regression.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ed9ae8359c2706c16ca4159f0f2f063aa2034e23

commit ed9ae8359c2706c16ca4159f0f2f063aa2034e23
Author: thakis <thakis@chromium.org>
Date: Thu Aug 25 16:35:45 2016

Revert of On asan/win bots building with goma, don't accidentally emit DWARF (patchset #1 id:1 of https://codereview.chromium.org/2224073003/ )

Reason for revert:
We have now rolled past r278139 and this shouldn't be necessary any more.

Original issue's description:
> On asan/win bots building with goma, don't accidentally emit DWARF
>
> This happens to work around a bug we don't yet understand, but seems
> like a good change in its own right too.
>
> BUG= 635715 
> R=rnk@chromium.org
>
> Committed: https://crrev.com/314120a8cfd5df331d825566d25db1ba1a1983fe
> Cr-Commit-Position: refs/heads/master@{#410785}

TBR=rnk@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 635715 

Review-Url: https://codereview.chromium.org/2281583002
Cr-Commit-Position: refs/heads/master@{#414454}

[modify] https://crrev.com/ed9ae8359c2706c16ca4159f0f2f063aa2034e23/build/config/sanitizers/BUILD.gn