rnk points out that these tests are relatively newly default-enabled: https://reviews.llvm.org/D24742

If the tests fail by default on some machines, that's not good. Here's the bot link for where they failed: http://build.chromium.org/p/tryserver.chromium.win/builders/win_upload_clang/builds/157

Ideas:
- Maybe the tests work better with win10? The bot is still on Win 7. (My workstation is also on Win7, I'll check if they pass there.)
- Maybe the compiler-rt testrunner could check if "enough" memory is available before running the tests for win64

In any case, we need to come up with something, this blocks rolling.

I excluded the asan tests from check-all on win64 in r298450.

I locally reverted your r298450. All the asan64 tests also fail on my Z840 win Win7. So maybe we could try enabling them if the windows version is high enough.

Do they run fine on the llvm win bot? Which windows version does that run?

I'm guessing you mean it runs fine on win10? The z840s also have quite a large amount of ram, so that's not that conclusive.

A good way to test this might be to stand up a Win10 GCE buildbot for upstream LLVM, connect it to the silent master, and reconfigure it with varying levels of RAM and swap to see what works.

I mean they fail on my desktop (win7 with lots of ram) and the trybot (win7 with small amounts of ram). They pass on your box which uses win10. They fail on http://lab.llvm.org:8011/builders/clang-x64-ninja-win7 which uses win7. Yes, it's fairly weak evidence.

...did anyone manually test the asan64 runtime on win7? Maybe it's just broken there?

They are working on 
 * my z640 win7
 * my z840 win10
 * my test laptop (low memory lenovo) with win7

Some tests were flaky if you do not increase the OS virtual memory limit.
But, IIRC, we disabled these tests a while ago.

LLVM/Clang ToT on win7 is ok:

D:\src\llvm\ninja64>ninja check-asan
[0/1] Re-running CMake...
-- Warning: Did not find file Compiler/MSVC-ASM
-- Native target architecture is X86
-- Threads enabled.
-- Doxygen disabled.
-- Sphinx disabled.
-- Go bindings disabled.
-- Could NOT find OCaml (missing:  OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH)
-- OCaml bindings disabled.
-- LLVM host triple: x86_64-pc-win32
-- LLVM default target triple: x86_64-pc-win32
-- Using Release VC++ CRT: MD
-- Constructing LLVMBuild project information
-- Could NOT find Subversion (missing:  Subversion_SVN_EXECUTABLE)
-- LLVMHello ignored -- Loadable modules not supported on this platform.
-- Targeting X86
-- Compiler-RT supported architectures: x86_64
-- Builtin supported architectures: x86_64
-- Clang version: 5.0.0
-- SampleAnalyzerPlugin ignored -- Loadable modules not supported on this platform.
-- PrintFunctionNames ignored -- Loadable modules not supported on this platform.
-- AnnotateFunctions ignored -- Loadable modules not supported on this platform.
-- BugpointPasses ignored -- Loadable modules not supported on this platform.
-- Configuring done
-- Generating done
-- Build files have been written to: D:/src/llvm/ninja64
[534/1933] Generating LLVMLTORevision.h
-- Could NOT find Subversion (missing:  Subversion_SVN_EXECUTABLE)
[1133/1933] Building CXX object projects\compiler-rt\lib\i...akeFiles\RTInterception.x86_64.dir\interception_win.cc.obj
C:\Program Files (x86)\Windows Kits\8.1\include\shared\minwindef.h(128): warning C4005: 'WINAPI': macro redefinition
d:\src\llvm\llvm\projects\compiler-rt\lib\sanitizer_common\sanitizer_win_defs.h(23): note: see previous definition of 'W
INAPI'
[1144/1933] Building CXX object projects\compiler-rt\lib\a...akeFiles\AsanDllThunk.x86_64.dir\asan_win_dll_thunk.cc.obj
D:\src\llvm\llvm\projects\compiler-rt\lib\asan\asan_win_dll_thunk.cc(66): warning C4392: 'void memcmp(void)': incorrect
number of arguments for intrinsic function, expected '3' arguments
D:\src\llvm\llvm\projects\compiler-rt\lib\asan\asan_win_dll_thunk.cc(67): warning C4392: 'void memcpy(void)': incorrect
number of arguments for intrinsic function, expected '3' arguments
D:\src\llvm\llvm\projects\compiler-rt\lib\asan\asan_win_dll_thunk.cc(69): warning C4392: 'void memset(void)': incorrect
number of arguments for intrinsic function, expected '3' arguments
D:\src\llvm\llvm\projects\compiler-rt\lib\asan\asan_win_dll_thunk.cc(70): warning C4392: 'void strcat(void)': incorrect
number of arguments for intrinsic function, expected '2' arguments
D:\src\llvm\llvm\projects\compiler-rt\lib\asan\asan_win_dll_thunk.cc(72): warning C4392: 'void strcmp(void)': incorrect
number of arguments for intrinsic function, expected '2' arguments
D:\src\llvm\llvm\projects\compiler-rt\lib\asan\asan_win_dll_thunk.cc(73): warning C4392: 'void strcpy(void)': incorrect
number of arguments for intrinsic function, expected '2' arguments
D:\src\llvm\llvm\projects\compiler-rt\lib\asan\asan_win_dll_thunk.cc(76): warning C4392: 'void strlen(void)': incorrect
number of arguments for intrinsic function, expected '1' arguments
D:\src\llvm\llvm\projects\compiler-rt\lib\asan\asan_win_dll_thunk.cc(86): warning C4392: 'void wcslen(void)': incorrect
number of arguments for intrinsic function, expected '1' arguments
[1152/1933] Building CXX object projects\compiler-rt\lib\a...timeThunk.x86_64.dir\asan_win_dynamic_runtime_thunk.cc.obj
C:\Program Files (x86)\Windows Kits\8.1\include\shared\minwindef.h(128): warning C4005: 'WINAPI': macro redefinition
D:\src\llvm\llvm\projects\compiler-rt\lib\sanitizer_common/sanitizer_win_defs.h(23): note: see previous definition of 'W
INAPI'
[1233/1933] Linking CXX static library lib\clang\5.0.0\lib\windows\clang_rt.profile-x86_64.lib
InstrProfilingMerge.c.obj : warning LNK4006: VPMergeHook already defined in InstrProfilingMergeFile.c.obj; second defini
tion ignored
[1235/1933] Linking CXX shared library lib\clang\5.0.0\lib\windows\clang_rt.asan_dynamic-x86_64.dll
   Creating library lib\clang\5.0.0\lib\windows\clang_rt.asan_dynamic-x86_64.lib and object lib\clang\5.0.0\lib\windows\
clang_rt.asan_dynamic-x86_64.exp
[1351/1932] Generating SVNVersion.inc
-- Found Subversion: D:/src/depot_tools/svn.bat (found version "1.6.6")
[1922/1932] cmd.exe /C "cd /D D:\src\llvm\ninja64\projects...inst-Test.exe -fms-compatibility-version=19.00.24215.1 -g"
   Creating library D:/src/llvm/ninja64/projects/compiler-rt/lib/asan/tests/default/Asan-x86_64-inline-Noinst-Test.lib a
nd object D:/src/llvm/ninja64/projects/compiler-rt/lib/asan/tests/default/Asan-x86_64-inline-Noinst-Test.exp
[1928/1932] cmd.exe /C "cd /D D:\src\llvm\ninja64\projects...compatibility-version=19.00.24215.1 -g -fsanitize=address"
   Creating library D:/src/llvm/ninja64/projects/compiler-rt/lib/asan/tests/default/Asan-x86_64-with-calls-Test.lib and
object D:/src/llvm/ninja64/projects/compiler-rt/lib/asan/tests/default/Asan-x86_64-with-calls-Test.exp
[1929/1932] cmd.exe /C "cd /D D:\src\llvm\ninja64\projects...inst-Test.exe -fms-compatibility-version=19.00.24215.1 -g"
   Creating library D:/src/llvm/ninja64/projects/compiler-rt/lib/asan/tests/default/Asan-x86_64-with-calls-Noinst-Test.l
ib and object D:/src/llvm/ninja64/projects/compiler-rt/lib/asan/tests/default/Asan-x86_64-with-calls-Noinst-Test.exp
[1931/1932] cmd.exe /C "cd /D D:\src\llvm\ninja64\projects...compatibility-version=19.00.24215.1 -g -fsanitize=address"
   Creating library D:/src/llvm/ninja64/projects/compiler-rt/lib/asan/tests/default/Asan-x86_64-inline-Test.lib and obje
ct D:/src/llvm/ninja64/projects/compiler-rt/lib/asan/tests/default/Asan-x86_64-inline-Test.exp
[1931/1932] Running the AddressSanitizer tests
-- Testing: 566 tests, 16 threads --
Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
Testing Time: 67.25s
  Expected Passes    : 386
  Expected Failures  : 17
  Unsupported Tests  : 163

ToT has the tests enabled unless you explicitly revert r298450, did you do that?

The same command causes lots of failures on my win7 box.

ToT : fresh checkout and build
I didn't revert anything

Then you didn't run the 64-bit tests, rnk disabled them in r298450.

got it, revert a revert!

On my side, that's the failing test on win7 64-bit

Failing Tests (1):
    AddressSanitizer-Unit :: Asan-x86_64-with-calls-Test.exe/AddressSanitizer.LargeMallocTest

and, it's flaky:

D:\src\llvm\ninja64>ninja check-asan
[3/5] cmd.exe /C "cd /D D:\src\llvm\ninja64\projects\compi...compatibility-version=19.00.24215.1 -g -fsanitize=address"
   Creating library D:/src/llvm/ninja64/projects/compiler-rt/lib/asan/tests/default/Asan-x86_64-with-calls-Test.lib and
object D:/src/llvm/ninja64/projects/compiler-rt/lib/asan/tests/default/Asan-x86_64-with-calls-Test.exp
[4/5] cmd.exe /C "cd /D D:\src\llvm\ninja64\projects\compi...compatibility-version=19.00.24215.1 -g -fsanitize=address"
   Creating library D:/src/llvm/ninja64/projects/compiler-rt/lib/asan/tests/default/Asan-x86_64-inline-Test.lib and obj
ct D:/src/llvm/ninja64/projects/compiler-rt/lib/asan/tests/default/Asan-x86_64-inline-Test.exp
[4/5] Running the AddressSanitizer tests
-- Testing: 566 tests, 16 threads --
Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
Testing Time: 64.70s
  Expected Passes    : 385
  Passes With Retry  : 1
  Expected Failures  : 17
  Unsupported Tests  : 163

Failed the first time.
Worked fine the 9 other times.

I guess then it's not due to win7. Over here, all the win64 asan tests fail reliably.

Retitling and removing blocker now that they're disabled again.

We need to figure out under which circumstances they work reliably, and only run them then, or something like that.

It may be related to win7.
Win7 needs memory pages to back-up the VirtualAlloc done for the shadow.
On Win10, these pages are allocated on-demand.

I've changes the default virtual memory limit on my computer to bypass the limitation. It won't solve the problem when my computer is under high memory pressure (tons of chrome tabs, compiling llvm, ...), but it's less flaky.

Here is how to repro a failure due to virtual memory limitation:

C:\Users\etienneb>systeminfo | grep -i Virtual
Virtual Memory: Max Size:  130,918 MB
Virtual Memory: Available: 94,528 MB
Virtual Memory: In Use:    36,390 MB



D:\src\SysInternals>testlimit64 -d -c 50000

Testlimit v5.24 - test Windows limits
Copyright (C) 2012-2015 Mark Russinovich
Sysinternals - www.sysinternals.com

Process ID: 37384

Leaking private bytes with touch (MB)...
Leaked 50000 MB of private memory (50000 MB total leaked). Lasterror: 0
The operation completed successfully.




Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
FAIL: AddressSanitizer-Unit :: Asan-x86_64-inline-Test.exe/AddressSanitizer.StrCmpOOBTest (565 of 566)
******************** TEST 'AddressSanitizer-Unit :: Asan-x86_64-inline-Test.exe/AddressSanitizer.StrCmpOOBTest' FAILED *
*******************
Note: Google Test filter = AddressSanitizer.StrCmpOOBTest
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from AddressSanitizer
[ RUN      ] AddressSanitizer.StrCmpOOBTest
D:/src/llvm/llvm/projects/compiler-rt/lib/asan/tests/asan_str_test.cc(328): error: Death test: Ident(StrCmp)(s1, s2 + si
ze)



So, virtual memory pressure is probably the reason is failiong on low-memory win7 computer.


We could claim it's supported for win10, and it's working on win7 (at your own risk).

I think we need to target ASAN 64-bit to Win10+, and simply drop support for Win7 and Win8/8.1. The problem is the shadow memory reservation causes GB of page table space to be allocated in the kernel. On Win10, the page table is implemented differently and only grows as memory is actually committed, and not only when reserved.

Perhaps we need detection and strong user friendly error messages? And potentially a flag to allow it to continue despite these warnings? (Sometimes it will work, if the conditions are right.)

There are similar issues around the hot patching logic being bound to certain versions of the runtime, and other issues related to the linking conditions (statis vs dynamic). In Syzygy we actually hard-coded the list of supported runtimes and would fail loudly if trying to run against an unknown and not explicitly supported runtime. We would add new runtime support with every toolchain version.

Supporting Win10+ sounds good to me.

I think we can change the ASan error message on VirtualAlloc failure to discuss virtual memory limitations in Win<10 instead of ulimit -v, which is obviously not relevant on Windows.

Issue has not been modified or commented on in the last 365 days, please re-open or file a new bug if this is still an issue.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Sigh, useless shheriffbot

Available, but no owner or component? Please find a component, as no one will ever find this without one.