git revert ff69d3018c42e

and everything works normally again. Note that the issue happens when the build runs a built tool (genmacro) in a built step.

I typed step 2 wrong, should be

gn gen out/asan --args="is_component_build=true is_debug=true is_asan=true sanitizer_keep_symbols=true"

(I should clear my command history of these sorts of things...)

Can't repro on my box:

thakis@thakis:~/src/chrome/src$ gn gen out/asan --args="is_component_build=true is_debug=true is_asan=true sanitizer_keep_symbols=true"
Done. Made 6969 targets from 1381 files in 1633ms
thakis@thakis:~/src/chrome/src$  ninja -C out/asan/ paint_op_buffer_fuzzer
ninja: Entering directory `out/asan/'
[1800/1800] LINK ./paint_op_buffer_fuzzer
thakis@thakis:~/src/chrome/src$ git log | head -1
commit 5bf3f4f0a2defd3227f8d960825da523ceb6ddfa

$ head -3 /etc/lsb-release
# $Id: //depot/google3/googledata/corp/puppet/goobuntu/common/modules/base/templates/lsb-release.erb#2 $
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04 LTS"



Bots are happy too.

I have a hard time imagining how that change could affect asan.

So I just went and set everything up again to check, and maybe it wasn't this commit and was indeed a result of the clang roll.

If I revert the clang roll from (c4bc163e7ceb1ffc00fc20efbd92316635b984fe) and nostdlib++ (ff69d3018c42eb994533de337f81d417a0209913 , since the old clang didn't support this flag) and make sure to gclient sync, then everything works.

If I re-apply the clang roll (c4bc163e7ceb1ffc00fc20efbd92316635b984fe) but keep nostdlib++ reverted, I apparently still get this issue.

But this makes not a whole lot of sense, because I just asked someone next to me with a checkout from this morning to follow the reproducing steps and everything worked.

"worked" as in "they could repro" or "it worked fine for them"? If the latter, does it also work if they sync across the nostdlib++ change?

"worked" as in "it worked fine for them". I'll go see if I can find them again. I'll admit, after taking a better look into this (and blowing away my out directory a few times) it seems really weird.

Taking a quick look at the code that's asserting

https://github.com/llvm-mirror/compiler-rt/blob/master/lib/asan/asan_posix.cc#L102

this is a runtime assert on a translation unit global. Note that I get this failure immediately when just running

out/asan/genmodule

without any arguments or anything. I went and ran gdb with

gdb out/asan/genmodule
(gdb) catch syscall 60
(gdb) catch syscall 231
(gdb) run

and got the following

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
==36166==AddressSanitizer CHECK failed: /b/build/slave/linux_upload_clang/build/src/third_party/llvm/compiler-rt/lib/asan/asan_posix.cc:102 "((tsd_key_inited)) != (0)" (0x0, 0x0)
    <empty stack>


Catchpoint 7 (call to syscall exit_group), 0x00000000004d153b in __sanitizer::internal__exit(int) ()
(gdb) bt
#0  0x00000000004d153b in __sanitizer::internal__exit(int) ()
#1  0x00000000004d8914 in __sanitizer::Die() ()
#2  0x00000000004c3994 in __asan::AsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) ()
#3  0x00000000004d89e0 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) ()
#4  0x00000000004bfac3 in __asan::AsanTSDGet() ()
#5  0x00000000004c6f26 in __asan::GetCurrentTidOrInvalid() ()
#6  0x00000000004c1359 in __asan::ScopedInErrorReport::ScopedInErrorReport(bool) ()
#7  0x00000000004c033a in __asan::ReportFreeNotMalloced(unsigned long, __sanitizer::BufferedStackTrace*) ()
#8  0x00000000004bbc97 in free ()
#9  0x00007ffff6c235d5 in _dlerror_run (operate=operate@entry=0x7ffff6c23000 <dlsym_doit>, args=args@entry=0x7fffffffdb40) at dlerror.c:159
#10 0x00007ffff6c23068 in __dlsym (handle=<optimized out>, name=<optimized out>) at dlsym.c:70
#11 0x00000000004c7680 in __interception::GetRealFunctionAddress(char const*, unsigned long*, unsigned long, unsigned long) ()
#12 0x00000000004aad34 in InitializeCommonInterceptors() ()
#13 0x00000000004a901a in __asan::InitializeAsanInterceptors() ()
#14 0x00000000004c3490 in __asan::AsanInitInternal() ()
#15 0x00007ffff7de8a12 in _dl_init (main_map=0x7ffff7ffe170, argc=1, argv=0x7fffffffdc68, env=0x7fffffffdc78) at dl-init.c:105
#16 0x00007ffff7dd9c5a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#17 0x0000000000000001 in ?? ()
#18 0x00007fffffffe00c in ?? ()
#19 0x0000000000000000 in ?? ()

so apparently that happened before AsanTSDInit got called. Maybe I'm running into some startup order issue?

What happens if you run

thakis@thakis:~/src/chrome/src$ cat hi.cc
int main() {}
thakis@thakis:~/src/chrome/src$ third_party/llvm-build/Release+Asserts/bin/clang -fsanitize=address -o hi hi.cc
thakis@thakis:~/src/chrome/src$ ./hi
thakis@thakis:~/src/chrome/src$ echo $?
0


?

bungeman$ ./hi
==37405==AddressSanitizer CHECK failed: /b/build/slave/linux_upload_clang/build/src/third_party/llvm/compiler-rt/lib/asan/asan_allocator.cc:537 "((m->free_tid)) == ((kInvalidTid))" (0x0, 0xffffff)
    <empty stack>

bungeman$ echo $?
1

But with

 third_party/llvm-build/Release+Asserts/bin/clang -fsanitize=address -o hi hi.cc -nodefaultlibs -lc -lm -lgcc_s -lrt -lpthread -ldl -lrt


things work (??)

If so, please provide the output of `ldd hi` in both build modes.

Note that with the 'hi' example I get a slightly different stack trace

bungeman$ gdb hi 
Reading symbols from hi...(no debugging symbols found)...done.
(gdb) catch syscall 231
Catchpoint 1 (syscall 'exit_group' [231])
(gdb) run
Starting program: hi 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
==37584==AddressSanitizer CHECK failed: /b/build/slave/linux_upload_clang/build/src/third_party/llvm/compiler-rt/lib/asan/asan_allocator.cc:537 "((m->free_tid)) == ((kInvalidTid))" (0x0, 0xffffff)
    <empty stack>


Catchpoint 1 (call to syscall exit_group), 0x00000000004cf91b in __sanitizer::internal__exit(int) ()
(gdb) bt
#0  0x00000000004cf91b in __sanitizer::internal__exit(int) ()
#1  0x00000000004d6cf4 in __sanitizer::Die() ()
#2  0x00000000004c1d74 in __asan::AsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) ()
#3  0x00000000004d6dc0 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) ()
#4  0x000000000041df4b in __asan::Allocator::QuarantineChunk(__asan::AsanChunk*, void*, __sanitizer::BufferedStackTrace*) ()
#5  0x00000000004ba087 in free ()
#6  0x00007ffff74ad5d5 in _dlerror_run (operate=operate@entry=0x7ffff74ad000 <dlsym_doit>, args=args@entry=0x7fffffffdb60) at dlerror.c:159
#7  0x00007ffff74ad068 in __dlsym (handle=<optimized out>, name=<optimized out>) at dlsym.c:70
#8  0x00000000004c5a60 in __interception::GetRealFunctionAddress(char const*, unsigned long*, unsigned long, unsigned long) ()
#9  0x00000000004a9124 in InitializeCommonInterceptors() ()
#10 0x00000000004a740a in __asan::InitializeAsanInterceptors() ()
#11 0x00000000004c1870 in __asan::AsanInitInternal() ()
#12 0x00007ffff7de8a12 in _dl_init (main_map=0x7ffff7ffe170, argc=1, argv=0x7fffffffdc88, env=0x7fffffffdc98) at dl-init.c:105
#13 0x00007ffff7dd9c5a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#14 0x0000000000000001 in ?? ()
#15 0x00007fffffffe02c in ?? ()
#16 0x0000000000000000 in ?? ()


which is a different CHECK at https://github.com/llvm-mirror/compiler-rt/blob/master/lib/asan/asan_allocator.cc#L537 .

third_party/llvm-build/Release+Asserts/bin/clang -fsanitize=address -o hi hi.cc -nodefaultlibs -lc -lm -lgcc_s -lrt -lpthread -ldl -lrt

does not work (I get the same errors).

kcc, does this make any sense to you? (who's a good asan runtime contact nowadays?)

Could this be https://bugs.llvm.org/show_bug.cgi?id=34085 ? 
(I.e. bad LLVM revision, fixed in trunk)

> who's a good asan runtime contact nowadays?
I or anyone in my team

By any chance, is there any difference between:
third_party/llvm-build/Release+Asserts/bin/clang -fsanitize=address -o hi hi.cc -nodefaultlibs -lc -lrt
third_party/llvm-build/Release+Asserts/bin/clang -fsanitize=address -o hi hi.cc -nodefaultlibs -lrt -lc

It does sound like https://bugs.llvm.org/show_bug.cgi?id=34085 . Do you know why none of our bots is seeing this and why things are happy on my system?

From https://bugs.llvm.org/show_bug.cgi?id=34085 it looks like this was bisected on Aug 6. A workaround wasn't landed until a full day later [1].

Please, let's always keep LLVM head green and working. If someone bisects something to a commit of yours, revert first, investigate later. Like you'd do in any other project.

1: http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20170807/476890.html

Sorry that I didn't notice that in time.
Aug 6 was a Saturday and I reverted Sunday quite soon as I notice the report.

I was not able to reproduce this as well, only report from Roman.
I've speculatively reverted the line which was likely incorrect and it helped.

@17 Flipping -lc and -lrt doesn't seem to make a difference (the lines you propose by themselves don't link as they are missing symbols, but both fail the same way).

bungeman$ third_party/llvm-build/Release+Asserts/bin/clang -fsanitize=address -o hi hi.cc -nodefaultlibs -lrt -lc -lm -lgcc_s -lpthread -ldl
bungeman@phos:~/src/chromium/src$ ./hi
==42565==AddressSanitizer CHECK failed: /b/build/slave/linux_upload_clang/build/src/third_party/llvm/compiler-rt/lib/asan/asan_allocator.cc:537 "((m->free_tid)) == ((kInvalidTid))" (0x0, 0xffffff)
    <empty stack>

bungeman$ third_party/llvm-build/Release+Asserts/bin/clang -fsanitize=address -o hi hi.cc -nodefaultlibs -lc -lrt -lm -lgcc_s -lpthread -ldl
bungeman@phos:~/src/chromium/src$ ./hi
==42575==AddressSanitizer CHECK failed: /b/build/slave/linux_upload_clang/build/src/third_party/llvm/compiler-rt/lib/asan/asan_allocator.cc:537 "((m->free_tid)) == ((kInvalidTid))" (0x0, 0xffffff)
    <empty stack>



@18 From https://bugs.llvm.org/show_bug.cgi?id=34085#c14 it sounds like this may vary based on the dynamic linker and various other environment details, and I'm currently running a setup that's probably quite different from what most others working on Chromium have.



If this is affecting only me as the odd one out for the moment I'm fine waiting a short bit for the fix (I can work around locally by either backing up clang or compiling a new one for the time being).

We rolled in a new clang with the fix.