New issue
Advanced search Search tips

Issue 914947 link

Starred by 3 users
Status: Fixed
Owner:
vtsyrklevich@chromium.org
Closed: Dec 21
Cc:
brucedaw...@chromium.org
tom....@microsoft.com
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 3
Type: Bug

Blocked on:
issue 904337
Blocking:
issue 915046



Sign in to add a comment

win-asan bot broke with updated VS toolchain

Project Member Reported by vtsyrklevich@chromium.org, Dec 13 
The win-asan bot broke after https://chromium-review.googlesource.com/c/chromium/src/+/1342814/10 landed. I believe the failure is because the _aligned_* UCRT functions, which are not hooked by ASan, were updated to use _msize_base instead of _msize which is also not hooked by the Windows version of ASan. (The originally blamed CL https://chromium-review.googlesource.com/c/1374959 is not actually pulled in on the ASan build.)

The bot failed with errors like the following [1]: 
[ RUN      ] WebMediaPlayerImplTest.LoadAndDestroy
=================================================================
==1436==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x1261265a06e7 at pc 0x7ff7b62b2df4 bp 0x00c7a44fcbe0 sp 0x00c7a44fcc20
READ of size 32768 at 0x1261265a06e7 thread T3
==1436==*** WARNING: Failed to initialize DbgHelp!              ***
==1436==*** Most likely this means that the app is already      ***
==1436==*** using DbgHelp, possibly with incompatible flags.    ***
==1436==*** Due to technical reasons, symbolization might crash ***
==1436==*** or produce wrong results.                           ***
    #0 0x7ff7b62b2e1c in __asan_wrap_memmove C:\b\rr\tmph59meq\w\src\third_party\llvm\projects\compiler-rt\lib\sanitizer_common\sanitizer_common_interceptors.inc:775
    #1 0x7ff7bd0736f0 in _aligned_realloc C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\minkernel\crts\ucrt\src\appcrt\heap\align.cpp:622
    #2 0x7ff7b68faf0e in av_realloc_f C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\third_party\ffmpeg\libavutil\mem.c:158
    #3 0x7ff7b68ee92a in ffio_rewind_with_probe_data C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\third_party\ffmpeg\libavformat\aviobuf.c:1136
    #4 0x7ff7b7a69966 in av_probe_input_buffer2 C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\third_party\ffmpeg\libavformat\format.c:304
    #5 0x7ff7b68c8b40 in avformat_open_input C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\third_party\ffmpeg\libavformat\utils.c:573
    #6 0x7ff7b37437f8 in media::FFmpegGlue::OpenContext C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\media\filters\ffmpeg_glue.cc:117
    #7 0x7ff7b343fe1a in base::internal::ReturnAsParamAdapter<bool> C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\post_task_and_reply_with_result_internal.h:22
    #8 0x7ff7b34402fc in base::internal::Invoker<base::internal::BindState<void (*)(base::OnceCallback<bool ()>, std::unique_ptr<bool,std::default_delete<bool> > *),base::OnceCallback<bool ()>,std::unique_ptr<bool,std::default_delete<bool> > *>,void ()>::RunOnce C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\bind_internal.h:658
    #9 0x7ff7b74d10cf in base::`anonymous namespace'::PostTaskAndReplyRelay::RunTaskAndPostReply C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\threading\post_task_and_reply_impl.cc:79
    #10 0x7ff7b74d1a15 in base::internal::Invoker<base::internal::BindState<void (*)(base::(anonymous namespace)::PostTaskAndReplyRelay),base::(anonymous namespace)::PostTaskAndReplyRelay>,void ()>::RunOnce C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\bind_internal.h:658
    #11 0x7ff7b90e3a13 in base::debug::TaskAnnotator::RunTask C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\debug\task_annotator.cc:99
    #12 0x7ff7b75181dc in base::internal::TaskTracker::RunOrSkipTask C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\task\task_scheduler\task_tracker.cc:647
    #13 0x7ff7b64b1cb7 in base::test::ScopedTaskEnvironment::TestTaskTracker::RunOrSkipTask C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\test\scoped_task_environment.cc:407
    #14 0x7ff7b7515fcd in base::internal::TaskTracker::RunAndPopNextTask C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\task\task_scheduler\task_tracker.cc:503
    #15 0x7ff7baf06dbb in base::internal::SchedulerWorker::RunWorker C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\task\task_scheduler\scheduler_worker.cc:337
    #16 0x7ff7baf06057 in base::internal::SchedulerWorker::RunPooledWorker C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\task\task_scheduler\scheduler_worker.cc:229
    #17 0x7ff7b6430bd8 in base::`anonymous namespace'::ThreadFunc C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\threading\platform_thread_win.cc:97
    #18 0x7ff7b62a6628 in __asan::AsanThread::ThreadStart C:\b\rr\tmph59meq\w\src\third_party\llvm\projects\compiler-rt\lib\asan\asan_thread.cc:262
    #19 0x7ffc75302773 in BaseThreadInitThunk+0x13 (C:\Windows\System32\KERNEL32.DLL+0x180012773)
    #20 0x7ffc77980d50 in RtlUserThreadStart+0x20 (C:\Windows\SYSTEM32\ntdll.dll+0x180070d50)

0x1261265a06e7 is located 0 bytes to the right of 2151-byte region [0x12612659fe80,0x1261265a06e7)
allocated by thread T3 here:
    #0 0x7ff7b62af110 in malloc C:\b\rr\tmph59meq\w\src\third_party\llvm\projects\compiler-rt\lib\asan\asan_malloc_win.cc:69
    #1 0x7ff7bd07359c in _aligned_realloc C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\minkernel\crts\ucrt\src\appcrt\heap\align.cpp:622
    #2 0x7ff7b68faf8a in av_reallocp C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\third_party\ffmpeg\libavutil\mem.c:174
    #3 0x7ff7b7a69a5a in av_probe_input_buffer2 C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\third_party\ffmpeg\libavformat\format.c:260
    #4 0x7ff7b68c8b40 in avformat_open_input C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\third_party\ffmpeg\libavformat\utils.c:573
    #5 0x7ff7b37437f8 in media::FFmpegGlue::OpenContext C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\media\filters\ffmpeg_glue.cc:117
    #6 0x7ff7b343fe1a in base::internal::ReturnAsParamAdapter<bool> C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\post_task_and_reply_with_result_internal.h:22
    #7 0x7ff7b34402fc in base::internal::Invoker<base::internal::BindState<void (*)(base::OnceCallback<bool ()>, std::unique_ptr<bool,std::default_delete<bool> > *),base::OnceCallback<bool ()>,std::unique_ptr<bool,std::default_delete<bool> > *>,void ()>::RunOnce C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\bind_internal.h:658
    #8 0x7ff7b74d10cf in base::`anonymous namespace'::PostTaskAndReplyRelay::RunTaskAndPostReply C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\threading\post_task_and_reply_impl.cc:79
    #9 0x7ff7b74d1a15 in base::internal::Invoker<base::internal::BindState<void (*)(base::(anonymous namespace)::PostTaskAndReplyRelay),base::(anonymous namespace)::PostTaskAndReplyRelay>,void ()>::RunOnce C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\bind_internal.h:658
    #10 0x7ff7b90e3a13 in base::debug::TaskAnnotator::RunTask C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\debug\task_annotator.cc:99
    #11 0x7ff7b75181dc in base::internal::TaskTracker::RunOrSkipTask C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\task\task_scheduler\task_tracker.cc:647
    #12 0x7ff7b64b1cb7 in base::test::ScopedTaskEnvironment::TestTaskTracker::RunOrSkipTask C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\test\scoped_task_environment.cc:407
    #13 0x7ff7b7515fcd in base::internal::TaskTracker::RunAndPopNextTask C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\task\task_scheduler\task_tracker.cc:503
    #14 0x7ff7baf06dbb in base::internal::SchedulerWorker::RunWorker C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\task\task_scheduler\scheduler_worker.cc:337
    #15 0x7ff7baf06057 in base::internal::SchedulerWorker::RunPooledWorker C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\task\task_scheduler\scheduler_worker.cc:229
    #16 0x7ff7b6430bd8 in base::`anonymous namespace'::ThreadFunc C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\threading\platform_thread_win.cc:97
    #17 0x7ff7b62a6628 in __asan::AsanThread::ThreadStart C:\b\rr\tmph59meq\w\src\third_party\llvm\projects\compiler-rt\lib\asan\asan_thread.cc:262
    #18 0x7ffc75302773 in BaseThreadInitThunk+0x13 (C:\Windows\System32\KERNEL32.DLL+0x180012773)
    #19 0x7ffc77980d50 in RtlUserThreadStart+0x20 (C:\Windows\SYSTEM32\ntdll.dll+0x180070d50)

Thread T3 created by T0 here:
    #0 0x7ff7b62a5460 in __asan_wrap_CreateThread C:\b\rr\tmph59meq\w\src\third_party\llvm\projects\compiler-rt\lib\asan\asan_win.cc:146
    #1 0x7ff7b643016e in base::`anonymous namespace'::CreateThreadInternal C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\threading\platform_thread_win.cc:136
    #2 0x7ff7baf04f8a in base::internal::SchedulerWorker::Start C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\task\task_scheduler\scheduler_worker.cc:79
    #3 0x7ff7b910493f in base::internal::SchedulerWorkerPoolImpl::SchedulerWorkerStarter::~SchedulerWorkerStarter C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\task\task_scheduler\scheduler_worker_pool_impl.cc:74
    #4 0x7ff7b9103b8a in base::internal::SchedulerWorkerPoolImpl::Start C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\task\task_scheduler\scheduler_worker_pool_impl.cc:303
    #5 0x7ff7b750b4e7 in base::internal::TaskSchedulerImpl::Start C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\task\task_scheduler\task_scheduler_impl.cc:197
    #6 0x7ff7b64b0a76 in base::test::ScopedTaskEnvironment::ScopedTaskEnvironment C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\test\scoped_task_environment.cc:172
    #7 0x7ff7b2728f08 in BlinkPlatformWithTaskEnvironment::BlinkPlatformWithTaskEnvironment C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\media\blink\run_all_unittests.cc:40
    #8 0x7ff7b2728cf1 in RunTests C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\media\blink\run_all_unittests.cc:80
    #9 0x7ff7b64b9c78 in base::`anonymous namespace'::LaunchUnitTestsInternal C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\test\launcher\unit_test_launcher.cc:225
    #10 0x7ff7b64b97f2 in base::LaunchUnitTests C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\base\test\launcher\unit_test_launcher.cc:575
    #11 0x7ff7b2728ad4 in main C:/b/swarming/w/ir/cache/builder/src/out/Release_x64\..\..\media\blink\run_all_unittests.cc:90
    #12 0x7ff7bd064263 in __scrt_common_main_seh d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #13 0x7ffc75302773 in BaseThreadInitThunk+0x13 (C:\Windows\System32\KERNEL32.DLL+0x180012773)
    #14 0x7ffc77980d50 in RtlUserThreadStart+0x20 (C:\Windows\SYSTEM32\ntdll.dll+0x180070d50)

SUMMARY: AddressSanitizer: heap-buffer-overflow C:\b\rr\tmph59meq\w\src\third_party\llvm\projects\compiler-rt\lib\sanitizer_common\sanitizer_common_interceptors.inc:775 in __asan_wrap_memmove
Shadow bytes around the buggy address:
  0x04734b234080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x04734b234090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x04734b2340a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x04734b2340b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x04734b2340c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x04734b2340d0: 00 00 00 00 00 00 00 00 00 00 00 00[07]fa fa fa
  0x04734b2340e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x04734b2340f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x04734b234100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x04734b234110: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x04734b234120: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==1436==ABORTING

[1] https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8927275104247189904/+/steps/media_blink_unittests_on_Windows-10-15063/0/logs/WebMediaPlayerImplTest.LoadAndDestroy/0

Comment 1 by brucedaw...@chromium.org, Dec 13

Cc: tom....@microsoft.com
CCing tom.tan@ because he is investigating a link failure in Windows ARM64 builds which might be vaguely related (or at least we ultimately want a fix that works for everything).

Comment 2 by vtsyrklevich@chromium.org, Dec 13 

Tom, what's the link failure you're seeing?

Comment 3 by brucedaw...@chromium.org, Dec 13 

When doing release arm64 builds for Win32 (requires the new toolchain or a local install of VS 2017 15.9):

ninja -C out\release_arm base_unittests
ninja: Entering directory `out\release_arm'
[116 processes, 595/711 @ 6.9/s : 86.473s ] LINK_MODULE(DLL) scoped_handle_test_dll.dll scoped_handle_test_dll.dll.pdb
FAILED: scoped_handle_test_dll.dll scoped_handle_test_dll.dll.pdb
c:/src/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../build/toolchain/win/tool_wrapper.py link-wrapper environment.arm64 False link.exe /nologo /DLL /OUT:./scoped_handle_test_dll.dll /PDB:./scoped_handle_test_dll.dll.pdb @./scoped_handle_test_dll.dll.rsp
libucrt.lib(msize.obj) : error LNK2005: _msize already defined in base.lib(allocator_shim.obj)
./scoped_handle_test_dll.dll : fatal error LNK1169: one or more multiply defined symbols found
[1 processes, 710/711 @ 4.1/s : 172.771s ] CXX obj/base/base_unittests/safe_numerics_unittest.obj
ninja: build stopped: subcommand failed.

This is related to a recent change where we started defining _msize - https://chromium-review.googlesource.com/c/chromium/src/+/1354219 - but you already know about that change.

Presumably the ARM64 msize.obj file in libucrt.lib contains another symbol that we are also pulling in. I have some tools that can help us find that (using /verbose linking) if help is needed.

Comment 4 by vtsyrklevich@chromium.org, Dec 13 

Hmm, sounds somewhat unrelated to this particular failure though perhaps there's some common justification for why MS made that change. Want to open a separate ticket and CC me?

Comment 5 by brucedaw...@chromium.org, Dec 13

Status: Assigned (was: Untriaged)
I filed  crbug.com/915016 . I also investigated briefly by linking with /verbose and that shows that _msize_base is what is pulling in msize.obj. So, in fact this bug appears to be a duplicate of 915016.

Comment 6 by vtsyrklevich@chromium.org, Dec 14 

Talked with bruce, the bugs are separate.

Had a bunch of issues getting an LLVM build and reproducing this locally but finally tested it and put out the trivial fix for review https://reviews.llvm.org/D55684

Comment 7 by brucedaw...@chromium.org, Dec 14

Blocking: 915046

Comment 8 by brucedaw...@chromium.org, Dec 14 

Thanks for the fix. I'd say that the root cause is the same, but the fix is different, but that's just semantics.

Comment 9 by vtsyrklevich@chromium.org, Dec 14

Blockedon: 904337

Comment 10 by vtsyrklevich@chromium.org, Dec 14 

Landed, now just blocked on a clang roll

Comment 11 by brucedaw...@chromium.org, Dec 19 

Please update (and close?) this when the next clang roll happens since I believe this is now the only blocker for  crbug.com/915046  (toolchain roll).

But, the roll will probably not be retried until after the holidays.

Comment 12 by vtsyrklevich@chromium.org, Dec 21

Status: Fixed (was: Assigned)
The clang roll seems to have stuck.
►

Sign in to add a comment