I've been soaking pairs of tests with this little script https://chromium-review.googlesource.com/c/chromium/src/+/646851 on a very slow VM.
I'm not sure if this one is the task scheduler or misuse of the api in the timer sequence test yet.
scottmg@humid:/usr/local/google/cr-perry/src$ cat out/asan/args.gn
is_asan = true
enable_nacl = false
is_debug = false
use_goma = true
scottmg@humid:/usr/local/google/cr-perry/src$ out/asan/base_unittests --gtest_filter=TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed:TimerSequenceTest.OneShotTimerTaskOnPoolThread --gtest_shuffle --gtest_repeat=100 --gtest_break_on_failure 2>&1 | tools/valgrind/asan/asan_symbolize.py
IMPORTANT DEBUGGING NOTE: batches of tests are run inside their
own process. For debugging a test inside a debugger, use the
--gtest_filter=<your_test_name> flag along with
--single-process-tests.
Using sharding settings from environment. This is shard 0/1
Using 1 parallel jobs.
Note: Google Test filter = TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed:TimerSequenceTest.OneShotTimerTaskOnPoolThread
Note: Randomizing tests' orders with a seed of 20486 .
[==========] Running 2 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from TimerSequenceTest
[ RUN ] TimerSequenceTest.OneShotTimerTaskOnPoolThread
[ OK ] TimerSequenceTest.OneShotTimerTaskOnPoolThread (4 ms)
[----------] 1 test from TimerSequenceTest (4 ms total)
[----------] 1 test from TaskSchedulerSingleThreadTaskRunnerManagerTest
[ RUN ] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed
[ OK ] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed (16 ms)
[----------] 1 test from TaskSchedulerSingleThreadTaskRunnerManagerTest (16 ms total)
[----------] Global test environment tear-down
[==========] 2 tests from 2 test cases ran. (20 ms total)
[ PASSED ] 2 tests.
[1/2] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed (16 ms)
[2/2] TimerSequenceTest.OneShotTimerTaskOnPoolThread (4 ms)
SUCCESS: all tests passed.
Note: Google Test filter = TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed:TimerSequenceTest.OneShotTimerTaskOnPoolThread
Note: Randomizing tests' orders with a seed of 20664 .
[==========] Running 2 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from TimerSequenceTest
[ RUN ] TimerSequenceTest.OneShotTimerTaskOnPoolThread
[ OK ] TimerSequenceTest.OneShotTimerTaskOnPoolThread (13 ms)
[----------] 1 test from TimerSequenceTest (13 ms total)
[----------] 1 test from TaskSchedulerSingleThreadTaskRunnerManagerTest
[ RUN ] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed
[ OK ] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed (58 ms)
[----------] 1 test from TaskSchedulerSingleThreadTaskRunnerManagerTest (59 ms total)
[----------] Global test environment tear-down
[==========] 2 tests from 2 test cases ran. (73 ms total)
[ PASSED ] 2 tests.
[1/2] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed (58 ms)
[2/2] TimerSequenceTest.OneShotTimerTaskOnPoolThread (13 ms)
SUCCESS: all tests passed.
Note: Google Test filter = TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed:TimerSequenceTest.OneShotTimerTaskOnPoolThread
Note: Randomizing tests' orders with a seed of 20846 .
[==========] Running 2 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from TaskSchedulerSingleThreadTaskRunnerManagerTest
[ RUN ] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed
[ OK ] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed (23 ms)
[----------] 1 test from TaskSchedulerSingleThreadTaskRunnerManagerTest (23 ms total)
[----------] 1 test from TimerSequenceTest
[ RUN ] TimerSequenceTest.OneShotTimerTaskOnPoolThread
[ OK ] TimerSequenceTest.OneShotTimerTaskOnPoolThread (2 ms)
[----------] 1 test from TimerSequenceTest (2 ms total)
[----------] Global test environment tear-down
[==========] 2 tests from 2 test cases ran. (27 ms total)
[ PASSED ] 2 tests.
[1/2] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed (23 ms)
[2/2] TimerSequenceTest.OneShotTimerTaskOnPoolThread (2 ms)
SUCCESS: all tests passed.
Note: Google Test filter = TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed:TimerSequenceTest.OneShotTimerTaskOnPoolThread
Note: Randomizing tests' orders with a seed of 21009 .
[==========] Running 2 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from TaskSchedulerSingleThreadTaskRunnerManagerTest
[ RUN ] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed
[ OK ] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed (11 ms)
[----------] 1 test from TaskSchedulerSingleThreadTaskRunnerManagerTest (11 ms total)
[----------] 1 test from TimerSequenceTest
[ RUN ] TimerSequenceTest.OneShotTimerTaskOnPoolThread
[ OK ] TimerSequenceTest.OneShotTimerTaskOnPoolThread (2 ms)
[----------] 1 test from TimerSequenceTest (2 ms total)
[----------] Global test environment tear-down
[==========] 2 tests from 2 test cases ran. (13 ms total)
[ PASSED ] 2 tests.
[1/2] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed (11 ms)
[2/2] TimerSequenceTest.OneShotTimerTaskOnPoolThread (2 ms)
SUCCESS: all tests passed.
Note: Google Test filter = TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed:TimerSequenceTest.OneShotTimerTaskOnPoolThread
Note: Randomizing tests' orders with a seed of 21160 .
[==========] Running 2 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from TaskSchedulerSingleThreadTaskRunnerManagerTest
[ RUN ] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed
[ OK ] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed (22 ms)
[----------] 1 test from TaskSchedulerSingleThreadTaskRunnerManagerTest (22 ms total)
[----------] 1 test from TimerSequenceTest
[ RUN ] TimerSequenceTest.OneShotTimerTaskOnPoolThread
=================================================================
==2172==ERROR: AddressSanitizer: heap-use-after-free on address 0x6150000055a8 at pc 0x000002432764 bp 0x7fdaa4875d10 sp 0x7fdaa4875d08
WRITE of size 4 at 0x6150000055a8 thread T3 (TaskSchedulerSi)
#0 0x2432763 in fetch_add /usr/local/google/cr-perry/src/out/asan/../../buildtools/third_party/libc++/trunk/include/atomic:1014:17
#1 0x2432763 in Barrier_AtomicIncrement /usr/local/google/cr-perry/src/out/asan/../../base/atomicops_internals_portable.h:91:0
#2 0x2432763 in DecrementNumPendingUndelayedTasks /usr/local/google/cr-perry/src/out/asan/../../base/task_scheduler/task_tracker.cc:521:0
#3 0x2432763 in base::internal::TaskTracker::RunNextTask(base::internal::Sequence*) /usr/local/google/cr-perry/src/out/asan/../../base/task_scheduler/task_tracker.cc:256:0
#4 0x24245e0 in base::internal::SchedulerWorker::Thread::ThreadMain() /usr/local/google/cr-perry/src/out/asan/../../base/task_scheduler/scheduler_worker.cc:73:34
#5 0x244e2e4 in base::(anonymous namespace)::ThreadFunc(void*) /usr/local/google/cr-perry/src/out/asan/../../base/threading/platform_thread_posix.cc:75:13
#6 0x7fdaad4d3183 in start_thread /build/eglibc-SvCtMH/eglibc-2.19/nptl/pthread_create.c:312:0
0x6150000055a8 is located 40 bytes inside of 472-byte region [0x615000005580,0x615000005758)
freed by thread T0 here:
#0 0x69edd2 in operator delete(void*) ??:0:0
#1 0x228f9f8 in DeleteSelf_ /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/include/gtest/gtest.h:453:24
#2 0x228f9f8 in HandleExceptionsInMethodIfSupported<testing::Test, void> /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:2456:0
#3 0x228f9f8 in testing::TestInfo::Run() /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:2659:0
#4 0x2290c76 in testing::TestCase::Run() /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:2772:28
#5 0x22a6d96 in testing::internal::UnitTestImpl::RunAllTests() /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:4677:43
#6 0x22a6308 in HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:2456:12
#7 0x22a6308 in testing::UnitTest::Run() /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:4285:0
#8 0x2556892 in RUN_ALL_TESTS /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/include/gtest/gtest.h:2237:46
#9 0x2556892 in base::TestSuite::Run() /usr/local/google/cr-perry/src/out/asan/../../base/test/test_suite.cc:270:0
#10 0x2573f64 in Run /usr/local/google/cr-perry/src/out/asan/../../base/callback.h:92:12
#11 0x2573f64 in base::(anonymous namespace)::LaunchUnitTestsInternal(base::RepeatingCallback<int ()> const&, unsigned long, int, bool, base::RepeatingCallback<void ()> const&) /usr/local/google/cr-perry/src/out/asan/../../base/test/launcher/unit_test_launcher.cc:216:0
#12 0x2573b66 in base::LaunchUnitTests(int, char**, base::RepeatingCallback<int ()> const&) /usr/local/google/cr-perry/src/out/asan/../../base/test/launcher/unit_test_launcher.cc:475:10
#13 0x252ca3c in main /usr/local/google/cr-perry/src/out/asan/../../base/test/run_all_base_unittests.cc:12:10
#14 0x7fdaac0a0f44 in __libc_start_main /build/eglibc-SvCtMH/eglibc-2.19/csu/libc-start.c:287:0
previously allocated by thread T0 here:
#0 0x69e1b2 in operator new(unsigned long) ??:0:0
#1 0x1b434ff in testing::internal::TestFactoryImpl<base::internal::TaskSchedulerSingleThreadTaskRunnerManagerTest_DifferentThreadsUsed_Test>::CreateTest() /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/include/gtest/internal/gtest-internal.h:484:39
#2 0x228f7ee in HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test *> /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:2456:12
#3 0x228f7ee in testing::TestInfo::Run() /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:2645:0
#4 0x2290c76 in testing::TestCase::Run() /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:2772:28
#5 0x22a6d96 in testing::internal::UnitTestImpl::RunAllTests() /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:4677:43
#6 0x22a6308 in HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:2456:12
#7 0x22a6308 in testing::UnitTest::Run() /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:4285:0
#8 0x2556892 in RUN_ALL_TESTS /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/include/gtest/gtest.h:2237:46
#9 0x2556892 in base::TestSuite::Run() /usr/local/google/cr-perry/src/out/asan/../../base/test/test_suite.cc:270:0
#10 0x2573f64 in Run /usr/local/google/cr-perry/src/out/asan/../../base/callback.h:92:12
#11 0x2573f64 in base::(anonymous namespace)::LaunchUnitTestsInternal(base::RepeatingCallback<int ()> const&, unsigned long, int, bool, base::RepeatingCallback<void ()> const&) /usr/local/google/cr-perry/src/out/asan/../../base/test/launcher/unit_test_launcher.cc:216:0
#12 0x2573b66 in base::LaunchUnitTests(int, char**, base::RepeatingCallback<int ()> const&) /usr/local/google/cr-perry/src/out/asan/../../base/test/launcher/unit_test_launcher.cc:475:10
#13 0x252ca3c in main /usr/local/google/cr-perry/src/out/asan/../../base/test/run_all_base_unittests.cc:12:10
#14 0x7fdaac0a0f44 in __libc_start_main /build/eglibc-SvCtMH/eglibc-2.19/csu/libc-start.c:287:0
Thread T3 (TaskSchedulerSi) created by T0 here:
#0 0x65e7dd in __interceptor_pthread_create ??:0:0
#1 0x244d5fa in base::(anonymous namespace)::CreateThread(unsigned long, bool, base::PlatformThread::Delegate*, base::PlatformThreadHandle*, base::ThreadPriority) /usr/local/google/cr-perry/src/out/asan/../../base/threading/platform_thread_posix.cc:114:13
#2 0x2423b31 in Initialize /usr/local/google/cr-perry/src/out/asan/../../base/task_scheduler/scheduler_worker.cc:118:5
#3 0x2423b31 in base::internal::SchedulerWorker::Thread::Create(scoped_refptr<base::internal::SchedulerWorker>) /usr/local/google/cr-perry/src/out/asan/../../base/task_scheduler/scheduler_worker.cc:31:0
#4 0x24236ba in base::internal::SchedulerWorker::Start() /usr/local/google/cr-perry/src/out/asan/../../base/task_scheduler/scheduler_worker.cc:206:13
#5 0x241fb5e in CreateTaskRunnerWithTraitsImpl<base::internal::(anonymous namespace)::SchedulerWorkerDelegate> /usr/local/google/cr-perry/src/out/asan/../../base/task_scheduler/scheduler_single_thread_task_runner_manager.cc:452:13
#6 0x241fb5e in base::internal::SchedulerSingleThreadTaskRunnerManager::CreateSingleThreadTaskRunnerWithTraits(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, base::TaskTraits const&, base::SingleThreadTaskRunnerThreadMode) /usr/local/google/cr-perry/src/out/asan/../../base/task_scheduler/scheduler_single_thread_task_runner_manager.cc:395:0
#7 0x1b33cb1 in base::internal::TaskSchedulerSingleThreadTaskRunnerManagerTest_DifferentThreadsUsed_Test::TestBody() /usr/local/google/cr-perry/src/out/asan/../../base/task_scheduler/scheduler_single_thread_task_runner_manager_unittest.cc:104:13
#8 0x228dc1c in HandleExceptionsInMethodIfSupported<testing::Test, void> /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:2456:12
#9 0x228dc1c in testing::Test::Run() /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:2472:0
#10 0x228f8c4 in testing::TestInfo::Run() /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:2654:11
#11 0x2290c76 in testing::TestCase::Run() /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:2772:28
#12 0x22a6d96 in testing::internal::UnitTestImpl::RunAllTests() /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:4677:43
#13 0x22a6308 in HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:2456:12
#14 0x22a6308 in testing::UnitTest::Run() /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/src/gtest.cc:4285:0
#15 0x2556892 in RUN_ALL_TESTS /usr/local/google/cr-perry/src/out/asan/../../third_party/googletest/src/googletest/include/gtest/gtest.h:2237:46
#16 0x2556892 in base::TestSuite::Run() /usr/local/google/cr-perry/src/out/asan/../../base/test/test_suite.cc:270:0
#17 0x2573f64 in Run /usr/local/google/cr-perry/src/out/asan/../../base/callback.h:92:12
#18 0x2573f64 in base::(anonymous namespace)::LaunchUnitTestsInternal(base::RepeatingCallback<int ()> const&, unsigned long, int, bool, base::RepeatingCallback<void ()> const&) /usr/local/google/cr-perry/src/out/asan/../../base/test/launcher/unit_test_launcher.cc:216:0
#19 0x2573b66 in base::LaunchUnitTests(int, char**, base::RepeatingCallback<int ()> const&) /usr/local/google/cr-perry/src/out/asan/../../base/test/launcher/unit_test_launcher.cc:475:10
#20 0x252ca3c in main /usr/local/google/cr-perry/src/out/asan/../../base/test/run_all_base_unittests.cc:12:10
#21 0x7fdaac0a0f44 in __libc_start_main /build/eglibc-SvCtMH/eglibc-2.19/csu/libc-start.c:287:0
SUMMARY: AddressSanitizer: heap-use-after-free (/usr/local/google/cr-perry/src/out/asan/base_unittests+0x2432763)
Shadow bytes around the buggy address:
0x0c2a7fff8a60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2a7fff8a70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2a7fff8a80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2a7fff8a90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2a7fff8aa0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c2a7fff8ab0: fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd
0x0c2a7fff8ac0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2a7fff8ad0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2a7fff8ae0: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
0x0c2a7fff8af0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c2a7fff8b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==2172==ABORTING
[1/2] TaskSchedulerSingleThreadTaskRunnerManagerTest.DifferentThreadsUsed (22 ms)
[2/2] TimerSequenceTest.OneShotTimerTaskOnPoolThread (CRASHED)
1 test crashed:
TimerSequenceTest.OneShotTimerTaskOnPoolThread (../../base/timer/timer_unittest.cc:810)
Summary of all test iterations:
1 test crashed:
TimerSequenceTest.OneShotTimerTaskOnPoolThread (../../base/timer/timer_unittest.cc:810)
End of the summary.
Tests took 0 seconds.
scottmg@humid:/usr/local/google/cr-perry/src$
Comment 1 by scottmg@chromium.org
, Sep 7 2017