The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/81edcd62390e832531d79ec77c5cbd19fbd0327a

commit 81edcd62390e832531d79ec77c5cbd19fbd0327a
Author: reillyg <reillyg@chromium.org>
Date: Mon Mar 14 23:12:36 2016

Add suppression for net::ShouldSniffMimeType.

BUG= 594808 
TBR=thestig@chromium.org
NOTRY=true

Review URL: https://codereview.chromium.org/1802823002

Cr-Commit-Position: refs/heads/master@{#381106}

[modify] https://crrev.com/81edcd62390e832531d79ec77c5cbd19fbd0327a/tools/valgrind/drmemory/suppressions_full.txt

Could this be a Dr. Memory bug? Are there details on how to further diagnose?

Based on the line reported, this is uh... seemingly impossible, provided that //net's invariant that it's called on a single thread is true (and from the tests, this seems to be the case)

I'm building locally to see if I can reproduce. As you can see from the state of the memory.fyi waterfall I've just been trying to get these bots under control after the VS2015 upgrade.

Many tests are crashing at net\base\mime_sniffer.cc:831

https://build.chromium.org/p/chromium.memory.fyi/builders/Windows%20Content%20Browser%20%28DrMemory%20full%29%20%282%29/builds/3347/steps/memory%20test%3A%20content_browsertests_5/logs/CrossSiteTransferTest.ReplaceEntryCrossProcessThenTransfer

https://build.chromium.org/p/chromium.memory.fyi/builders/Windows%20Content%20Browser%20%28DrMemory%20full%29%20%282%29/builds/3347/steps/memory%20test%3A%20content_browsertests_8/logs/DumpAccessibilityTreeTest.AccessibilityDialog

I looked at the code, and I'm puzzled how that line can cause this problem too. The number of failing tests this way is pretty high (*) so I'm not sure if excluding all of them is a good thing to do. zhaoqin@, what's your opinion?

Bruce, would you mind running one of failing test on 2015+DrMemroy and see what's causing the problem?

(*)
https://build.chromium.org/p/chromium.memory.fyi/builders/Windows%20Content%20Browser%20%28DrMemory%20full%29%20%281%29/builds/7498
https://build.chromium.org/p/chromium.memory.fyi/builders/Windows%20Content%20Browser%20%28DrMemory%20full%29%20%282%29/builds/3347
https://build.chromium.org/p/chromium.memory.fyi/builders/Windows%20Content%20Browser%20%28DrMemory%20full%29%20%283%29/builds/7228
https://build.chromium.org/p/chromium.memory.fyi/builders/Windows%20Content%20Browser%20%28DrMemory%20full%29%20%284%29/builds/7525

Can you point me to instructions for using DrMemory?

https://www.chromium.org/developers/how-tos/using-drmemory

Thanks for the link to the instructions, but they seem to be missing two crucial details:
1) If I run content_browsertests then I get 1302 tests, which is unwieldy when trying to investigate a single failure. Normally I would use --gtest_filter=ManifestBrowserTest.DummyManifest but I'm not sure how to use that with chrome_tests.bat.
2) Is there a supported way to run the tests under a debugger in order to understand what is happening, or to at least get a crash dump when the failure happens? How do you investigate whether there's a real bug or a suppression needed, especially in the CRT where seeing the exact instructions generated can be crucial.

I'm running drmemory like this:

tools\valgrind\chrome_tests.bat -t content_browsertests --tool drmemory_light --build-dir=out\Release

1) The wrapper supports --gtest_filter.

2) I don't know. Skimming the Dr. Memory documentation (http://drmemory.org/docs/index.html) doesn't indicate it has an equivalent to Valgrind's option to attach GDB on errors.

Are you talking about the Valgrind feature "Debugging your program using Valgrind gdbserver and GDB"? It uses gdbserver remote debugging feature, which is not supported on Windows.

You could pause the execution and attach windbg to the running process, but it may not help much as all the application code are executed in code cache, which make debugging much harder.

The simplest way is just run the test normally and set breakpoint at the reported line, and check if any value is uninit.

Thanks. Taking a look now.

When I run locally I'm not seeing the access violation that I'm seeing on the bot's output, so I looked at the first failure that I am seeing, this one:

~~Dr.M~~ Error #1: INVALID HEAP ARGUMENT to _msize 0x05fe1440
~~Dr.M~~ # 0 replace_malloc_usable_size                                             [e:\b\build\slave\win-builder\drmemory\common\alloc_replace.c:2736]
~~Dr.M~~ # 1 _recalloc_base                                                         [d:\th\minkernel\crts\ucrt\src\appcrt\heap\recalloc.cpp:26]
~~Dr.M~~ # 2 <lambda_4e60a939b0d047cfe11ddc22648dfba9>::operator()                  [d:\th\minkernel\crts\ucrt\src\appcrt\startup\onexit.cpp:112]
~~Dr.M~~ # 3 __crt_seh_guarded_call<>::operator()<>                                 [d:\th\minkernel\crts\ucrt\devdiv\vcruntime\inc\internal_shared.h:199]
~~Dr.M~~ # 4 __acrt_lock_and_call<>                                                 [d:\th\minkernel\crts\ucrt\inc\corecrt_internal.h:906]
~~Dr.M~~ # 5 _register_onexit_function                                              [d:\th\minkernel\crts\ucrt\src\appcrt\startup\onexit.cpp:148]
~~Dr.M~~ # 6 _crt_atexit                                                            [d:\th\minkernel\crts\ucrt\src\appcrt\startup\onexit.cpp:43]
~~Dr.M~~ # 7 _onexit                                                                [f:\dd\vctools\crt\vcstartup\src\utility\utility.cpp:262]
~~Dr.M~~ # 8 atexit                                                                 [f:\dd\vctools\crt\vcstartup\src\utility\utility.cpp:276]
~~Dr.M~~ # 9 testing::`dynamic initializer for 'FLAGS_gmock_verbose''               [c:\src\chromium3\src\testing\gmock\src\gmock.cc:44]
~~Dr.M~~ #10 _initterm                                                              [d:\th\minkernel\crts\ucrt\src\appcrt\startup\initterm.cpp:21]
~~Dr.M~~ #11 __scrt_common_main_seh                                                 [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:230]
~~Dr.M~~ #12 KERNEL32.dll!BaseThreadInitThunk                                      +0x23     (0x75773744 <KERNEL32.dll+0x13744>)
~~Dr.M~~ Note: @0:00:03.363 in thread 67264
~~Dr.M~~ Note: refers to -128 byte(s) beyond last valid byte in prior malloc
~~Dr.M~~ Note: prev lower malloc:  0x05fe1440-0x05fe14c0

The output is confusing because the argument that they say is invalid is the same value that they say is the start of the prior malloc, but they are saying it is 128 below.

When I debug this call (gmock.cc:44) I see a call to the OS HeapSize function that returns 128, which is consistent with the size shown by 0x05fe1440-0x05fe14c0.

This smells like a DrMemory bug and suggests that we definitely need that VS 2015 compatible version.

That's all for now.

crrev.com/1808023003 updated DrMemory to the latest version that supports VS2015. Unfortunately there is a bug in chrome_tests.bat that causes this change to not take effect until the machine reboots are starts a new command shell. I opened crbug.com/595867 to track this.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4ecae6cf0186d351706937b23d00ccaa5db9f1be

commit 4ecae6cf0186d351706937b23d00ccaa5db9f1be
Author: brucedawson <brucedawson@chromium.org>
Date: Thu Mar 17 23:56:42 2016

Fix DrMemory warning in VS 2015 CRT

In VS 2015 builds every call to realloc triggers a warning like this:

Error #1: INVALID HEAP ARGUMENT: allocated with Windows API layer, queried with C library layer
  0 replace_malloc_usable_size
  1 _recalloc_base
  2 <lambda_4e60a939b0d047cfe11ddc22648dfba9>::operator()
  3 __crt_seh_guarded_call<>::operator()<>
  4 __acrt_lock_and_call<>
  5 _register_onexit_function
  6 _crt_atexit
  7 _onexit
  8 atexit
  9 testing::`dynamic initializer for 'FLAGS_gmock_verbose''

This mismatch is harmless (because the CRT always uses the Windows heap)
and out of our control. Therefore it should be suppressed.

With this suppression I now get a clean run on this previously failing
test:

tools\valgrind\chrome_tests.bat -t content_browsertests --tool \
    drmemory_light --build-dir=out\Release \
    --gtest_filter=ManifestBrowserTest.DummyManifest

BUG= 440500 , 594808 

Review URL: https://codereview.chromium.org/1813063002

Cr-Commit-Position: refs/heads/master@{#381840}

[modify] https://crrev.com/4ecae6cf0186d351706937b23d00ccaa5db9f1be/tools/valgrind/drmemory/suppressions.txt

Repurposed the bug to reflect actual problem. I think we should file a bug at https://github.com/DynamoRIO/drmemory/issues. Can someone who is more familiar with DrMemory/Windows file a bug? If none can, I will.

Even with new Drmemory, ~1000 tests are still failing due to this.
(browser_tests: 329, contnt_browsertests:656, see attached file for full list)
This number does not include the tests that have already been excluded.

Is it possible to switch back the compiler on DrMemory bots until it's fixed? I can exclude all of them, but I don't feel good about it.

Deleted: drmemory.fail.browsertests 15.5 KB

drmemory.fail.browsertests 15.5 KB Download

Deleted: drmemory.fail.content_browsertests 32.6 KB

drmemory.fail.content_browsertests 32.6 KB Download

It looks like Dr. Memory has its own builder so this should be possible as long as we want to maintain support for building with VS2013:

https://build.chromium.org/p/chromium.memory.fyi/builders/Chromium%20Windows%20Builder%20(DrMemory)

As discussed with oshima@ this would be a temporary fix while we wait for a DrMemory update. Chromium currently builds with both VS 2015 and VS 2013 but it is unlikely that this state will last for more than a couple of weeks.

Developers are just itching to do more advanced constexpr work, for instance.

I ran a test on Windows 10 and hit no crashes, but did hit several warnings, which I created suppressions for. This is the test I ran - intended to simulate one of the buildbot command lines:

tools\valgrind\chrome_tests.bat --test-launcher-bot-mode --test-launcher-total-shards=60 --test-launcher-shard-index=50 --test content_browsertests --tool drmemory_full --target Release --build-dir out

Running the same test on Windows 8.1 I hit no warnings except for this one, which appears to be an error when creating a minidump, in other words an error when handling a crash!

Given that my results are varying wildly between buildbot, Windows 10, and Windows 8.1, I am unsure as to how to proceed. It's difficult to investigate a crash that will not repro, I don't know why the crash was captured on my local machine but not on buildbot.

13:15:13 drmemory_analyze.py [INFO] Found 1 error reports
13:15:13 drmemory_analyze.py [INFO] Report #1
### BEGIN MEMORY TOOL REPORT (error hash=#91FBDB45D3476A7B#)
UNADDRESSABLE ACCESS beyond heap bounds: reading 0x07dcb470-0x07dcb474 4 byte(s)
# 0 dbghelp.dll!MiniDumpReadDumpStream        +0x3b033  (0x080d9a83 <dbghelp.dll+0x99a83>)
# 1 dbghelp.dll!MiniDumpReadDumpStream        +0x6d228  (0x0810bc79 <dbghelp.dll+0xcbc79>)
# 2 dbghelp.dll!MiniDumpReadDumpStream        +0x6e036  (0x0810ca87 <dbghelp.dll+0xcca87>)
# 3 dbghelp.dll!?                             +0x0      (0x08061401 <dbghelp.dll+0x21401>)
# 4 dbghelp.dll!GetTimestampForLoadedLibrary  +0x1120   (0x0806ec21 <dbghelp.dll+0x2ec21>)
# 5 dbghelp.dll!SymGetLineFromAddrW64         +0x14a    (0x08077e4b <dbghelp.dll+0x37e4b>)
# 6 dbghelp.dll!SymGetLineFromAddr64          +0x3d     (0x08077cae <dbghelp.dll+0x37cae>)
# 7 base.dll!?                                +0x0      (0x7761ef3f <base.dll+0x2ef3f>)
# 8 base.dll!?                                +0x0      (0x7761ee08 <base.dll+0x2ee08>)
# 9 base.dll!?                                +0x0      (0x7763eb53 <base.dll+0x4eb53>)
#10 gin.dll!?                                 +0x0      (0x5f027eb2 <gin.dll+0x17eb2>)
#11 gin.dll!?                                 +0x0      (0x5f027666 <gin.dll+0x17666>)
#12 content_browsertests.exe!?                +0x0      (0x018d2d71 <content_browsertests.exe+0x5e2d71>)
#13 content_browsertests.exe!?                +0x0      (0x018d6329 <content_browsertests.exe+0x5e6329>)
#14 content_browsertests.exe!?                +0x0      (0x0189bb70 <content_browsertests.exe+0x5abb70>)
#15 content_browsertests.exe!?                +0x0      (0x0174747c <content_browsertests.exe+0x45747c>)
#16 content_browsertests.exe!?                +0x0      (0x0189bbe1 <content_browsertests.exe+0x5abbe1>)
#17 content_browsertests.exe!?                +0x0      (0x01899f5b <content_browsertests.exe+0x5a9f5b>)
#18 KERNEL32.dll!BaseThreadInitThunk          +0x23     (0x75617c04 <KERNEL32.dll+0x17c04>)
Note: @0:00:07.649 in thread 40608
Note: instruction: mov    0xfffffffc(%eax,%ecx,4) -> %eax
The report came from the `ManifestBrowserTest.PushStateNavigation` test.

The suppressions that I need on Windows 10 are probably because the CRT is compiled differently for that OS due to different minimum hardware requirements. Therefore the suppression differences make sense and those suppressions should be committed.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/8c6bd89e3a6af7d23a08ac48e8522e7edafb6add

commit 8c6bd89e3a6af7d23a08ac48e8522e7edafb6add
Author: oshima@chromium.org <oshima@chromium.org>
Date: Fri Mar 18 21:07:54 2016

Temporarily switch the compiler back to VS2013 on DrMemory bots.

Because Too many tests are failing (~1000). I'll file a DrMemory issue.

BUG= 594808 

Review URL: https://codereview.chromium.org/1811113005

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/build@299388 0039d316-1c4b-4281-b951-d872f2087c98

[modify] https://crrev.com/8c6bd89e3a6af7d23a08ac48e8522e7edafb6add/masters/master.chromium.memory.fyi/master.cfg

This is the backtrace where it's crashing. It's same as where the uninitialized read is reported.

https://build.chromium.org/p/chromium.memory.fyi/builders/Windows%20Content%20Browser%20%28DrMemory%20full%29%20%281%29/builds/7502/steps/memory%20test%3A%20content_browsertests/logs/DownloadContentTest.CancelInterruptedDownloadnet::ShouldSniffMimeType [0x6B2535F0+160] (c:\b\build\slave\drm-cr\build\src\net\base\mime_sniffer.cc:831)
content::MimeTypeResourceHandler::ShouldSniffContent [0x68AEDF16+198] (c:\b\build\slave\drm-cr\build\src\content\browser\loader\mime_type_resource_handler.cc:236)
content::MimeTypeResourceHandler::OnResponseStarted [0x68AED2C8+72] (c:\b\build\slave\drm-cr\build\src\content\browser\loader\mime_type_resource_handler.cc:96)
content::ThrottlingResourceHandler::OnResponseStarted [0x68B0F0B4+100] (c:\b\build\slave\drm-cr\build\src\content\browser\loader\throttling_resource_handler.cc:130)
content::ThrottlingResourceHandler::ResumeResponse [0x68B0F4A3+51] (c:\b\build\slave\drm-cr\build\src\content\browser\loader\throttling_resource_handler.cc:225)
content::ThrottlingResourceHandler::Resume [0x68B0F280+96] (c:\b\build\slave\drm-cr\build\src\content\browser\loader\throttling_resource_handler.cc:169)
content::NavigationResourceThrottle::OnUIChecksPerformed [0x68AEF243+51] (c:\b\build\slave\drm-cr\build\src\content\browser\loader\navigation_resource_throttle.cc:243)
base::internal::Invoker<base::IndexSequence<0>,base::internal::BindState<base::internal::RunnableAdapter<void (__thiscall content::NavigationResourceThrottle::*)(enum content::NavigationThrottle::ThrottleCheckResult)>,void __cdecl(content::NavigationResou [0x68AEF5A5+85] (c:\b\build\slave\drm-cr\build\src\base\bind_internal.h:352)
base::internal::Invoker<base::IndexSequence<0>,base::internal::BindState<base::Callback<void __cdecl(enum content::NavigationThrottle::ThrottleCheckResult),1>,void __cdecl(enum content::NavigationThrottle::ThrottleCheckResult),enum content::NavigationThro [0x68AEF4EF+47] (c:\b\build\slave\drm-cr\build\src\base\bind_internal.h:352)
base::debug::TaskAnnotator::RunTask [0x6D0BFD73+243] (c:\b\build\slave\drm-cr\build\src\base\debug\task_annotator.cc:51)
base::MessageLoop::RunTask [0x6D0E74F3+419] (c:\b\build\slave\drm-cr\build\src\base\message_loop\message_loop.cc:477)
base::MessageLoop::DeferOrRunPendingTask [0x6D0E65DD+45] (c:\b\build\slave\drm-cr\build\src\base\message_loop\message_loop.cc:488)
base::MessageLoop::DoWork [0x6D0E6ADD+157] (c:\b\build\slave\drm-cr\build\src\base\message_loop\message_loop.cc:597)
base::MessagePumpForIO::DoRunLoop [0x6D0E8EEA+122] (c:\b\build\slave\drm-cr\build\src\base\message_loop\message_pump_win.cc:485)
base::MessagePumpWin::Run [0x6D0E98B6+54] (c:\b\build\slave\drm-cr\build\src\base\message_loop\message_pump_win.cc:52)
base::MessageLoop::RunHandler [0x6D0E7341+17] (c:\b\build\slave\drm-cr\build\src\base\message_loop\message_loop.cc:440)
base::RunLoop::Run [0x6D10C819+41] (c:\b\build\slave\drm-cr\build\src\base\run_loop.cc:36)
base::MessageLoop::Run [0x6D0E730D+29] (c:\b\build\slave\drm-cr\build\src\base\message_loop\message_loop.cc:294)
base::Thread::Run [0x6D132F5B+11] (c:\b\build\slave\drm-cr\build\src\base\threading\thread.cc:203)
content::BrowserThreadImpl::IOThreadRun [0x689054E0+32] (c:\b\build\slave\drm-cr\build\src\content\browser\browser_thread_impl.cc:216)
content::BrowserThreadImpl::Run [0x68905F0A+362] (c:\b\build\slave\drm-cr\build\src\content\browser\browser_thread_impl.cc:251)
base::Thread::ThreadMain [0x6D1333BB+347] (c:\b\build\slave\drm-cr\build\src\base\threading\thread.cc:257)
base::`anonymous namespace'::ThreadFunc [0x6D12B6A0+128] (c:\b\build\slave\drm-cr\build\src\base\threading\platform_thread_win.cc:86)
BaseThreadInitThunk [0x76B7337A+18]
RtlInitializeExceptionChain [0x772592B2+99]
RtlInitializeExceptionChain [0x77259285+54]
~~Dr.M~~ WARNING: application exited with abnormal code 0xc0000005

I filed a DrMemory issue: https://github.com/DynamoRIO/drmemory/issues/1871

crrev.com/1814273002 suppresses some warnings that only occur on AVX machines. It's possible that the crashes may be related to the available instruction sets.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/829c480de2a8619918d1656acfc16127a3498e8e

commit 829c480de2a8619918d1656acfc16127a3498e8e
Author: oshima@chromium.org <oshima@chromium.org>
Date: Sat Mar 19 22:24:31 2016

Revert of Temporarily switch the compiler back to VS2013 on DrMemory bots. (patchset #1 id:1 of https://codereview.chromium.org/1811113005/ )

Reason for revert:
THe code base is no longer compatible with VS2013. I'll exlucde failing tests instead.

Original issue's description:
> Temporarily switch the compiler back to VS2013 on DrMemory bots.
> 
> Because Too many tests are failing (~1000). I'll file a DrMemory issue.
> 
> BUG= 594808 
> 
> Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=299388

TBR=stip@chromium.org,thestig@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 594808 

Review URL: https://codereview.chromium.org/1823453002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/build@299407 0039d316-1c4b-4281-b951-d872f2087c98

[modify] https://crrev.com/829c480de2a8619918d1656acfc16127a3498e8e/masters/master.chromium.memory.fyi/master.cfg

These crashes are due to the same underlying bug as  issue 595792 : improper handling of modrm nop wrt shadow translation sharing.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/899620623be8f7b5004cea13f7da96ff7ca80c27

commit 899620623be8f7b5004cea13f7da96ff7ca80c27
Author: bruening <bruening@chromium.org>
Date: Sun Mar 20 13:37:54 2016

Remove Dr. Memory suppressions and exclusions that are no longer needed.

Remove the suppressions and exclusions put in place to work around Dr. Memory bugs that are now fixed.

BUG= 594614 , 594618 , 594785 , 594618 , 594808 , 595158 , 595490 
TBR=reillyg,oshima
NOTRY=true

Review URL: https://codereview.chromium.org/1817853002

Cr-Commit-Position: refs/heads/master@{#382222}

[modify] https://crrev.com/899620623be8f7b5004cea13f7da96ff7ca80c27/tools/valgrind/drmemory/suppressions_full.txt
[modify] https://crrev.com/899620623be8f7b5004cea13f7da96ff7ca80c27/tools/valgrind/gtest_exclude/unit_tests.gtest-drmemory_win32.txt