New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 638370 link

Starred by 1 user
Status: Untriaged
Owner: ----
Cc:
█ ananta@chromium.org
█ bustamante@chromium.org
█ kerz@chromium.org
chrisha@chromium.org
scottmg@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug-Regression



Sign in to add a comment

Crash in CrashPad::SetKeyValue / asan::SetUpAsanRuntime

Project Member Reported by rsleevi@chromium.org, Aug 16 2016 
Version: 54.0.2830.1
OS: Win 7

What steps will reproduce the problem?
(1) Open Chrome

What is the expected output?
It Opens

What do you see instead?
It crashes

Attaching WinDBG to it shows a null pointer deref in crash_reporter::`anonymous namespace`::SetCrashKeyValue, due to g_simple_string_dictionary being NULL. The call sequence appears to be ASAN's startup calling before CrashPad's InitializeCrashPadImpl has been called, which allocates g_simple_string_dictionary, and related to SyzygyAsan builds.

Full WinDBG backtrace:
# ChildEBP RetAddr  
00 004af11c 0015dde1 chrome!strncmp+0x49 [d:\th\minkernel\crts\ucrt\src\appcrt\string\i386\strncmp.asm @ 143]
01 004af13c 0015e1ed chrome!crashpad::TSimpleStringDictionary<256,256,64>::GetConstEntryForKey+0x1a [c:\b\build\slave\win-asan\build\src\third_party\crashpad\crashpad\client\simple_string_dictionary.h @ 227]
02 (Inline) -------- chrome!crashpad::TSimpleStringDictionary<256,256,64>::GetEntryForKey+0x6 [c:\b\build\slave\win-asan\build\src\third_party\crashpad\crashpad\client\simple_string_dictionary.h @ 235]
03 004af158 0015e1bc chrome!crashpad::TSimpleStringDictionary<256,256,64>::SetKeyValue+0x2f [c:\b\build\slave\win-asan\build\src\third_party\crashpad\crashpad\client\simple_string_dictionary.h @ 168]
04 004af168 0015e51a chrome!crash_reporter::`anonymous namespace'::SetCrashKeyValue+0x18 [c:\b\build\slave\win-asan\build\src\components\crash\content\app\crashpad.cc @ 58]
05 004af1c0 64427f6d chrome!SetCrashKeyValueImpl+0x76 [c:\b\build\slave\win-asan\build\src\components\crash\content\app\crashpad.cc @ 469]
06 (Inline) -------- syzyasan_rtl!agent::asan::reporters::ExportedFunctionImpl<0,void __cdecl(wchar_t const *,wchar_t const *),void,wchar_t const *,wchar_t const *>::Run+0x12 [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\asan\reporters\exported_function.h @ 110]
07 004af210 644170a9 syzyasan_rtl!agent::asan::reporters::KaskoReporter::SetCrashKey+0x5d [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\asan\reporters\kasko_reporter.cc @ 85]
08 004af230 64418484 syzyasan_rtl!agent::asan::`anonymous namespace'::SetEarlyCrashKeysIfPossible+0x69 [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\asan\runtime.cc @ 165]
09 004af3b4 64421451 syzyasan_rtl!agent::asan::AsanRuntime::SetUp+0x424 [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\asan\runtime.cc @ 571]
0a 004af4a0 644123ac syzyasan_rtl!agent::asan::SetUpAsanRuntime+0x1c1 [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\asan\runtime_util.cc @ 214]
0b 004af4c8 6445566d syzyasan_rtl!DllMain+0x6c [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\asan\syzyasan_rtl.cc @ 176]
0c 004af508 644555f4 syzyasan_rtl!__DllMainCRTStartup+0x72 [f:\dd\vctools\crt\crtw32\startup\dllcrt0.c @ 377]
0d 004af51c 77e69364 syzyasan_rtl!_DllMainCRTStartup+0x1c [f:\dd\vctools\crt\crtw32\startup\dllcrt0.c @ 340]
0e 004af53c 77e6dc81 ntdll!LdrpCallInitRoutine+0x14
0f 004af630 77e6c901 ntdll!LdrpRunInitializeRoutines+0x26f
10 004af7a4 77ead39f ntdll!LdrpLoadDll+0x472
11 004af7e0 77572e0f ntdll!LdrLoadDll+0xc7
12 004af828 0015086a KERNELBASE!LoadLibraryExW+0x233
13 004af864 0015075f chrome!`anonymous namespace'::LoadModuleWithDirectory+0xa0 [c:\b\build\slave\win-asan\build\src\chrome\app\main_dll_loader_win.cc @ 79]
14 004af954 0015061c chrome!MainDllLoader::Load+0xe7 [c:\b\build\slave\win-asan\build\src\chrome\app\main_dll_loader_win.cc @ 123]
15 004af9ec 0014f989 chrome!MainDllLoader::Launch+0x1b9 [c:\b\build\slave\win-asan\build\src\chrome\app\main_dll_loader_win.cc @ 178]
16 004afb10 002d0e6f chrome!wWinMain+0x125 [c:\b\build\slave\win-asan\build\src\chrome\app\chrome_exe_main_win.cc @ 251]


As this crash happens before crash reporting is connected, chrome://crashes shows nothing, so we may have no visibility.

Comment 1 by scottmg@chromium.org, Aug 16 2016

Cc: chrisha@chromium.org scottmg@chromium.org
Failing at startup because the crash keys have moved to chrome_elf, but syzyasan/kasko is still looking in chrome.exe.

chrome.exe is still accidentally exporting the set functions which are not initialized, which ananta and I will fix now.

But Canary will not launch for some (all?) users if they got asan.

Comment 2 by scottmg@chromium.org, Aug 16 2016

Owner: ananta@chromium.org
Status: Started (was: Untriaged)
Ananta has a change out https://codereview.chromium.org/2250263002 to fix chrome.exe "lying" about implementing those functions.

I *think* from looking at kasko_reporter.cc it'll quietly fail after than change instead of crashing, but chrisha, maybe you could confirm?

(I also thought that code was already removed from syzyasan? I guess it's still pending? Or we're still in a 50/50 trial? Or something needs to be rolled? I'm confused.)

Comment 3 by chrisha@chromium.org, Aug 16 2016 

SyzyASAN is also soon going to be using Crashpad directly which will also fix this.

scottmg: we're still blocked on "custom memory ranges" not making it into the minidumps. Haven't had any time to look into it, but we can confirm it by looking at crash reports with asan-crash-reporter=CrashpadReporter. None of the reports have "block" or "shadow" data in the associated "ASAN" tab. This data is provided to the reporter via the "bag of address ranges" functionality. We are indeed still in the 50/50 trial and got sidetracked with a bazillion other things :/

Removing the exports from chrome.exe will also simply break SyzyASAN reporting entirely right now. This is a benign error in that it won't stop the browser from launching.

rsleevi: You asked about opt-out and opt-in mechanisms. Refer to go/asan-optin for details.

Comment 4 by scottmg@chromium.org, Aug 17 2016 

Sorry, must have missed the bag of addresses problem after being on vacation for a while. I'll take a look at some of those crashes tomorrow.

Is this the first SyzyASAN canary in quite a while? The change to flip crashpad to chrome_elf and remove the initialization from chrome.exe was a month ago now https://bugs.chromium.org/p/chromium/issues/detail?id=604923#c19 so I would have expected the "bad news" sooner.
Project Member

Comment 5 by bugdroid1@chromium.org, Aug 17 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4bb92a6392a2f8bdc5c6c1319690800748870f1c

commit 4bb92a6392a2f8bdc5c6c1319690800748870f1c
Author: ananta <ananta@chromium.org>
Date: Wed Aug 17 01:03:05 2016

Fix ASAN canary crashes.

The ASAN startup code now uses an exported function GetUploadsEnabled from chrome_elf to determine
whether uploads are enabled. Additionally we don't link with crashpad/app in chrome.exe. chrome.exe
now links with the new library target crashpad/run_as_crashpad_handler. The ASAN kasko code today
gets exported crash functions from the exe, which won't work correctly due to crashpad having moved
to chrome_elf.dll. This needs to be fixed for ASAN crashes to be reported correctly.

BUG=638370, 604923

Review-Url: https://codereview.chromium.org/2250263002
Cr-Commit-Position: refs/heads/master@{#412405}

[modify] https://crrev.com/4bb92a6392a2f8bdc5c6c1319690800748870f1c/chrome/BUILD.gn
[modify] https://crrev.com/4bb92a6392a2f8bdc5c6c1319690800748870f1c/chrome/app/OWNERS
[modify] https://crrev.com/4bb92a6392a2f8bdc5c6c1319690800748870f1c/chrome/app/main_dll_loader_win.cc
[modify] https://crrev.com/4bb92a6392a2f8bdc5c6c1319690800748870f1c/chrome/installer/setup/BUILD.gn
[modify] https://crrev.com/4bb92a6392a2f8bdc5c6c1319690800748870f1c/chrome_elf/chrome_elf_main.cc
[modify] https://crrev.com/4bb92a6392a2f8bdc5c6c1319690800748870f1c/components/crash/content/app/BUILD.gn

Comment 6 by grt@chromium.org, Aug 17 2016 

Is this the cause of  issue 635568 ?

Comment 7 by siggi@chromium.org, Aug 17 2016 

I'm guessing this is what broke the win-asan build:

[15863/34168] CXX obj/url/mojo/url_mojom_origin_cpp_sources/origin.mojom.obj
[15864/34168] CXX obj/chrome/chrome_initial/kasko_client.obj
FAILED: obj/chrome/chrome_initial/kasko_client.obj 
ninja -t msvc -e environment.x86 -- "C:\b\depot_tools\win_toolchain\vs_files\95ddda401ec5678f15eeed01d2bee08fcbc5ee97\VC\bin\amd64_x86/cl.exe" /nologo /showIncludes /FC @obj/chrome/chrome_initial/kasko_client.obj.rsp /c ../../chrome/app/kasko_client.cc /Foobj/chrome/chrome_initial/kasko_client.obj /Fd"obj/chrome/chrome_initial_cc.pdb"
c:\b\build\slave\win-asan\build\src\chrome\app\kasko_client.h(13): fatal error C1083: Cannot open include file: 'syzygy/kasko/api/minidump_type.h': No such file or directory
[15865/34168] LIB obj/chrome/common/net/net.lib
[15866/34168] CXX obj/ui/views/mus/mus/screen_mus.obj
[15867/34168] CXX obj/chrome/chrome_initial/crash_keys.obj
[15868/34168] CXX obj/chrome/chrome_initial/chrome_exe_main_win.obj
FAILED: obj/chrome/chrome_initial/chrome_exe_main_win.obj 
ninja -t msvc -e environment.x86 -- "C:\b\depot_tools\win_toolchain\vs_files\95ddda401ec5678f15eeed01d2bee08fcbc5ee97\VC\bin\amd64_x86/cl.exe" /nologo /showIncludes /FC @obj/chrome/chrome_initial/chrome_exe_main_win.obj.rsp /c ../../chrome/app/chrome_exe_main_win.cc /Foobj/chrome/chrome_initial/chrome_exe_main_win.obj /Fd"obj/chrome/chrome_initial_cc.pdb"
c:\b\build\slave\win-asan\build\src\components\crash\content\app\crashpad.h(19): fatal error C1083: Cannot open include file: 'syzygy/kasko/api/crash_key.h': No such file or directory
[15869/34168] CXX obj/media/mojo/interfaces/interfaces_cpp_sources/audio_parameters.mojom.obj
[15870/34168] CXX obj/chrome/chrome_initial/chrome_watcher_command_line_win.obj
[15871/34168] CXX obj/chrome/chrome_initial/chrome_watcher_client_win.obj
[15872/34168] CXX obj/chrome/chrome_initial/main_dll_loader_win.obj
FAILED: obj/chrome/chrome_initial/main_dll_loader_win.obj 
ninja -t msvc -e environment.x86 -- "C:\b\depot_tools\win_toolchain\vs_files\95ddda401ec5678f15eeed01d2bee08fcbc5ee97\VC\bin\amd64_x86/cl.exe" /nologo /showIncludes /FC @obj/chrome/chrome_initial/main_dll_loader_win.obj.rsp /c ../../chrome/app/main_dll_loader_win.cc /Foobj/chrome/chrome_initial/main_dll_loader_win.obj /Fd"obj/chrome/chrome_initial_cc.pdb"
c:\b\build\slave\win-asan\build\src\chrome\app\kasko_client.h(13): fatal error C1083: Cannot open include file: 'syzygy/kasko/api/minidump_type.h': No such file or directory

Comment 8 by scottmg@chromium.org, Aug 17 2016 

How do I build that? `gn args --list out\release | grep -i kasko` doesn't list anything.

Comment 9 by scottmg@chromium.org, Aug 17 2016 

Oh, I see

third_party\kasko\kasko.gni:9:enable_kasko = is_official_build && is_syzyasan

now. Building...

Comment 10 by lafo...@chromium.org, Aug 17 2016

Cc: -lafo...@chromium.org kerz@chromium.org

Comment 12 by rsleevi@chromium.org, Aug 18 2016

Labels: -Pri-0 -ReleaseBlock-Dev M-54 Pri-1
Seems like consensus that we weren't going to be releasing SyzysyAsan beyond Canary, so this isn't RB-D/P0 - just ReleaseBlock-AsanCanary? :)

Comment 13 by chrisha@chromium.org, Aug 18 2016 

No, not a RB-D IMO. Still a P0 in that we are shipping some people an install of Chrome that simply won't start.

Comment 14 by chrisha@chromium.org, Aug 18 2016 

No, not a RB-D IMO. Still a P0 in that we are shipping some people an install of Chrome that simply won't start.

Comment 15 by chrisha@chromium.org, Aug 18 2016 

No, not a RB-D IMO. Still a P0 in that we are shipping some people an install of Chrome that simply won't start.
Project Member

Comment 16 by bugdroid1@chromium.org, Aug 25 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/crashpad/crashpad.git/+/c6f88d164e44c0c710b783f7b89676b38f79f440

commit c6f88d164e44c0c710b783f7b89676b38f79f440
Author: Scott Graham <scottmg@chromium.org>
Date: Thu Aug 25 22:01:27 2016

Have MinidumpMemoryListWriter deal directly in SnapshotMinidumpMemoryWriters

This is as a precursor to
https://chromium-review.googlesource.com/374539 which merges
MemorySnapshots and so needs to be able to update them from the minidump
code.

MinidumpMemoryWriter existed to be able to mock for tests; that
behaviour is wrapped up in TestMemorySnapshot now.

BUG=crashpad:61, chromium:638370

Change-Id: I825ec57493b12fc1848018585c14544faa7e66d4
Reviewed-on: https://chromium-review.googlesource.com/374019
Reviewed-by: Mark Mentovai <mark@chromium.org>

[modify] https://crrev.com/c6f88d164e44c0c710b783f7b89676b38f79f440/minidump/minidump_memory_writer.cc
[modify] https://crrev.com/c6f88d164e44c0c710b783f7b89676b38f79f440/minidump/minidump_memory_writer.h
[modify] https://crrev.com/c6f88d164e44c0c710b783f7b89676b38f79f440/minidump/minidump_thread_writer.cc
[modify] https://crrev.com/c6f88d164e44c0c710b783f7b89676b38f79f440/minidump/minidump_thread_writer.h
[modify] https://crrev.com/c6f88d164e44c0c710b783f7b89676b38f79f440/minidump/test/minidump_memory_writer_test_util.cc
[modify] https://crrev.com/c6f88d164e44c0c710b783f7b89676b38f79f440/minidump/test/minidump_memory_writer_test_util.h
Project Member

Comment 17 by bugdroid1@chromium.org, Feb 2 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/crashpad/crashpad.git/+/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087

commit 9b6c69cbb59f85f3b7f1fb2423e2c63111b92087
Author: Scott Graham <scottmg@chromium.org>
Date: Fri Feb 02 00:04:20 2018

Coalesce memory ranges

Follows https://chromium-review.googlesource.com/c/374019/.

Causes MinidumpMemoryListWriter to merge all overlapping ranges before
writing the MINIDUMP_MEMORY_LIST. This is:

1) Necessary for the Google internal crash processor, which in some
   cases attempts to read the raw memory (displaying ASAN red zones),
   and aborts if there are any overlapping ranges in the minidump on
   load;

2) Necessary for new-ish versions of windbg (see  bug 216  below). It is
   believed that this is a change in behavior in the tool that made
   dumps with overlapping ranges unreadable;

3) More efficient. The .dmp for crashy_program goes from 306K to 140K
   with this enabled. In Chrome minidumps where
   set_gather_indirectly_referenced_memory() is used (in practice this
   means Chrome Windows Beta, Dev, and Canary), the savings are expected
   to be substantial.

Bug: crashpad:61, chromium:638370,  crashpad:216 

Change-Id: I969e1a52da555ceba59a727d933bfeef6787c7a5
Reviewed-on: https://chromium-review.googlesource.com/374539
Commit-Queue: Scott Graham <scottmg@chromium.org>
Reviewed-by: Mark Mentovai <mark@chromium.org>

[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/minidump/minidump_memory_writer.cc
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/minidump/minidump_memory_writer.h
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/minidump/minidump_memory_writer_test.cc
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/minidump/minidump_thread_writer.cc
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/minidump/minidump_thread_writer_test.cc
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/snapshot/BUILD.gn
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/snapshot/linux/memory_snapshot_linux.cc
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/snapshot/linux/memory_snapshot_linux.h
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/snapshot/mac/memory_snapshot_mac.cc
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/snapshot/mac/memory_snapshot_mac.h
[add] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/snapshot/memory_snapshot.cc
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/snapshot/memory_snapshot.h
[add] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/snapshot/memory_snapshot_test.cc
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/snapshot/snapshot.gyp
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/snapshot/snapshot_test.gyp
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/snapshot/test/test_memory_snapshot.cc
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/snapshot/test/test_memory_snapshot.h
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/snapshot/win/memory_snapshot_win.cc
[modify] https://crrev.com/9b6c69cbb59f85f3b7f1fb2423e2c63111b92087/snapshot/win/memory_snapshot_win.h

Comment 18 by robliao@chromium.org, Aug 21

Owner: ----
Status: Untriaged (was: Started)
Unassigning all bugs for ananta@chromium.org
►

Sign in to add a comment