Bruce will probably direct you to go/etw. :)

One random guess might be that you have the DCHECK build and it's doing something wacky.

Assigning to jochen for ETW traces.

An alternate way that doesn't involve ETW is to attach a debugger on to Chrome and breaking in when you see the app is not reacting. The culprit stack will more than likely be on the UI thread.

If it is hung and then recovers then an ETW trace will reveal all. I've seen repeated 5 second hangs on various laptops, and once on my desktop. Every time I've managed to get an ETW trace it's turned out to be bad network drivers that hang for five seconds on closesocket. It's crucial that the trace contain the time period where the browser recovers from the hang - we can then see what the blocked threads had been waiting on.

It it doesn't recover then a crash dump is what is needed. You can either attach a debugger and analyze it yourself, attach a debugger and record a crash dump, or use sysinternal's procdump.exe. A full crash dump is best.

created two traces and uploaded them here: https://drive.google.com/a/google.com/file/d/1iB7J4RuHBmpz2LcPenOfuWdzJ8IPAaqF/view?usp=sharing

Thanks for the trace. It's very helpful. I looked at the first one (2017-11-21_09-31-50_eisinger.etl).

The UI Delays graph showed a 120 s hang (!!!) on the browser process UI thread. So that's bad.

During that time the browser process was 100% CPU bound so I looked at the CPU sampled data. It's best summarized as a flame graph (screen shot attached).

The flame graph shows that the main thread is spending all of its time in av_get_cpu_flags, doing regex work. Unfortunately when I code search I can't find the code that initiates the re work - av_get_cpu_flags doesn't. Maybe it's password related?

Meanwhile another thread is busy doing syzyasan memory work. syzyasan slows everything down a lot so that's certainly not helping, but the regex work is probably the main problem.

So, my best bet is that syzyasan has somehow confused the symbols which is hiding the real cause. And syzyasan is slowing down everything, including the regex work (which would probably be slow anyway).

Assigning to siggi@ for syzyasan advice. A minidump of the browser process while it's doing this work might also be helpful since it might make it easier to understand what is actually triggering the regex work.

As a mitigation you could move off of syzyasan. In the trace you are running 64.0.3244.1. There is a reg key you can set if you want off of syzyasan.

Deleted: re and asan flamegraph.PNG 231 KB

	re and asan flamegraph.PNG 231 KB View Download

how do I create a minidump on windows?

it looks like the only parts that invoke this RE2 function are from the feedback component which if my codesearch foo is not failing me are only used on chromeos and via extension APIs

A minidump might be helpful, this could be some kind of degenerate combo of profile data and the SyzyASAN allocator's overhead. Normally SyzyASAN doesn't add more than ~3X CPU overhead, so that'd be quite degenerate.
Does this also happen on regular 32/64 bit canary?

The simplest way is to use procdump:

https://docs.microsoft.com/en-us/sysinternals/downloads/procdump

You can find out in advance what the browser-process PID is and have the command all queued up in a command prompt. Then, when Chrome hangs you can just switch to the command prompt and record one.

I would recommend just the default settings which will give a stack-only minidump which should be sufficient. So "procdump 250240" (assuming that your browser-process PID is the same as mine :-)

Actually, record several. They are fairly small (~5 MB) and that will increase the odds of recording what matters.

here are some dumps: https://drive.google.com/a/google.com/file/d/1vncMAEKDsgOfp4vWrnwhzk0gcl6jBh2P/view?usp=sharing

Looks like autofill is doing a shitload of work on the UI thread, and SyzyASAN is multiplying up the cost by a large factor, presumably due to allocation churn.

I'm not sure this is intended to run on the UI thread, but there's just a bucketload of work, and it's poorly executed.

At the top of this loop:
void ContentPasswordManagerDriver::PasswordFormsParsed(
    const std::vector<autofill::PasswordForm>& forms) {
  for (const auto& form : forms)
    if (!CheckChildProcessSecurityPolicy(
            form.origin, BadMessageReason::CPMD_BAD_ORIGIN_FORMS_PARSED))
      return;

  OnPasswordFormsParsedNoRenderCheck(forms);
}

we have many forms:

0:000:x86> ? (0x7a85f3a0 -0x7a85d060 )/4
Evaluate expression: 2256 = 000008d0

This code looks particularly heinous:

// Returns a copy of |input| without all occurrences of
// |kIgnorePatternInFieldName|
std::string StripDigitsIfRequired(const base::string16& input) {
  std::string return_string = base::UTF16ToUTF8(input);
  re2::RE2::GlobalReplace(&return_string, re2::RE2(kIgnorePatternInFieldName),
                          re2::StringPiece());
  return return_string;
}

as it'll allocate, compile, run and discard a RE2 on every execution, to do a VERY simple thing.

Why all this work is happening I can't say, maybe our password manager folks can?

0:000:x86> kv
 # ChildEBP RetAddr  Args to Child              
00 04decf54 6fe57899 04df0000 00000001 0000003e syzyasan_rtl!agent::common::WalkStackImpl+0xab (FPO: [Non-Fpo]) (CONV: cdecl) [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\common\stack_walker.cc @ 173]
01 04decf80 6fe57108 04ded038 04ded02c 04ded028 syzyasan_rtl!agent::common::WalkStack+0x59 (FPO: [Non-Fpo]) (CONV: cdecl) [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\common\stack_walker.cc @ 131]
02 04decf98 6fe4b579 00000028 00000028 00000000 syzyasan_rtl!agent::common::StackCapture::InitFromStack+0x28 (FPO: [Non-Fpo]) (CONV: thiscall) [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\common\stack_capture.cc @ 108]
03 04ded138 6fe441a3 053a3e20 00000028 00000028 syzyasan_rtl!agent::asan::heap_managers::BlockHeapManager::Allocate+0xc9 (FPO: [Non-Fpo]) (CONV: thiscall) [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\asan\heap_managers\block_heap_manager.cc @ 211]
04 (Inline) -------- -------- -------- -------- syzyasan_rtl!agent::asan::WindowsHeapAdapter::HeapAlloc+0x3 (Inline Function @ 6fe441a3) (CONV: cdecl) [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\asan\windows_heap_adapter.cc @ 74]
05 04ded158 672751df 053a3e20 00000000 00000028 syzyasan_rtl!asan_HeapAlloc+0x43 (FPO: [Non-Fpo]) (CONV: stdcall) [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\asan\rtl_impl.cc @ 103]
06 (Inline) -------- -------- -------- -------- chrome_670c0000!base::allocator::WinHeapMalloc+0x19 (Inline Function @ 672751df) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\allocator\winheap_stubs_win.cc @ 34]
07 04ded16c 671a2b0c 6a810620 00000028 00000000 chrome_670c0000!`anonymous namespace'::DefaultWinHeapMallocImpl+0x1c (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\allocator\allocator_shim_default_dispatch_to_winheap.cc @ 18]
08 (Inline) -------- -------- -------- -------- chrome_670c0000!ShimMalloc+0x1e (Inline Function @ 671a2b0c) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\allocator\allocator_shim.cc @ 194]
09 04ded18c 6997efbf 00000028 04ded1b0 68b5cb2e chrome_670c0000!malloc+0x18 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\allocator\allocator_shim_override_ucrt_symbols_win.h @ 51]
0a 04ded198 68b5cb2e 00000028 00000000 00000000 chrome_670c0000!operator new+0x1a (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt\vcstartup\src\heap\new_scalar.cpp @ 19]
0b 04ded1b0 68b60a6f 0000000a 00000000 04ded26c chrome_670c0000!re2::SparseArray<int>::SparseArray<int>+0x2e (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\third_party\re2\src\util\sparse_array.h @ 503]
0c 04ded240 68b70443 00000000 00000009 04ded26c chrome_670c0000!re2::Prog::Flatten+0x45 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\third_party\re2\src\re2\prog.cc @ 573]
0d 04ded258 68b7003c 04ded4e0 04ded490 04ded480 chrome_670c0000!re2::Compiler::Finish+0x4e (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\third_party\re2\src\re2\compile.cc @ 1205]
0e 04ded2f0 68b70063 004f77c0 00000000 00555555 chrome_670c0000!re2::Compiler::Compile+0x133 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\third_party\re2\src\re2\compile.cc @ 1186]
0f 04ded308 68b5dd4c 00555555 00000000 0000000f chrome_670c0000!re2::Regexp::CompileToProg+0x11 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\third_party\re2\src\re2\compile.cc @ 1224]
10 04ded420 68b5cf91 04ded454 04ded43c 7a85dbbc chrome_670c0000!re2::RE2::Init+0x25d (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\third_party\re2\src\re2\re2.cc @ 212]
11 04ded45c 6a018d83 6a7c0110 04ded478 7a85d930 chrome_670c0000!re2::RE2::RE2+0x6d (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\third_party\re2\src\re2\re2.cc @ 109]
12 (Inline) -------- -------- -------- -------- chrome_670c0000!autofill::?A0x3b6280d5::StripDigitsIfRequired+0x82f58d55 (Inline Function @ 6a018d83) (CONV: cdecl) [c:\b\c\b\win_asan\src\components\autofill\core\common\signatures_util.cc @ 28]
13 04ded5fc 69d83256 7a85dbbc 7a330edc 7a85d930 chrome_670c0000!autofill::CalculateFormSignature+0x10d (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\components\autofill\core\common\signatures_util.cc @ 55]
14 04ded678 69d75e48 7a85d930 2f1e9f30 04deddac chrome_670c0000!password_manager::PasswordFormManager::DoesManage+0x1f2 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\components\password_manager\core\browser\password_form_manager.cc @ 340]
15 04ded6ec 694882f2 2f1e9f30 04deddac 7a85f3a0 chrome_670c0000!password_manager::PasswordManager::CreatePendingLoginManagers+0xf8 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\components\password_manager\core\browser\password_manager.cc @ 542]
16 04ded700 694883c7 04deddac 04dee11c 2c3d1068 chrome_670c0000!password_manager::ContentPasswordManagerDriver::OnPasswordFormsParsedNoRenderCheck+0x16 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\components\password_manager\content\browser\content_password_manager_driver.cc @ 209]
17 04ded718 684add07 7a85f3a0 3e1fbc68 6a2cfe50 chrome_670c0000!password_manager::ContentPasswordManagerDriver::PasswordFormsParsed+0x40 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\components\password_manager\content\browser\content_password_manager_driver.cc @ 204]
18 04deddd4 6948762e 2f1e9f38 04dee11c 04dedec0 chrome_670c0000!autofill::mojom::PasswordManagerDriverStubDispatch::Accept+0x156 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\out\release\gen\components\autofill\content\common\autofill_driver.mojom.cc @ 1830]
19 04dedde4 6867c161 04dee11c 3e1fbc88 41ea568c chrome_670c0000!autofill::mojom::PasswordManagerDriverStub<mojo::RawPtrImplRefTraits<autofill::mojom::PasswordManagerDriver> >::Accept+0x18 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\out\release\gen\components\autofill\content\common\autofill_driver.mojom.h @ 466]
1a 04dedec0 6867efa3 00000000 3e1fbc68 3e1fbc68 chrome_670c0000!mojo::InterfaceEndpointClient::HandleValidatedMessage+0x246 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\interface_endpoint_client.cc @ 420]
1b 04dedf8c 6867bdda 04dee11c 3d9d2750 00000000 chrome_670c0000!mojo::FilterChain::Accept+0x7a (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\filter_chain.cc @ 41]
1c 04dee054 68675384 04dee11c 3d9d28d0 3d9d2750 chrome_670c0000!mojo::InterfaceEndpointClient::HandleIncomingMessage+0x68 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\interface_endpoint_client.cc @ 305]
1d 04dee154 6867326c 04dee254 00000000 273883d0 chrome_670c0000!mojo::internal::MultiplexRouter::ProcessIncomingMessage+0x29c (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\multiplex_router.cc @ 885]
1e 04dee298 6867efa3 04dee42c 3d9d2784 3d9d286c chrome_670c0000!mojo::internal::MultiplexRouter::Accept+0x11d (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\multiplex_router.cc @ 604]
1f 04dee364 68670a0a 04dee42c 3d9d286c 3d9d2784 chrome_670c0000!mojo::FilterChain::Accept+0x7a (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\filter_chain.cc @ 41]
20 04dee4ac 6867067b 6a8e2220 00000000 3d9d2784 chrome_670c0000!mojo::Connector::ReadSingleMessage+0x16d (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\connector.cc @ 439]
21 04dee588 68670246 422e7e18 425f5be8 422e7e1b chrome_670c0000!mojo::Connector::ReadAllAvailableMessages+0x3f (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\connector.cc @ 469]
22 04dee64c 689ea648 00000000 04dee66c 67dbea70 chrome_670c0000!mojo::Connector::OnHandleReadyInternal+0x7e (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\connector.cc @ 376]
23 (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::FunctorTraits<void (__thiscall net::HttpProxyClientSocketWrapper::*)(int),void>::Invoke+0x8192a637 (Inline Function @ 689ea648) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 194]
24 (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::InvokeHelper<0,void>::MakeItSo+0x8192a637 (Inline Function @ 689ea648) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 277]
25 (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::Invoker<base::internal::BindState<void (__thiscall net::HttpProxyClientSocketWrapper::*)(int),base::internal::UnretainedWrapper<net::HttpProxyClientSocketWrapper> >,void __cdecl(int)>::RunImpl+0x8192a637 (Inline Function @ 689ea648) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 349]
26 04dee658 67dbea70 422e8160 04dee678 425f5bc0 chrome_670c0000!base::internal::Invoker<base::internal::BindState<void (__thiscall net::HttpProxyClientSocketWrapper::*)(int),base::internal::UnretainedWrapper<net::HttpProxyClientSocketWrapper> >,void __cdecl(int)>::Run+0x11 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 334]
27 (Inline) -------- -------- -------- -------- chrome_670c0000!base::RepeatingCallback<void __cdecl(unsigned int)>::Run+0x80cfea5c (Inline Function @ 67dbea70) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\callback.h @ 92]
28 04dee66c 683ccad4 422e7e18 00000000 41c89fc8 chrome_670c0000!mojo::SimpleWatcher::DiscardReadyState+0x1f (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\mojo\public\cpp\system\simple_watcher.h @ 193]
29 (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::FunctorTraits<void (__cdecl*)(base::RepeatingCallback<void __cdecl(enum extensions::BluetoothApiSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)> const &,enum device::BluetoothSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &),void>::Invoke+0x8130cab1 (Inline Function @ 683ccad4) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 149]
2a (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::InvokeHelper<0,void>::MakeItSo+0x8130cab1 (Inline Function @ 683ccad4) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 277]
2b (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::Invoker<base::internal::BindState<void (__cdecl*)(base::RepeatingCallback<void __cdecl(enum extensions::BluetoothApiSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)> const &,enum device::BluetoothSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &),base::RepeatingCallback<void __cdecl(enum extensions::BluetoothApiSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)> >,void __cdecl(enum device::BluetoothSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)>::RunImpl+0x8130cab1 (Inline Function @ 683ccad4) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 349]
2c 04dee680 68686b9b 422e7e00 04dee768 41c89fc8 chrome_670c0000!base::internal::Invoker<base::internal::BindState<void (__cdecl*)(base::RepeatingCallback<void __cdecl(enum extensions::BluetoothApiSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)> const &,enum device::BluetoothSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &),base::RepeatingCallback<void __cdecl(enum extensions::BluetoothApiSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)> >,void __cdecl(enum device::BluetoothSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)>::Run+0x15 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 331]
2d (Inline) -------- -------- -------- -------- chrome_670c0000!base::RepeatingCallback<void __cdecl(unsigned int,mojo::HandleSignalsState const &)>::Run+0x815c6b87 (Inline Function @ 68686b9b) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\callback.h @ 92]
2e 04dee75c 690b7ca1 6a2841a8 00000000 41c89fc8 chrome_670c0000!mojo::SimpleWatcher::OnHandleReady+0xcc (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\system\simple_watcher.cc @ 276]
2f (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::FunctorTraits<void (__thiscall storage::LocalFileStreamWriter::*)(net::IOBuffer *,int,base::RepeatingCallback<void __cdecl(int)> const &),void>::Invoke+0x81ff7c7d (Inline Function @ 690b7ca1) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 194]
30 04dee774 68685fa5 41c89fc4 41c89fd8 41c89fd4 chrome_670c0000!base::internal::InvokeHelper<1,void>::MakeItSo<void (__thiscall storage::LocalFileStreamWriter::*const &)(net::IOBuffer *,int,base::RepeatingCallback<void __cdecl(int)> const &),base::WeakPtr<storage::LocalFileStreamWriter> const &,net::IOBuffer *,int const &,base::RepeatingCallback<void __cdecl(int)> const &>+0x2e (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 297]
31 04dee790 68686c33 41c89fc4 41c89fc8 41c89fb0 chrome_670c0000!base::internal::Invoker<base::internal::BindState<void (__thiscall mojo::SimpleWatcher::*)(int,unsigned int,mojo::HandleSignalsState const &),base::WeakPtr<mojo::SimpleWatcher>,int,unsigned int,mojo::HandleSignalsState>,void __cdecl(void)>::RunImpl<void (__thiscall mojo::SimpleWatcher::*const &)(int,unsigned int,mojo::HandleSignalsState const &),std::tuple<base::WeakPtr<mojo::SimpleWatcher>,int,unsigned int,mojo::HandleSignalsState> const &,0,1,2,3>+0x1b (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 349]
32 04dee7a4 672a8232 41c89fb0 273877c0 04deed00 chrome_670c0000!base::internal::Invoker<base::internal::BindState<void (__thiscall mojo::SimpleWatcher::*)(int,unsigned int,mojo::HandleSignalsState const &),base::WeakPtr<mojo::SimpleWatcher>,int,unsigned int,mojo::HandleSignalsState>,void __cdecl(void)>::Run+0x16 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 331]
33 (Inline) -------- -------- -------- -------- chrome_670c0000!base::OnceCallback<void __cdecl(void)>::Run+0x30 (Inline Function @ 672a8232) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\callback.h @ 64]
34 04dee808 671e9639 6a811770 41c89fb0 00000000 chrome_670c0000!base::debug::TaskAnnotator::RunTask+0xe6 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\debug\task_annotator.cc @ 57]
35 (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::IncomingTaskQueue::RunTask+0x801295e4 (Inline Function @ 671e9639) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\message_loop\incoming_task_queue.cc @ 130]
36 04deece4 671e99cd 04deed00 273877c0 00000000 chrome_670c0000!base::MessageLoop::RunTask+0x385 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\message_loop\message_loop.cc @ 393]
37 04deecf8 671e9edc 00000000 6a80ef20 6a2d0fd8 chrome_670c0000!base::MessageLoop::DeferOrRunPendingTask+0x4e (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\message_loop\message_loop.cc @ 404]
38 04deee48 67282f92 00000000 273893e8 00000000 chrome_670c0000!base::MessageLoop::DoWork+0x309 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\message_loop\message_loop.cc @ 450]
39 04deee78 67282a0c 04def0ec 27387cf8 273877c0 chrome_670c0000!base::MessagePumpForUI::DoRunLoop+0x5f (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\message_loop\message_pump_win.cc @ 174]
3a 04deeea4 671d3c6c 273877c0 00000000 27387cf8 chrome_670c0000!base::MessagePumpWin::Run+0x55 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\message_loop\message_pump_win.cc @ 58]
3b (Inline) -------- -------- -------- -------- chrome_670c0000!base::MessageLoop::Run+0x80113c61 (Inline Function @ 671d3c6c) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\message_loop\message_loop.cc @ 344]
3c 04def028 685efd89 00000000 2737f5a8 6a808c34 chrome_670c0000!base::RunLoop::Run+0xef (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\run_loop.cc @ 121]
3d 04def118 67e07102 2737f5b8 6a1f4c54 274ad0a8 chrome_670c0000!ChromeBrowserMainParts::MainMessageLoopRun+0xf0 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\chrome\browser\chrome_browser_main.cc @ 1921]
3e 04def160 67e0850c 274a99c8 274ad0a8 67e0851b chrome_670c0000!content::BrowserMainLoop::RunMainMessageLoopParts+0x50 (FPO: [0,0,4]) (CONV: thiscall) [c:\b\c\b\win_asan\src\content\browser\browser_main_loop.cc @ 1207]
3f 04def1f4 67e0279b 00000000 274a99c8 274a5f40 chrome_670c0000!content::BrowserMainRunnerImpl::Initialize+0x2e8 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\content\browser\browser_main_runner.cc @ 129]
40 04def20c 6851c591 04def39c 00000000 00000010 chrome_670c0000!content::BrowserMain+0x82 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\content\browser\browser_main.cc @ 46]
41 04def2cc 6851c484 04def3b8 04def39c 04def628 chrome_670c0000!content::RunNamedProcessTypeMain+0xb1 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\content\app\content_main_runner.cc @ 429]
42 04def3d4 6853b103 06b51ae8 04def620 06b54f90 chrome_670c0000!content::ContentMainRunnerImpl::Run+0x125 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\content\app\content_main_runner.cc @ 707]
43 04def5e4 6851bb8d 04def614 04def620 6a29945c chrome_670c0000!service_manager::Main+0x21e (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\services\service_manager\embedder\main.cc @ 453]
44 04def618 67981a81 04def64c 00000000 6a153304 chrome_670c0000!content::ContentMain+0x27 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\content\app\content_main.cc @ 19]
*** WARNING: Unable to verify checksum for chrome.exe
45 04def670 00b3dcbe 00ad0000 04def694 e3449cee chrome_670c0000!ChromeMain+0xd6 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\chrome\app\chrome_main.cc @ 123]
46 04def6f8 00b3ce32 00ad0000 e3449cee 000006be chrome!MainDllLoader::Launch+0x1ba (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\chrome\app\main_dll_loader_win.cc @ 201]
47 04def838 00b63888 00ad0000 00000000 053723bc chrome!wWinMain+0x217 (FPO: [Non-Fpo]) (CONV: stdcall) [c:\b\c\b\win_asan\src\chrome\app\chrome_exe_main_win.cc @ 231]
48 (Inline) -------- -------- -------- -------- chrome!invoke_main+0x1a (Inline Function @ 00b63888) (CONV: cdecl) [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 118]
49 04def884 75ac62c4 04e75000 75ac62a0 33c509e1 chrome!__scrt_common_main_seh+0xf6 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283]
4a 04def898 77590f69 04e75000 cdc9c0cb 00000000 kernel32!BaseThreadInitThunk+0x24 (FPO: [Non-Fpo])
4b 04def8e0 77590f34 ffffffff 775b2eab 00000000 ntdll_77530000!__RtlUserThreadStart+0x2f (FPO: [SEH])
4c 04def8f0 00000000 00b63900 04e75000 00000000 ntdll_77530000!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])


Here's a typical call stack:
0:000:x86> kv
 # ChildEBP RetAddr  Args to Child              
00 04decf54 6fe57899 04df0000 00000001 0000003e syzyasan_rtl!agent::common::WalkStackImpl+0xab (FPO: [Non-Fpo]) (CONV: cdecl) [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\common\stack_walker.cc @ 173]
01 04decf80 6fe57108 04ded038 04ded02c 04ded028 syzyasan_rtl!agent::common::WalkStack+0x59 (FPO: [Non-Fpo]) (CONV: cdecl) [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\common\stack_walker.cc @ 131]
02 04decf98 6fe4b579 00000028 00000028 00000000 syzyasan_rtl!agent::common::StackCapture::InitFromStack+0x28 (FPO: [Non-Fpo]) (CONV: thiscall) [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\common\stack_capture.cc @ 108]
03 04ded138 6fe441a3 053a3e20 00000028 00000028 syzyasan_rtl!agent::asan::heap_managers::BlockHeapManager::Allocate+0xc9 (FPO: [Non-Fpo]) (CONV: thiscall) [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\asan\heap_managers\block_heap_manager.cc @ 211]
04 (Inline) -------- -------- -------- -------- syzyasan_rtl!agent::asan::WindowsHeapAdapter::HeapAlloc+0x3 (Inline Function @ 6fe441a3) (CONV: cdecl) [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\asan\windows_heap_adapter.cc @ 74]
05 04ded158 672751df 053a3e20 00000000 00000028 syzyasan_rtl!asan_HeapAlloc+0x43 (FPO: [Non-Fpo]) (CONV: stdcall) [e:\b\build\slave\syzygy_official\build\src\syzygy\agent\asan\rtl_impl.cc @ 103]
06 (Inline) -------- -------- -------- -------- chrome_670c0000!base::allocator::WinHeapMalloc+0x19 (Inline Function @ 672751df) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\allocator\winheap_stubs_win.cc @ 34]
07 04ded16c 671a2b0c 6a810620 00000028 00000000 chrome_670c0000!`anonymous namespace'::DefaultWinHeapMallocImpl+0x1c (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\allocator\allocator_shim_default_dispatch_to_winheap.cc @ 18]
08 (Inline) -------- -------- -------- -------- chrome_670c0000!ShimMalloc+0x1e (Inline Function @ 671a2b0c) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\allocator\allocator_shim.cc @ 194]
09 04ded18c 6997efbf 00000028 04ded1b0 68b5cb2e chrome_670c0000!malloc+0x18 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\allocator\allocator_shim_override_ucrt_symbols_win.h @ 51]
0a 04ded198 68b5cb2e 00000028 00000000 00000000 chrome_670c0000!operator new+0x1a (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt\vcstartup\src\heap\new_scalar.cpp @ 19]
0b 04ded1b0 68b60a6f 0000000a 00000000 04ded26c chrome_670c0000!re2::SparseArray<int>::SparseArray<int>+0x2e (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\third_party\re2\src\util\sparse_array.h @ 503]
0c 04ded240 68b70443 00000000 00000009 04ded26c chrome_670c0000!re2::Prog::Flatten+0x45 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\third_party\re2\src\re2\prog.cc @ 573]
0d 04ded258 68b7003c 04ded4e0 04ded490 04ded480 chrome_670c0000!re2::Compiler::Finish+0x4e (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\third_party\re2\src\re2\compile.cc @ 1205]
0e 04ded2f0 68b70063 004f77c0 00000000 00555555 chrome_670c0000!re2::Compiler::Compile+0x133 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\third_party\re2\src\re2\compile.cc @ 1186]
0f 04ded308 68b5dd4c 00555555 00000000 0000000f chrome_670c0000!re2::Regexp::CompileToProg+0x11 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\third_party\re2\src\re2\compile.cc @ 1224]
10 04ded420 68b5cf91 04ded454 04ded43c 7a85dbbc chrome_670c0000!re2::RE2::Init+0x25d (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\third_party\re2\src\re2\re2.cc @ 212]
11 04ded45c 6a018d83 6a7c0110 04ded478 7a85d930 chrome_670c0000!re2::RE2::RE2+0x6d (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\third_party\re2\src\re2\re2.cc @ 109]
12 (Inline) -------- -------- -------- -------- chrome_670c0000!autofill::?A0x3b6280d5::StripDigitsIfRequired+0x82f58d55 (Inline Function @ 6a018d83) (CONV: cdecl) [c:\b\c\b\win_asan\src\components\autofill\core\common\signatures_util.cc @ 28]
13 04ded5fc 69d83256 7a85dbbc 7a330edc 7a85d930 chrome_670c0000!autofill::CalculateFormSignature+0x10d (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\components\autofill\core\common\signatures_util.cc @ 55]
14 04ded678 69d75e48 7a85d930 2f1e9f30 04deddac chrome_670c0000!password_manager::PasswordFormManager::DoesManage+0x1f2 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\components\password_manager\core\browser\password_form_manager.cc @ 340]
15 04ded6ec 694882f2 2f1e9f30 04deddac 7a85f3a0 chrome_670c0000!password_manager::PasswordManager::CreatePendingLoginManagers+0xf8 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\components\password_manager\core\browser\password_manager.cc @ 542]
16 04ded700 694883c7 04deddac 04dee11c 2c3d1068 chrome_670c0000!password_manager::ContentPasswordManagerDriver::OnPasswordFormsParsedNoRenderCheck+0x16 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\components\password_manager\content\browser\content_password_manager_driver.cc @ 209]
17 04ded718 684add07 7a85f3a0 3e1fbc68 6a2cfe50 chrome_670c0000!password_manager::ContentPasswordManagerDriver::PasswordFormsParsed+0x40 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\components\password_manager\content\browser\content_password_manager_driver.cc @ 204]
18 04deddd4 6948762e 2f1e9f38 04dee11c 04dedec0 chrome_670c0000!autofill::mojom::PasswordManagerDriverStubDispatch::Accept+0x156 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\out\release\gen\components\autofill\content\common\autofill_driver.mojom.cc @ 1830]
19 04dedde4 6867c161 04dee11c 3e1fbc88 41ea568c chrome_670c0000!autofill::mojom::PasswordManagerDriverStub<mojo::RawPtrImplRefTraits<autofill::mojom::PasswordManagerDriver> >::Accept+0x18 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\out\release\gen\components\autofill\content\common\autofill_driver.mojom.h @ 466]
1a 04dedec0 6867efa3 00000000 3e1fbc68 3e1fbc68 chrome_670c0000!mojo::InterfaceEndpointClient::HandleValidatedMessage+0x246 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\interface_endpoint_client.cc @ 420]
1b 04dedf8c 6867bdda 04dee11c 3d9d2750 00000000 chrome_670c0000!mojo::FilterChain::Accept+0x7a (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\filter_chain.cc @ 41]
1c 04dee054 68675384 04dee11c 3d9d28d0 3d9d2750 chrome_670c0000!mojo::InterfaceEndpointClient::HandleIncomingMessage+0x68 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\interface_endpoint_client.cc @ 305]
1d 04dee154 6867326c 04dee254 00000000 273883d0 chrome_670c0000!mojo::internal::MultiplexRouter::ProcessIncomingMessage+0x29c (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\multiplex_router.cc @ 885]
1e 04dee298 6867efa3 04dee42c 3d9d2784 3d9d286c chrome_670c0000!mojo::internal::MultiplexRouter::Accept+0x11d (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\multiplex_router.cc @ 604]
1f 04dee364 68670a0a 04dee42c 3d9d286c 3d9d2784 chrome_670c0000!mojo::FilterChain::Accept+0x7a (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\filter_chain.cc @ 41]
20 04dee4ac 6867067b 6a8e2220 00000000 3d9d2784 chrome_670c0000!mojo::Connector::ReadSingleMessage+0x16d (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\connector.cc @ 439]
21 04dee588 68670246 422e7e18 425f5be8 422e7e1b chrome_670c0000!mojo::Connector::ReadAllAvailableMessages+0x3f (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\connector.cc @ 469]
22 04dee64c 689ea648 00000000 04dee66c 67dbea70 chrome_670c0000!mojo::Connector::OnHandleReadyInternal+0x7e (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\bindings\lib\connector.cc @ 376]
23 (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::FunctorTraits<void (__thiscall net::HttpProxyClientSocketWrapper::*)(int),void>::Invoke+0x8192a637 (Inline Function @ 689ea648) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 194]
24 (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::InvokeHelper<0,void>::MakeItSo+0x8192a637 (Inline Function @ 689ea648) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 277]
25 (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::Invoker<base::internal::BindState<void (__thiscall net::HttpProxyClientSocketWrapper::*)(int),base::internal::UnretainedWrapper<net::HttpProxyClientSocketWrapper> >,void __cdecl(int)>::RunImpl+0x8192a637 (Inline Function @ 689ea648) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 349]
26 04dee658 67dbea70 422e8160 04dee678 425f5bc0 chrome_670c0000!base::internal::Invoker<base::internal::BindState<void (__thiscall net::HttpProxyClientSocketWrapper::*)(int),base::internal::UnretainedWrapper<net::HttpProxyClientSocketWrapper> >,void __cdecl(int)>::Run+0x11 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 334]
27 (Inline) -------- -------- -------- -------- chrome_670c0000!base::RepeatingCallback<void __cdecl(unsigned int)>::Run+0x80cfea5c (Inline Function @ 67dbea70) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\callback.h @ 92]
28 04dee66c 683ccad4 422e7e18 00000000 41c89fc8 chrome_670c0000!mojo::SimpleWatcher::DiscardReadyState+0x1f (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\mojo\public\cpp\system\simple_watcher.h @ 193]
29 (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::FunctorTraits<void (__cdecl*)(base::RepeatingCallback<void __cdecl(enum extensions::BluetoothApiSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)> const &,enum device::BluetoothSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &),void>::Invoke+0x8130cab1 (Inline Function @ 683ccad4) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 149]
2a (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::InvokeHelper<0,void>::MakeItSo+0x8130cab1 (Inline Function @ 683ccad4) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 277]
2b (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::Invoker<base::internal::BindState<void (__cdecl*)(base::RepeatingCallback<void __cdecl(enum extensions::BluetoothApiSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)> const &,enum device::BluetoothSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &),base::RepeatingCallback<void __cdecl(enum extensions::BluetoothApiSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)> >,void __cdecl(enum device::BluetoothSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)>::RunImpl+0x8130cab1 (Inline Function @ 683ccad4) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 349]
2c 04dee680 68686b9b 422e7e00 04dee768 41c89fc8 chrome_670c0000!base::internal::Invoker<base::internal::BindState<void (__cdecl*)(base::RepeatingCallback<void __cdecl(enum extensions::BluetoothApiSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)> const &,enum device::BluetoothSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &),base::RepeatingCallback<void __cdecl(enum extensions::BluetoothApiSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)> >,void __cdecl(enum device::BluetoothSocket::ErrorReason,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)>::Run+0x15 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 331]
2d (Inline) -------- -------- -------- -------- chrome_670c0000!base::RepeatingCallback<void __cdecl(unsigned int,mojo::HandleSignalsState const &)>::Run+0x815c6b87 (Inline Function @ 68686b9b) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\callback.h @ 92]
2e 04dee75c 690b7ca1 6a2841a8 00000000 41c89fc8 chrome_670c0000!mojo::SimpleWatcher::OnHandleReady+0xcc (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\mojo\public\cpp\system\simple_watcher.cc @ 276]
2f (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::FunctorTraits<void (__thiscall storage::LocalFileStreamWriter::*)(net::IOBuffer *,int,base::RepeatingCallback<void __cdecl(int)> const &),void>::Invoke+0x81ff7c7d (Inline Function @ 690b7ca1) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 194]
30 04dee774 68685fa5 41c89fc4 41c89fd8 41c89fd4 chrome_670c0000!base::internal::InvokeHelper<1,void>::MakeItSo<void (__thiscall storage::LocalFileStreamWriter::*const &)(net::IOBuffer *,int,base::RepeatingCallback<void __cdecl(int)> const &),base::WeakPtr<storage::LocalFileStreamWriter> const &,net::IOBuffer *,int const &,base::RepeatingCallback<void __cdecl(int)> const &>+0x2e (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 297]
31 04dee790 68686c33 41c89fc4 41c89fc8 41c89fb0 chrome_670c0000!base::internal::Invoker<base::internal::BindState<void (__thiscall mojo::SimpleWatcher::*)(int,unsigned int,mojo::HandleSignalsState const &),base::WeakPtr<mojo::SimpleWatcher>,int,unsigned int,mojo::HandleSignalsState>,void __cdecl(void)>::RunImpl<void (__thiscall mojo::SimpleWatcher::*const &)(int,unsigned int,mojo::HandleSignalsState const &),std::tuple<base::WeakPtr<mojo::SimpleWatcher>,int,unsigned int,mojo::HandleSignalsState> const &,0,1,2,3>+0x1b (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 349]
32 04dee7a4 672a8232 41c89fb0 273877c0 04deed00 chrome_670c0000!base::internal::Invoker<base::internal::BindState<void (__thiscall mojo::SimpleWatcher::*)(int,unsigned int,mojo::HandleSignalsState const &),base::WeakPtr<mojo::SimpleWatcher>,int,unsigned int,mojo::HandleSignalsState>,void __cdecl(void)>::Run+0x16 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\base\bind_internal.h @ 331]
33 (Inline) -------- -------- -------- -------- chrome_670c0000!base::OnceCallback<void __cdecl(void)>::Run+0x30 (Inline Function @ 672a8232) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\callback.h @ 64]
34 04dee808 671e9639 6a811770 41c89fb0 00000000 chrome_670c0000!base::debug::TaskAnnotator::RunTask+0xe6 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\debug\task_annotator.cc @ 57]
35 (Inline) -------- -------- -------- -------- chrome_670c0000!base::internal::IncomingTaskQueue::RunTask+0x801295e4 (Inline Function @ 671e9639) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\message_loop\incoming_task_queue.cc @ 130]
36 04deece4 671e99cd 04deed00 273877c0 00000000 chrome_670c0000!base::MessageLoop::RunTask+0x385 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\message_loop\message_loop.cc @ 393]
37 04deecf8 671e9edc 00000000 6a80ef20 6a2d0fd8 chrome_670c0000!base::MessageLoop::DeferOrRunPendingTask+0x4e (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\message_loop\message_loop.cc @ 404]
38 04deee48 67282f92 00000000 273893e8 00000000 chrome_670c0000!base::MessageLoop::DoWork+0x309 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\message_loop\message_loop.cc @ 450]
39 04deee78 67282a0c 04def0ec 27387cf8 273877c0 chrome_670c0000!base::MessagePumpForUI::DoRunLoop+0x5f (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\message_loop\message_pump_win.cc @ 174]
3a 04deeea4 671d3c6c 273877c0 00000000 27387cf8 chrome_670c0000!base::MessagePumpWin::Run+0x55 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\message_loop\message_pump_win.cc @ 58]
3b (Inline) -------- -------- -------- -------- chrome_670c0000!base::MessageLoop::Run+0x80113c61 (Inline Function @ 671d3c6c) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\message_loop\message_loop.cc @ 344]
3c 04def028 685efd89 00000000 2737f5a8 6a808c34 chrome_670c0000!base::RunLoop::Run+0xef (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\base\run_loop.cc @ 121]
3d 04def118 67e07102 2737f5b8 6a1f4c54 274ad0a8 chrome_670c0000!ChromeBrowserMainParts::MainMessageLoopRun+0xf0 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\chrome\browser\chrome_browser_main.cc @ 1921]
3e 04def160 67e0850c 274a99c8 274ad0a8 67e0851b chrome_670c0000!content::BrowserMainLoop::RunMainMessageLoopParts+0x50 (FPO: [0,0,4]) (CONV: thiscall) [c:\b\c\b\win_asan\src\content\browser\browser_main_loop.cc @ 1207]
3f 04def1f4 67e0279b 00000000 274a99c8 274a5f40 chrome_670c0000!content::BrowserMainRunnerImpl::Initialize+0x2e8 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\content\browser\browser_main_runner.cc @ 129]
40 04def20c 6851c591 04def39c 00000000 00000010 chrome_670c0000!content::BrowserMain+0x82 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\content\browser\browser_main.cc @ 46]
41 04def2cc 6851c484 04def3b8 04def39c 04def628 chrome_670c0000!content::RunNamedProcessTypeMain+0xb1 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\content\app\content_main_runner.cc @ 429]
42 04def3d4 6853b103 06b51ae8 04def620 06b54f90 chrome_670c0000!content::ContentMainRunnerImpl::Run+0x125 (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\content\app\content_main_runner.cc @ 707]
43 04def5e4 6851bb8d 04def614 04def620 6a29945c chrome_670c0000!service_manager::Main+0x21e (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\services\service_manager\embedder\main.cc @ 453]
44 04def618 67981a81 04def64c 00000000 6a153304 chrome_670c0000!content::ContentMain+0x27 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\content\app\content_main.cc @ 19]
*** WARNING: Unable to verify checksum for chrome.exe
45 04def670 00b3dcbe 00ad0000 04def694 e3449cee chrome_670c0000!ChromeMain+0xd6 (FPO: [Non-Fpo]) (CONV: cdecl) [c:\b\c\b\win_asan\src\chrome\app\chrome_main.cc @ 123]
46 04def6f8 00b3ce32 00ad0000 e3449cee 000006be chrome!MainDllLoader::Launch+0x1ba (FPO: [Non-Fpo]) (CONV: thiscall) [c:\b\c\b\win_asan\src\chrome\app\main_dll_loader_win.cc @ 201]
47 04def838 00b63888 00ad0000 00000000 053723bc chrome!wWinMain+0x217 (FPO: [Non-Fpo]) (CONV: stdcall) [c:\b\c\b\win_asan\src\chrome\app\chrome_exe_main_win.cc @ 231]
48 (Inline) -------- -------- -------- -------- chrome!invoke_main+0x1a (Inline Function @ 00b63888) (CONV: cdecl) [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 118]
49 04def884 75ac62c4 04e75000 75ac62a0 33c509e1 chrome!__scrt_common_main_seh+0xf6 (FPO: [Non-Fpo]) (CONV: cdecl) [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283]
4a 04def898 77590f69 04e75000 cdc9c0cb 00000000 kernel32!BaseThreadInitThunk+0x24 (FPO: [Non-Fpo])
4b 04def8e0 77590f34 ffffffff 775b2eab 00000000 ntdll_77530000!__RtlUserThreadStart+0x2f (FPO: [SEH])
4c 04def8f0 00000000 00b63900 04e75000 00000000 ntdll_77530000!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

looks like a duplicate of  issue 777206 

the RE2 code was removed, however, the code still runs on the UI thread.

Maxim, any reason this can't be posted to a non-interactive thread?

Is it expected to have 2256 forms to process in one go?

Well, we can post it to UI thread, but then we should do it for the whole form processing...

I am a bit confused about 2K forms. Is it a particular test site? 
The password manager usually faces <=5 forms on the page so it shouldn't be a bottleneck. 

The point is that it should *not* run on the UI thread.

I don't know whether those forms come from, I only have inbox, chat, irc, and calendar open

The 2256 forms is my misinterpretation of the crash dump.
I thought the forms vector was a vector of pointers, but it's a vector of objects. Each object is 1128 bytes large, so this is only processing two forms at a go here - sorry about the confusion.

Thanks for clarification siggi@ but then I have another question. 

Is the password manager code (CalculateFormSignature) still a problem? #13 points to callstack which doesn't exist in since M63. Or is the problem in Feedback component?

I can't get symbols to load for any of the crash dumps (I looked at two). The ETW call stacks look weird, but I assume there must be some truth to them, yes?

The traces and crash dumps are all from 64.0.3244.1 so there is clearly something going on there beyond M63.

The ETW data showing CPU consumption is best understood through the flame graph attached to comment #6. I'm not familiar enough with the code to know which parts are plausible and which parts are likely to be stack walk issues. I'm not sure why there would be stack walk issues - ETW stack walking is generally very reliable, and if it is uncertain it halts rather than producing garbage. So, any oddness in the ETW stacks is likely to be from problems in symbol lookup rather than problems with the stack walking.

The stacks are from M64. On trunk, we no longer have the calls to RE2, however, that doesn't change that we're doing potentially unbounded work on the UI thread (which we shouldn't)

After a quick chat with kolos@, there might be more alternatives to do less work on UI thread, assuming that what takes a long time is the CalculateFormSignature call:

(1) We can calculate it in renderer, directly when the form is parsed from a WebFormElement (and since then cache it inside PasswordForm). My understanding is that renderer should not be used for heavy computations, though. Is that correct?

(2) We can spin a signature-computing task on a TaskRunner with MayBlock() traits in browser, as soon as the form data enters the browser process. The rest of the code would need to be rewritten to wait for a completion callback from the signature computation.

(3) An alternative to (2) would be for the rest of the code not to wait for the signature computing completion, but to handle the missing signature gracefully. That might lead to surprising bugs and be actually more complex than (2), though.

it's ok to do computations in the renderer, but just as in the browser process, you shouldn't do them on the UI thread (or webkit thread on the renderer side)

> The stacks are from M64. On trunk, we no longer have the calls to RE2...

I'm worried that there is a contradiction here. The stacks are from M64, which is trunk, and they show calls to RE2. Do we understand why RE2 execution is showing up as a bottleneck after the RE2 code was removed? Either it wasn't fully removed, or this is some other usage of RE2, or else the stacks or symbols are lying to us. Or am I missing something here?

Also, do we have UMA data to record how expensive these calculations are in the non syzygy case? Our tolerance for CPU usage is higher on non-UI threads but not infinite. It would be good to know whether the cost is great enough to be concerning even when moved off of the UI thread.

The RE2 calls were removed just recently, and I produced the profiles on a slightly older Canary

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a5fa3ee8b4421717d29694e3dd77cdd7ea8c5200

commit a5fa3ee8b4421717d29694e3dd77cdd7ea8c5200
Author: Maxim Kolosovskiy <kolos@chromium.org>
Date: Wed Nov 29 08:58:51 2017

[Password Manager] Calculate form signature of a observed form of PasswordFormManager only once

The signature of |PasswordFormManager::observed_form_| is used for form matching. Since |observed_form| is const field, its signature doesn't need to be re-calculated many times.

Bug:  786941 
Change-Id: I9705a64055c18ca83fd50dce869468734adee337
Reviewed-on: https://chromium-review.googlesource.com/793835
Commit-Queue: Maxim Kolosovskiy <kolos@chromium.org>
Reviewed-by: Vaclav Brozek <vabr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#520050}
[modify] https://crrev.com/a5fa3ee8b4421717d29694e3dd77cdd7ea8c5200/components/password_manager/core/browser/password_form_manager.cc
[modify] https://crrev.com/a5fa3ee8b4421717d29694e3dd77cdd7ea8c5200/components/password_manager/core/browser/password_form_manager.h

I believe we need to collect the data because the profile was outdated and check whether the issue still happen. 

To me, PasswordFormsParsed event is not a blocking operation, there is no massive processing. Involving RE made it surprisingly slow, but RE was removed.

I have a question about the flame chart. As I understand, it is imprecise. For example, HideManualFallbackForSaving cannot call CreatePendingCredentials. Is it expected that a flame chart is just an approximation?  

The flame graph shouldn't be an approximation, as long as the stack walking and the symbol lookup are correct. However I agree that the flame graph seems to show impossible data.

I think that the likely culprit is symbol lookup errors, caused by syzyasan's manipulation of the binary. It would be great if the syzyasan team could investigate this. I don't know how common a problem this is.