Chrome DLL blacklist leads to crashes when AppVerifier is enabled with sandbox enabled
Reported by
mwodr...@microsoft.com,
Aug 3 2017
|
|||||||||
Issue descriptionUserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36 Steps to reproduce the problem: 1. Enable AppVerifier on chrome.exe 2. Launch Chrome, you will see tabs fail to open 3. Attached a debugger to chrome.exe with the -o option to debug children, you see a crash in one of the child chrome processes in NTDLL: 3:041> r rax=0000000000000000 rbx=00007fffde85dae0 rcx=00007fffde85dae0 rdx=ffffffffffffffff rsi=00007fffde85ef20 rdi=0000000000000000 rip=00007fffde71c7b9 rsp=000000fe0dbfbdc0 rbp=0000000000000001 r8=0000000000000000 r9=0000000000000000 r10=0000000000000000 r11=00007fffde7a3c9b r12=0000000000000000 r13=0000021707bf02a0 r14=000000fe0d947000 r15=0000000000000000 iopl=0 nv up ei pl nz ac pe cy cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010213 ntdll!RtlpIncrementCriticalSectionContentionCount+0xc [inlined in ntdll!RtlpWaitOnCriticalSection+0x85]: 00007fff`de71c7b9 ff4024 inc dword ptr [rax+24h] ds:00000000`00000024=???????? What is the expected behavior? What went wrong? From debugging it looks like this is due to the Chrome sandbox DLL blacklist feature, which places an API hook in sandboxed Chrome processes (e.g. renderer), on ntdll!NtMapViewOfSection. When AppVerifier is enabled for the process, NTDLL will load Verifier.dll as the first DLL module to be loaded, causing this API hook to be invoked. This is done before the process heaps are initialized. (Since App Verifier can be used to replace the heaps for PageHeap). The API hook invokes the Chrome DLL Blacklist logic, which ends up calling into the NTDLL Heap code. Since the heap manager is not yet initialized for the process, this leads to the crash we see. See the call in Chrome to ntdll!RtlCreateHeap here: 3:041> ub chrome!IsSandboxedProcess+0x4f58 chrome!IsSandboxedProcess+0x4f3b: 00007ff6`e140923b 753d jne chrome!IsSandboxedProcess+0x4f7a (00007ff6`e140927a) 00007ff6`e140923d 48895c2428 mov qword ptr [rsp+28h],rbx 00007ff6`e1409242 8d4b02 lea ecx,[rbx+2] 00007ff6`e1409245 4533c9 xor r9d,r9d 00007ff6`e1409248 48895c2420 mov qword ptr [rsp+20h],rbx 00007ff6`e140924d 4533c0 xor r8d,r8d 00007ff6`e1409250 33d2 xor edx,edx 00007ff6`e1409252 ff15c8c00900 call qword ptr [chrome!IsSandboxedProcess+0xa1020 (00007ff6`e14a5320)] 3:041> dqs 00007ff6`e14a5320 l1 00007ff6`e14a5320 00007fff`de74c2a0 ntdll!RtlCreateHeap It looks like a way to avoid this crash would be to not use the heap for the operations needed by the DLL blacklist logic (use stack buffers / pointers to data in the PE module). Crashed report ID: How much crashed? Just one tab Is it a problem with a plugin? No Did this work before? N/A Chrome version: 60.0.3112.90 Channel: n/a OS Version: 10.0 Flash Version: Running chrome with --no-sandbox avoids the crash, I'm not sure if you care to support AppVerifier on normal sandboxed scenarios? Some other Active bugs in the tracker seem to indicate you'd like to enable this scenario? Anyway, I thought I'd provide the info on the root-cause in case it wasn't already known.
,
Aug 4 2017
,
Aug 4 2017
Hrmm.. yes, that will happen. I wondered briefly why this doesn't kill the browser process too since we also hook NtMapViewOfSection there but that's because we install the hook in the browser process a tiny bit later on after the process heaps are initialized. https://codesearch.chromium.org/chromium/src/sandbox/win/src/target_interceptions.cc?l=37 is the thing that calls RtlCreateHeap and while we could re-jigger this not to (probably by using an image section as scratch space as the original report suggests) it would be a little bit of work because we use the created heap in quite a few places. +robliao@ was trying out AppVerifier in the not too distant past and might have an opinion how much we'd like to support running Chrome under AppVerifier with the sandbox enabled. Oh, and thanks for the detailed report! If you'd like slightly more convenient debugging in the future, Chrome's symbol server is here: https://chromium-browser-symsrv.commondatastorage.googleapis.com
,
Aug 4 2017
Thanks for the symbol server info! Looks like the heap is used to store the Unicode version of the ANSI strings (PE module name, filename). You could consider changing the code to just using the ANSI strings, which could be pointers to the string data in the mapped PE module / NTDLL Loader data. Another option is to switch to using stack buffers for the strings [MAX_PATH should be enough for each]. Does anyone hold a pointer to the strings after the agent->OnDllLoad call has returned? (I guess not since the heap buffers are freed immediately before TargetNtMapViewOfSection returns)
,
Aug 4 2017
I am adding 'RB-Stable' for M60. Please feel free to edit if required. Thank you!
,
Aug 4 2017
IIRC, Chrome has always needed --no-sandbox (and occassionally some heap flags) to run under AppVerifier. If we can get it to run with the sandbox enabled, that would certainly be nice, but I'm not sure how possible that is as we lock down many things in the sandbox. wfh@ would have better clarity on that.
,
Aug 4 2017
I'm not sure this qualifies for RB-Stable. Chrome historically has had problems running under AppVerifier out of the box, so I'm clearing the flag.
,
Feb 11 2018
,
Feb 20 2018
When testing fixes for this bug, don't forget to turn off App Verifier before typing "git cl web" - otherwise bad things happen. crrev.com/c/925821 is running try jobs now
,
Feb 20 2018
,
Feb 20 2018
,
Feb 22 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/da69db99fc30649947ba1f78b48fc9e6afe0de0b commit da69db99fc30649947ba1f78b48fc9e6afe0de0b Author: Bruce Dawson <brucedawson@chromium.org> Date: Thu Feb 22 01:50:38 2018 Fix Windows sandbox to work with Application Verifier Application Verifier is a useful tool for tracking down bugs on Windows. However Chrome's sandbox has not played well with App Verifier - it ends up initializing a heap before the process is ready for this. This change teaches the NtMapViewOfSection to skip InitHeap when handling the App Verifier DLLs, which don't need patching anyway. With this change I can run Chrome with the default App Verifier settings enabled with the exception of Handles and TLS (those failures probably don't represent real bugs), without using --no-sandbox. Bug: 752344 Change-Id: Id1fd4be9c0f080513bbdb649ed5cf2637df8474c Reviewed-on: https://chromium-review.googlesource.com/925821 Reviewed-by: James Forshaw <forshaw@chromium.org> Reviewed-by: Will Harris <wfh@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org> Cr-Commit-Position: refs/heads/master@{#538307} [modify] https://crrev.com/da69db99fc30649947ba1f78b48fc9e6afe0de0b/sandbox/win/src/sandbox_nt_util.cc [modify] https://crrev.com/da69db99fc30649947ba1f78b48fc9e6afe0de0b/sandbox/win/src/sandbox_nt_util.h [modify] https://crrev.com/da69db99fc30649947ba1f78b48fc9e6afe0de0b/sandbox/win/src/target_interceptions.cc
,
Feb 23 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b commit 1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b Author: Bruce Dawson <brucedawson@chromium.org> Date: Fri Feb 23 19:18:14 2018 Revert "Fix Windows sandbox to work with Application Verifier" This reverts commit da69db99fc30649947ba1f78b48fc9e6afe0de0b. Reason for revert: Suspected of causing the crashes tracked by crbug.com/814641 Original change's description: > Fix Windows sandbox to work with Application Verifier > > Application Verifier is a useful tool for tracking down bugs on Windows. > However Chrome's sandbox has not played well with App Verifier - it ends > up initializing a heap before the process is ready for this. This change > teaches the NtMapViewOfSection to skip InitHeap when handling the App > Verifier DLLs, which don't need patching anyway. > > With this change I can run Chrome with the default App Verifier settings > enabled with the exception of Handles and TLS (those failures probably > don't represent real bugs), without using --no-sandbox. > > Bug: 752344 > Change-Id: Id1fd4be9c0f080513bbdb649ed5cf2637df8474c > Reviewed-on: https://chromium-review.googlesource.com/925821 > Reviewed-by: James Forshaw <forshaw@chromium.org> > Reviewed-by: Will Harris <wfh@chromium.org> > Commit-Queue: Bruce Dawson <brucedawson@chromium.org> > Cr-Commit-Position: refs/heads/master@{#538307} TBR=brucedawson@chromium.org,forshaw@chromium.org,wfh@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. Bug: 752344 Change-Id: Icd8feea1779f2891389762db0773ecf0ed9ad762 Reviewed-on: https://chromium-review.googlesource.com/934862 Reviewed-by: Bruce Dawson <brucedawson@chromium.org> Reviewed-by: Will Harris <wfh@chromium.org> Commit-Queue: Bruce Dawson <brucedawson@chromium.org> Cr-Commit-Position: refs/heads/master@{#538851} [modify] https://crrev.com/1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b/sandbox/win/src/sandbox_nt_util.cc [modify] https://crrev.com/1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b/sandbox/win/src/sandbox_nt_util.h [modify] https://crrev.com/1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b/sandbox/win/src/target_interceptions.cc
,
Apr 23 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a695cf342559b85f2871a5851172fae9d0af3ae3 commit a695cf342559b85f2871a5851172fae9d0af3ae3 Author: Bruce Dawson <brucedawson@chromium.org> Date: Mon Apr 23 11:46:21 2018 Fix Windows sandbox to work with Application Verifier Application Verifier is a useful tool for tracking down bugs on Windows. However Chrome's sandbox has not played well with App Verifier - it ends up initializing a heap before the process is ready for this. This change teaches the NtMapViewOfSection to skip InitHeap when handling the App Verifier DLLs, which don't need patching anyway. With this change I can run Chrome with the default App Verifier settings enabled with the exception of Handles and TLS (those failures probably don't represent real bugs), without using --no-sandbox. This is an update to crrev.com/c/925821 with null checks added. The previous CL was reverted due to crashes on some machines, which unfortunately were never diagnosed. Bug: 752344 ,792289, 814641 Change-Id: I52eadcdbc2c1f3dcc76bddf3b97e903143461757 Reviewed-on: https://chromium-review.googlesource.com/1014666 Commit-Queue: Bruce Dawson <brucedawson@chromium.org> Reviewed-by: Will Harris <wfh@chromium.org> Cr-Commit-Position: refs/heads/master@{#552672} [modify] https://crrev.com/a695cf342559b85f2871a5851172fae9d0af3ae3/sandbox/win/src/sandbox_nt_util.cc [modify] https://crrev.com/a695cf342559b85f2871a5851172fae9d0af3ae3/sandbox/win/src/sandbox_nt_util.h [modify] https://crrev.com/a695cf342559b85f2871a5851172fae9d0af3ae3/sandbox/win/src/target_interceptions.cc
,
Apr 25 2018
Looks like the reland stuck. Closing as fixed. |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by manoranj...@chromium.org
, Aug 4 2017Labels: Needs-Triage-M60