New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 752344 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Apr 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 2
Type: Bug

Blocking:
issue 807500



Sign in to add a comment

Chrome DLL blacklist leads to crashes when AppVerifier is enabled with sandbox enabled

Reported by mwodr...@microsoft.com, Aug 3 2017

Issue description

UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36

Steps to reproduce the problem:
1. Enable AppVerifier on chrome.exe
2. Launch Chrome, you will see tabs fail to open
3. Attached a debugger to chrome.exe with the -o option to debug children, you see a crash in one of the child chrome processes in NTDLL:

3:041> r
rax=0000000000000000 rbx=00007fffde85dae0 rcx=00007fffde85dae0
rdx=ffffffffffffffff rsi=00007fffde85ef20 rdi=0000000000000000
rip=00007fffde71c7b9 rsp=000000fe0dbfbdc0 rbp=0000000000000001
 r8=0000000000000000  r9=0000000000000000 r10=0000000000000000
r11=00007fffde7a3c9b r12=0000000000000000 r13=0000021707bf02a0
r14=000000fe0d947000 r15=0000000000000000
iopl=0         nv up ei pl nz ac pe cy
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010213
ntdll!RtlpIncrementCriticalSectionContentionCount+0xc [inlined in ntdll!RtlpWaitOnCriticalSection+0x85]:
00007fff`de71c7b9 ff4024          inc     dword ptr [rax+24h] ds:00000000`00000024=????????

What is the expected behavior?

What went wrong?
From debugging it looks like this is due to the Chrome sandbox DLL blacklist feature, which places an API hook in sandboxed Chrome processes (e.g. renderer), on ntdll!NtMapViewOfSection. 

When AppVerifier is enabled for the process, NTDLL will load Verifier.dll as the first DLL module to be loaded, causing this API hook to be invoked. This is done before the process heaps are initialized. (Since App Verifier can be used to replace the heaps for PageHeap).

The API hook invokes the Chrome DLL Blacklist logic, which ends up calling into the NTDLL Heap code. Since the heap manager is not yet initialized for the process, this leads to the crash we see.

See the call in Chrome to ntdll!RtlCreateHeap here:
3:041> ub chrome!IsSandboxedProcess+0x4f58
chrome!IsSandboxedProcess+0x4f3b:
00007ff6`e140923b 753d            jne     chrome!IsSandboxedProcess+0x4f7a (00007ff6`e140927a)
00007ff6`e140923d 48895c2428      mov     qword ptr [rsp+28h],rbx
00007ff6`e1409242 8d4b02          lea     ecx,[rbx+2]
00007ff6`e1409245 4533c9          xor     r9d,r9d
00007ff6`e1409248 48895c2420      mov     qword ptr [rsp+20h],rbx
00007ff6`e140924d 4533c0          xor     r8d,r8d
00007ff6`e1409250 33d2            xor     edx,edx
00007ff6`e1409252 ff15c8c00900    call    qword ptr [chrome!IsSandboxedProcess+0xa1020 (00007ff6`e14a5320)]
3:041> dqs 00007ff6`e14a5320 l1
00007ff6`e14a5320  00007fff`de74c2a0 ntdll!RtlCreateHeap 

It looks like a way to avoid this crash would be to not use the heap for the operations needed by the DLL blacklist logic (use stack buffers / pointers to data in the PE module).

Crashed report ID: 

How much crashed? Just one tab

Is it a problem with a plugin? No 

Did this work before? N/A 

Chrome version: 60.0.3112.90  Channel: n/a
OS Version: 10.0
Flash Version: 

Running chrome with --no-sandbox avoids the crash, I'm not sure if you care to support AppVerifier on normal sandboxed scenarios? Some other Active bugs in the tracker seem to indicate you'd like to enable this scenario? 
Anyway, I thought I'd provide the info on the root-cause in case it wasn't already known.
 
Cc: brucedaw...@chromium.org scottmg@chromium.org
Labels: Needs-Triage-M60
Cc: penny...@chromium.org wfh@chromium.org robertshield@chromium.org
Cc: siggi@chromium.org robliao@chromium.org
Status: Untriaged (was: Unconfirmed)
Hrmm.. yes, that will happen. I wondered briefly why this doesn't kill the browser process too since we also hook NtMapViewOfSection there but that's because we install the hook in the browser process a tiny bit later on after the process heaps are initialized.

https://codesearch.chromium.org/chromium/src/sandbox/win/src/target_interceptions.cc?l=37 is the thing that calls RtlCreateHeap and while we could re-jigger this not to (probably by using an image section as scratch space as the original report suggests) it would be a little bit of work because we use the created heap in quite a few places.

+robliao@ was trying out AppVerifier in the not too distant past and might have an opinion how much we'd like to support running Chrome under AppVerifier with the sandbox enabled. 

Oh, and thanks for the detailed report! If you'd like slightly more convenient debugging in the future, Chrome's symbol server is here: https://chromium-browser-symsrv.commondatastorage.googleapis.com
Thanks for the symbol server info! Looks like the heap is used to store the Unicode version of the ANSI strings (PE module name, filename). You could consider changing the code to just using the ANSI strings, which could be pointers to the string data in the mapped PE module / NTDLL Loader data.

Another option is to switch to using stack buffers for the strings [MAX_PATH should be enough for each]. Does anyone hold a pointer to the strings after the agent->OnDllLoad call has returned? (I guess not since the heap buffers are freed immediately before TargetNtMapViewOfSection returns)
Cc: abdulsyed@chromium.org bustamante@chromium.org
Labels: ReleaseBlock-Stable M-60
I am adding 'RB-Stable' for M60. Please feel free to edit if required.

Thank you!
IIRC, Chrome has always needed --no-sandbox (and occassionally some heap flags) to run under AppVerifier.

If we can get it to run with the sandbox enabled, that would certainly be nice, but I'm not sure how possible that is as we lock down many things in the sandbox.

wfh@ would have better clarity on that.
Labels: -ReleaseBlock-Stable
I'm not sure this qualifies for RB-Stable. Chrome historically has had problems running under AppVerifier out of the box, so I'm clearing the flag.
Components: Internals>Sandbox
When testing fixes for this bug, don't forget to turn off App Verifier before typing "git cl web" - otherwise bad things happen.

crrev.com/c/925821 is running try jobs now

Owner: brucedaw...@chromium.org
Status: Started (was: Untriaged)
Blocking: 807500
Project Member

Comment 12 by bugdroid1@chromium.org, Feb 22 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/da69db99fc30649947ba1f78b48fc9e6afe0de0b

commit da69db99fc30649947ba1f78b48fc9e6afe0de0b
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Thu Feb 22 01:50:38 2018

Fix Windows sandbox to work with Application Verifier

Application Verifier is a useful tool for tracking down bugs on Windows.
However Chrome's sandbox has not played well with App Verifier - it ends
up initializing a heap before the process is ready for this. This change
teaches the NtMapViewOfSection to skip InitHeap when handling the App
Verifier DLLs, which don't need patching anyway.

With this change I can run Chrome with the default App Verifier settings
enabled with the exception of Handles and TLS (those failures probably
don't represent real bugs), without using --no-sandbox.

Bug:  752344 
Change-Id: Id1fd4be9c0f080513bbdb649ed5cf2637df8474c
Reviewed-on: https://chromium-review.googlesource.com/925821
Reviewed-by: James Forshaw <forshaw@chromium.org>
Reviewed-by: Will Harris <wfh@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#538307}
[modify] https://crrev.com/da69db99fc30649947ba1f78b48fc9e6afe0de0b/sandbox/win/src/sandbox_nt_util.cc
[modify] https://crrev.com/da69db99fc30649947ba1f78b48fc9e6afe0de0b/sandbox/win/src/sandbox_nt_util.h
[modify] https://crrev.com/da69db99fc30649947ba1f78b48fc9e6afe0de0b/sandbox/win/src/target_interceptions.cc

Project Member

Comment 13 by bugdroid1@chromium.org, Feb 23 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b

commit 1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Fri Feb 23 19:18:14 2018

Revert "Fix Windows sandbox to work with Application Verifier"

This reverts commit da69db99fc30649947ba1f78b48fc9e6afe0de0b.

Reason for revert: Suspected of causing the crashes tracked by  crbug.com/814641 

Original change's description:
> Fix Windows sandbox to work with Application Verifier
> 
> Application Verifier is a useful tool for tracking down bugs on Windows.
> However Chrome's sandbox has not played well with App Verifier - it ends
> up initializing a heap before the process is ready for this. This change
> teaches the NtMapViewOfSection to skip InitHeap when handling the App
> Verifier DLLs, which don't need patching anyway.
> 
> With this change I can run Chrome with the default App Verifier settings
> enabled with the exception of Handles and TLS (those failures probably
> don't represent real bugs), without using --no-sandbox.
> 
> Bug:  752344 
> Change-Id: Id1fd4be9c0f080513bbdb649ed5cf2637df8474c
> Reviewed-on: https://chromium-review.googlesource.com/925821
> Reviewed-by: James Forshaw <forshaw@chromium.org>
> Reviewed-by: Will Harris <wfh@chromium.org>
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#538307}

TBR=brucedawson@chromium.org,forshaw@chromium.org,wfh@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  752344 
Change-Id: Icd8feea1779f2891389762db0773ecf0ed9ad762
Reviewed-on: https://chromium-review.googlesource.com/934862
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Will Harris <wfh@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#538851}
[modify] https://crrev.com/1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b/sandbox/win/src/sandbox_nt_util.cc
[modify] https://crrev.com/1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b/sandbox/win/src/sandbox_nt_util.h
[modify] https://crrev.com/1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b/sandbox/win/src/target_interceptions.cc

Project Member

Comment 14 by bugdroid1@chromium.org, Apr 23 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a695cf342559b85f2871a5851172fae9d0af3ae3

commit a695cf342559b85f2871a5851172fae9d0af3ae3
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Mon Apr 23 11:46:21 2018

Fix Windows sandbox to work with Application Verifier

Application Verifier is a useful tool for tracking down bugs on Windows.
However Chrome's sandbox has not played well with App Verifier - it ends
up initializing a heap before the process is ready for this. This change
teaches the NtMapViewOfSection to skip InitHeap when handling the App
Verifier DLLs, which don't need patching anyway.

With this change I can run Chrome with the default App Verifier settings
enabled with the exception of Handles and TLS (those failures probably
don't represent real bugs), without using --no-sandbox.

This is an update to crrev.com/c/925821 with null checks added. The
previous CL was reverted due to crashes on some machines, which
unfortunately were never diagnosed.

Bug:  752344 ,792289, 814641 
Change-Id: I52eadcdbc2c1f3dcc76bddf3b97e903143461757
Reviewed-on: https://chromium-review.googlesource.com/1014666
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Will Harris <wfh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#552672}
[modify] https://crrev.com/a695cf342559b85f2871a5851172fae9d0af3ae3/sandbox/win/src/sandbox_nt_util.cc
[modify] https://crrev.com/a695cf342559b85f2871a5851172fae9d0af3ae3/sandbox/win/src/sandbox_nt_util.h
[modify] https://crrev.com/a695cf342559b85f2871a5851172fae9d0af3ae3/sandbox/win/src/target_interceptions.cc

Status: Fixed (was: Started)
Looks like the reland stuck. Closing as fixed.

Sign in to add a comment