New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Linux sandbox memory limits are overly restrictive

Project Member Reported by jgruber@chromium.org, Jan 9 2018

Issue description

See also: https://crbug.com/752185 (Google-only).

The Linux sandbox restricts allocations [0]; on 64-bit builds, address_space_limit to 16GB and new_data_segment_max_size to 8GB.

These limits can easily be reached when web workers are involved: each worker creates a new Isolate, which reserves 512M of memory for code space. Sometime before creating the 32nd isolate / web worker, we hit the limit and crash with OOM.

Real web apps have been running into this and crash reports are starting to trickle in ( https://crbug.com/792711 ,  https://crbug.com/798404 ).

[0] https://chromium-review.googlesource.com/c/chromium/src/+/641992/3/content/common/sandbox_linux/sandbox_linux.cc
 
mlippautz@, you had some thoughts on possible solutions?
Cc: palmer@chromium.org
FYI: palmer@

The only possible solution for ToT is to avoid enforcing virtual address space limits. AFAIK we only do that on Linux, which (personally) seems overly restrictive. 

The last time we evaluated that limit we only had other components (e.g. wasm) reserving address space in mind, so keeping the limit seemed fine. With the web worker use case in the mix this might change, or not?

In future, V8 might impose a limit on JITed code which would allow us to reserve less space for the CodeRange (512M contiguous reservation needed for relative calls) which would also help in this case.

Comment 3 by adamk@chromium.org, Jan 10 2018

Blocking: 800818

Comment 4 by adamk@chromium.org, Feb 12 2018

Blocking: -800818

Comment 5 by adamk@chromium.org, Feb 12 2018

Blocking: 800818
Issue 800818 has been merged into this issue.

Comment 6 by wfh@chromium.org, Feb 12 2018

Why not just limit the number of webworkers to something like, 20. I think 20 is what firefox defaults to...?

Comment 7 by adamk@chromium.org, Mar 6 2018

Issue 817441 has been merged into this issue.

Comment 8 by adamk@chromium.org, Mar 6 2018

Cc: ojan@chromium.org
Components: Blink>Workers
This would be the same issue with  issue 782982 .
Cc: zakerinasab@chromium.org jbroman@chromium.org
 Issue 819297  has been merged into this issue.
Blocking: 821348
Cc: ahaas@chromium.org titzer@chromium.org
+ahaas, titzer as this also seems to be blocking successfully running the wasm spec tests on Linux.
Agree with c#2. AFAIU Linux is the only platform on which we impose address space restrictions. Given how little of the userbase this impacts, do we still believe this is an effective security measure?
Cc: -u...@chromium.org jsc...@chromium.org
+jschuh

Ping palmer@
Cc: jorgelo@chromium.org penny...@chromium.org tsepez@chromium.org rsesek@chromium.org wfh@chromium.org
Labels: OS-Android OS-Chrome OS-Fuchsia OS-Mac OS-Windows
Owner: palmer@chromium.org
Status: Assigned (was: Available)
I had expected this to be the case on all platforms, not just Linux. That might be a regression. +all the various OS experts from Platform Security: what do you think?

The address_space_limit is 4 GiB, not 16 (the code lives in services/service_manager/sandbox/linux/sandbox_linux.cc now):

```
  // Limit the address space to 4GB.
  // This is in the hope of making some kernel exploits more complex and less
  // reliable. It also limits sprays a little on 64 bits.
  rlim_t address_space_limit = std::numeric_limits<uint32_t>::max();
  rlim_t address_space_limit_max = std::numeric_limits<uint32_t>::max();
```

The comment explains why we do this, and as far as I know the reasoning and context behind haven't changed. (+jorgelo and tsepez)

Might Site Isolation help here, because fewer origins will be creating workers/isolates in the same renderer process?

Imposing a limit on JITted code would also be an excellent idea (and/or moving compilation into a separate process, but that's obviously a longer-term thing). 512 MiB "seems like a lot" (but maybe it's not).

Anyway, we (Platform Security) will have a think. Assigning to myself to make sure it doesn't get lost.
So, that's not actually right:

if (sizeof(rlim_t) == 8) {
    // On 64 bits, V8 and possibly others will reserve massive memory ranges and
    // rely on on-demand paging for allocation.  Unfortunately, even
    // MADV_DONTNEED ranges count towards RLIMIT_AS so this is not an option.
    // See  crbug.com/169327  for a discussion.
    // On the GPU process, irrespective of V8, we can exhaust a 4GB address
    // space under normal usage, see  crbug.com/271119 .
    // For now, increase limit to 16GB for renderer, worker, and GPU processes
    // to accomodate.
    ...
      address_space_limit = 1ULL << 34;

I still agree we need to harmonize the settings across platforms.
On Mac we don't have the ability to limit the size of the address space via rlimit (issue 435269).

Comment 18 by wfh@chromium.org, Mar 16 2018

Comment 19 by neis@chromium.org, Apr 16 2018

Cc: petermarshall@chromium.org u...@chromium.org kozy@chromium.org
 Issue v8:7635  has been merged into this issue.
Cc: manoranj...@chromium.org abdulsyed@chromium.org ligim...@chromium.org ajha@chromium.org krajshree@chromium.org brajkumar@chromium.org
 Issue 827627  has been merged into this issue.
Project Member

Comment 21 by bugdroid1@chromium.org, May 4 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/6b1457cfd6baa332ffa968b74fff705bac72e762

commit 6b1457cfd6baa332ffa968b74fff705bac72e762
Author: Hannes Payer <hpayer@chromium.org>
Date: Fri May 04 13:47:09 2018

Reduce maximal code range size to 128M on Linux, Windows, and OSX for x64 and ia32.

Bug:  chromium:800348 ,  chromium:827627 ,  chromium:839750 
Change-Id: I112e20b83eb1937476ebb4f30cf5679113759c0c
Reviewed-on: https://chromium-review.googlesource.com/1044195
Reviewed-by: Michael Starzinger <mstarzinger@chromium.org>
Commit-Queue: Hannes Payer <hpayer@chromium.org>
Cr-Commit-Position: refs/heads/master@{#52996}
[modify] https://crrev.com/6b1457cfd6baa332ffa968b74fff705bac72e762/src/globals.h

Comment 22 by ojan@chromium.org, May 8 2018

Cc: -ojan@chromium.org
Blocking: v8:7703
Are there any plans for relaxing this restriction? It looks like current limits block pointer compression in V8.
Cc: ishell@chromium.org

Comment 26 by wfh@chromium.org, May 18 2018

re: #24 which specific restriction(s) are you talking about, and on which platforms?

My understanding was that the intent behind pointer compression was to *reduce* memory usage (the design doc [1] says "35% of V8 heap reduction") so I am slightly confused why the memory limits would have to be increased?

[1] https://docs.google.com/document/d/1TJTIYHNyG8KCCLQTu46VW3enRNtnC425eEobUp0eUQA/edit
I'm sorry, I was not precise. IIUC this issue is about the fact that the sandbox may restrict the amount of reserved (and not yet committed) virtual space.
Pointer compression aims at reducing the amount of committed memory and it requires the ability to reserve a contiguous 4Gbyte region of address space for each V8 isolate. Chrome creates one V8 isolate for the main thread and one V8 isolate per Web Worker instance. Not to mention Wasm which in addition reserves 8Gb of virtual address space for Wasm memory.

#16 mentions 16Gb limit which doesn't seem enough for this pointer compression idea to fly.
To what extent does the fix in #21 relieve the pressure?

Given #27, it seems like essentially no limit would fly — 4 GiB isolates; and although currently there is no bound on the number of isolates, even 20 (as proposed in #6) would mean we'd need an 80 GiB limit at least. And we very much want the memory savings from pointer compression, especially since we're shipping and relying so much on Site Isolation (-> more memory usage) now.

The code comment

  // This is in the hope of making some kernel exploits more complex and less
  // reliable. It also limits sprays a little on 64 bits.

doesn't go far enough in my mind to justify the limit — we probably should have written real documentation for it when the limit was created. I don't know what particular kernel exploits that would protect against; and does it still matter in a post-KPTI world? And speaking of post-Spectre: heap spraying is a way to work around ASLR, but in a post-Spectre world (https://chromium.googlesource.com/chromium/src/+/master/docs/security/side-channel-threat-model.md) we're assuming an attacker has a sufficient infoleak to beat ASLR (in-process) anyway, even without any actual infoleak bugs. (And of course there are infoleak bugs.)
The limit dates back to https://chromium.googlesource.com/chromium/src/+/a540e60eaaa7866abf0901b873ec6814d9cc81a9. Based on the referenced bugs ( issue 169327  and  issue 157177 ), this may have been done for two reasons:

1) "a good defence to a know WebKit issue with ref count overflowing"
2) "mitigate 32 bits sizes on 64 bits architecture types of bug, we should prevent contiguous mappings of more than 4GB"

Given that Linux is now 64-bit only, those reasons don't seem very relevant. Maybe jln@ can chime in?
Project Member

Comment 30 by bugdroid1@chromium.org, Jun 1 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/683115b5f36ba1b80a1fbcd589a6b2a3ee075a88

commit 683115b5f36ba1b80a1fbcd589a6b2a3ee075a88
Author: Chris Palmer <palmer@chromium.org>
Date: Fri Jun 01 16:50:31 2018

Document the sandbox memory limits marginally more.

Future maintainers will need to know why and to what extent we think this stops
exploits, and where the numbers came from.

Bug:  800348 
Change-Id: I8da01a89f290304aa99117676e1d9eb599a2f570
Reviewed-on: https://chromium-review.googlesource.com/1081391
Reviewed-by: Robert Sesek <rsesek@chromium.org>
Commit-Queue: Chris Palmer <palmer@chromium.org>
Cr-Commit-Position: refs/heads/master@{#563678}
[modify] https://crrev.com/683115b5f36ba1b80a1fbcd589a6b2a3ee075a88/services/service_manager/sandbox/linux/sandbox_linux.cc

Per rsesek, macOS honors RLIMIT_DATA, so we can and (I believe) should harmonize the policy across Linux and macOS in that way. We should define the limit in a single place so that if it changes (up or down), the platforms stay harmonized.
Status: Started (was: Assigned)
Issue 851626 has been merged into this issue.
One reason for keeping a virtual address space limit is that it guards against exploiting the rowhammer bug via spraying page table entries (as described in https://googleprojectzero.blogspot.com/2015/03/exploiting-dram-rowhammer-bug-to-gain.html).
Project Member

Comment 35 by bugdroid1@chromium.org, Jun 18 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b558ba514e82ec7624ab29733ca11cbc208b3d00

commit b558ba514e82ec7624ab29733ca11cbc208b3d00
Author: Chris Palmer <palmer@chromium.org>
Date: Mon Jun 18 18:10:40 2018

[sandbox] Remove address space limits on Linux.

Unfortunately, it does not appear possible to set RLIMIT_AS such that it will
both (a) be high enough to support V8's and WebAssembly's address space
requirements while also (b) being low enough to mitigate exploits using integer
overflows that require large allocations, heap spray, or other memory-hungry
attack modes.

Bug:  800348 
TBR: jln
Change-Id: I4ec234709a4e49abde1c22f9c63a87bac107b8a0
Reviewed-on: https://chromium-review.googlesource.com/1097553
Commit-Queue: Chris Palmer <palmer@chromium.org>
Reviewed-by: Robert Sesek <rsesek@chromium.org>
Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#568073}
[modify] https://crrev.com/b558ba514e82ec7624ab29733ca11cbc208b3d00/components/nacl/loader/sandbox_linux/nacl_sandbox_linux.cc
[modify] https://crrev.com/b558ba514e82ec7624ab29733ca11cbc208b3d00/sandbox/linux/services/resource_limits.cc
[modify] https://crrev.com/b558ba514e82ec7624ab29733ca11cbc208b3d00/sandbox/linux/services/resource_limits.h
[modify] https://crrev.com/b558ba514e82ec7624ab29733ca11cbc208b3d00/sandbox/linux/services/resource_limits_unittests.cc
[modify] https://crrev.com/b558ba514e82ec7624ab29733ca11cbc208b3d00/services/service_manager/sandbox/linux/sandbox_linux.cc
[modify] https://crrev.com/b558ba514e82ec7624ab29733ca11cbc208b3d00/services/service_manager/sandbox/linux/sandbox_linux.h

Status: Fixed (was: Started)
#34: Noted, thank you. Fundamentally, I don't think it's reasonable for probabilistic mitigations for probabilistic attacks to block legitimate application functionality, as was the case here. Especially when the root cause of the problem is several layers lower in the stack.

Additionally, the limit before the CL landed in #35 was already higher than it sounds like the precondition was (reading the section "Kernel privilege escalation"), and Rowhammer still has other preconditions for success. I realize that Rowhammer research has advanced since that post, and there may now be fewer and/or less difficult preconditions for attack, but as far as I understand Rowhammer still probabilistic, highly variable, and ultimately fixed by ECC RAM.

FWIW, we do intend to continue not using huge pages (https://bugs.chromium.org/p/chromium/issues/detail?id=806725).
Cc: rfbpb@google.com haraken@chromium.org bashi@chromium.org adamk@chromium.org hablich@chromium.org neis@chromium.org keishi@chromium.org verwa...@chromium.org thomasanderson@chromium.org
 Issue 782982  has been merged into this issue.

Sign in to add a comment