seems to me that the "chrome task manager" show very small amount of "gpu memory" ?

maybe a 4GB wall, monitered memory with an external tools gets me:

GPU 600 MB with no 3D tabs

1 tab SD 800 MB
1 tab HD 2800 MB
1 tab HD + 1 tab SD  3038 MB
2 tab HD =>  "context lost" at 3777 MB

( using gpu-z freeware tool )

I had no problem with this on macOS (Intel), so I wonder if ANGLE is doing some extra texture allocations here.

Jamie, Geoff, can you reproduce this on Windows?

FYI, no problem on Linux (NVIDIA) either.

Tested this issue on Windows 10, Ubuntu 14.04 and Mac 10.12.2 with chrome version #55.0.2883.87 using the  test url in comment #0

Observed that in Mac and Ubuntu after changing from SD to HD, no error is displayed, where as in Windows 10 while shifting from SD to HD there is 3D/WebGL context error

Note:
------
In the same windows machine, with Firefox when navigated to test url and shifted to SD to HD, the FF browser got  crashed.

Window 10 Configuration
----------------------
Architecture - 64Bit
RAM          - 8 GB
Processor    - Intel core I7

Attaching the screen-cast for your reference.

paul@ could you please look into the attached screen-cast and let us know whether this is the behavior you are experiencing ???

Thank You...

Deleted: Issue 675642.mp4 3.5 MB

	Issue 675642.mp4 3.5 MB View Download

Paul is on chrismas leave but we did additional tests.

The "ship" takes ~2.5Go VRAM (https://sketchfab.com/models/9b159aee95e04663ad94ae4a744017b3?pixel_budget=0)
The "boar" takes ~4.5Go VRAM (https://sketchfab.com/models/1f0948c6a71c45d6af7c9c4ad00c2d9c?pixel_budget=0)

On win10, and nvidia GTX 980 (4Go VRAM).
- on chrome 55.0.2883.87
 * If I open one "ship" it's ok, one another and then WebGL crash (> 4Go)
 * If I open one "boar", WebGL crash
- on firefox
 * one "ship" is ok, one another still ok, one another again still ok, etc... gpuz memory never exceeds 3.8Go
 * one "boar" is ok, gpuz states 3.8Go
 * sometimes it crashes though... (most of the time it works fine though, I'm suspecting RAM but didn't investigate further)
- on Edge
 * same as firefox
 * one "boar", it doesn't crash but it reload the page when gpuz hit 4Go, so it reloads indefinitely...

On win10, and nvidia GTX 1070 (8Go VRAM).
- on chrome 55.0.2883.87
 * If I open one "ship" it's ok, one another and then WebGL crash (when gpuz hit 4Go limit)
 * If I open one "boar", WebGL crash (when gpuz hit 4Go limit)
- on firefox
 * always ok ("boar" and "ship")
- on edge
 * always ok ("boar" and "ship")

On win10, and nvidia GTX 660 Ti (2Go VRAM).
- on chrome 55.0.2883.87, firefox and Edge it's always OK
  gpuz never exeeds 1.5Go VRAM

In conclusion, it looks like GPU are capable of swapping between RAM and VRAM when it run out of VRAM, but it is also driven by browsers.
So far my guess would be :
- chrome has a hard limit of 4Go VRAM before webgl crashes. Also, chrome doesn't release/manual swap any gl memory (hidden tab, inactive canvas (no active drawcall), etc...)
- firefox and edge handled better the VRAM limit than chrome on the GTX 980 and GTX 1070
- Ironically, The GTX 660 Ti (only 2Go VRAM) is impressive as all our tests worked fine in all 3 browsers (and very fluid in chrome)

The ?pixel_budget=0 will simply select HD by default.
I notice that on the GTX 980 computer with the "boar", firefox always crashes in that case, so it's probably best to load the model normally and then select HD.

There shouldn't be much difference, except that with the url option and cache on,
many more stuffs will happen on the same frame instead of being done on separate frames.

After new year ping for geofflang@ and jmadill@. :)

Easy repro of the 4gb GPU barrier on chrome:

(only tested on an nvidia gtx 1070 8gb, guess any gpu with high memory is the same)

https://www.khronos.org/registry/webgl/conformance-suites/1.0.3/extra/out-of-vram.html


outputs:
----------------
limit: 8192MB
...
creating texture #236 mem = 3776MB
PASS out of memory
-----------------

So interesting things:
1- it spawn a context loss
2- it does not gives a gl Error "out of Memory" 

ps: Not taht Having a gl.OUT_OF_MEMORY allows to handle gracefully, context lost does not? shouldn't that error sent instead of context lost ? (on a android one plus one, I get also a context lost at 1808mb instead of a gl.OUT_OF_MEMORY)

> paul@ could you please look into the attached screen-cast and let us know whether this is the behavior you are experiencing ???

yes.

See latest comment for far easier bug repro.

note that the same "out of VRAM" test page on firefox doesn't loose context, but gives a gl.OUT_OF_MEMORY at 2832MB on 32bits firefox exe (expected) and doesn't fail at all on 64bits firefox exe)

I'm really thinking it should not loose context at all if you get OUT_OF_MEMORY so that webapp can handle it on their side gracefully.

The difficulty with GL_OUT_OF_MEMORY errors is that they leave the context in an indeterminate state. To make sure that the application doesn't see corrupted rendering, Chrome forcibly exits the GPU process if an out-of-memory error is generated.

The only arbitrary limits that Chrome enforces are at the security sandbox level, where (from memory) even the 64-bit browser is limited to 8 GB of RAM per sub-process. Chrome doesn't enforce any limits on VRAM consumption.

Geoff, could I ask you to please triage this this coming week? We need to know if this is reproducible with 64-bit Chrome Canary on Windows 10 with a GPU with a decent amount of VRAM. Thanks.

I tried to repro in today's 64 bit canary.

I opened as many tabs of the sketchfab model as I could before chrome got bogged down and too slow.. about 15 tabs with the GPU process using over 15GB of memory but no context loss.

Running https://www.khronos.org/registry/webgl/conformance-suites/1.0.3/extra/out-of-vram.html runs all the way to the end:
----------------
creating texture #512 mem = 8192MB
PASS reached limit
----------------

I agree with Ken that it's really hard to handle GL_OUT_OF_MEMORY gracefully.  The spec basically says that everything is undefined at that point and if we try to add artificial limits based on heuristics, it ends up penalizing some applications.

Paul, can you check if you see the same behaviour with Canary?

Still have the bug here, just checked with 58.0.3007.0 canary (64-bit)

the out of vram test gives:
"creating texture #234 mem = 3744MB
PASS out of memory

***context lost***
"

Same for the sketchfab model test. (contextlost when switch 2 tabs of the 3D model to "HD")


(ok for context lost, will have to digg automatic contextrestore/visibility more)

Can you save the about:gpu page and upload it here? If you do this after running a test that generates the OOM, it should have a log in it.

here it is, on canary just after a "out-of-vram" context lost

Deleted: gpu.htm 263 KB

gpu.htm 263 KB View Download

I did more digging and I can reproduce now.  Turns out this only reproduces when the sandbox is enabled and I didn't see it before because I disabled the sandbox to debug chrome.

Does anyone know if there is a property of the sandbox that limits allocation sizes?

Yes, the sandbox does limit the total amount of allocations per-process. The last time I encountered this a while back, the limit was 8 GB, even in 64-bit mode.

Please talk with jschuh@ and wfh@ from the security team about this limitation. Perhaps increasing it to 16 GB would be feasible.

yes, the sandbox sets a limit on memory for sandbox targets via Job object restrictions.

The code is here -> https://cs.chromium.org/chromium/src/content/common/sandbox_win.cc?type=cs&l=578

if it hits this then the Job should terminate the process, you won't get any calls in the process, but the broker does the immediate process termination here:

https://cs.chromium.org/chromium/src/sandbox/win/src/broker_services.cc?type=cs&l=244

Thanks Will, that's very helpful.

Can we please raise that limit to 16 GB? Is there any fundamental problem with doing so?

I investigated this a bit with Will today, the difficult part is telling the difference between allocations on the GPU and allocations on the CPU.  Chrome looks at the working set which appears to include allocations made by the GPU driver for the GPU process so it's very easy to hit the current 4GB limit by requesting GPU textures that have no data in them.  I think it's very reasonable to raise the limit to 8 or 12 GB on x64 builds to match the current memory sizes of top-of-the line GPUs but I want to look a little more to see if it's possible to detect the difference in GPU and CPU memory usage.

The primary reason for the limit is that our code is riddled with size_t truncation bugs. If you enable the warnings to detect those issues and still get clean compiles, then I wouldn't object on security grounds to raising the limit for the GPU process.

However, on general resource consumption grounds, 4GB is really not enough? Really?

It's pretty easy to hit the limit with a couple of tabs running WebGL applications because the GPU process combines the graphics allocations of the other processes.

On Linux it looks like this limit doesn't exist and the GPU process often goes over 4GB.  Is there extra coverage needed to have the same limits on Windows?

Nope. Linux ulimits don't support the same restriction, so we're forced to settle for a larger limit on overall address space. We cannot regress Windows to be equivalent.

Does anyone want to take a look at verifying we're secure to raise the limit on Windows?  I don't have the time to do the due diligence on this right now.

Going to mark this as a feature instead because things are currently working as intended.

4 GB is not enough. The 64-bit upgrade has been eagerly anticipated on all platforms for many reasons, including being able to allocate large WebAssembly heaps. The 8 GB limit on Linux is better, and we'd at least like the same on Windows.

Justin, Will: which compiler warning needs to be enabled where? Is it:

C4267 - Conversion from 'size_t' to 'type', possible loss of data.
C4305 - 'identifier' : truncation from 'type1' to 'type2'

Or something else? Where would this be globally enabled?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ed8a8cfa1017522995d6be2961e5dc68064e747f

commit ed8a8cfa1017522995d6be2961e5dc68064e747f
Author: Ken Russell <kbr@chromium.org>
Date: Wed May 24 23:28:17 2017

Raise memory limit in the GPU process's sandbox on Windows.

If machine has more than 8 GB physical memory, allow 8 GB in the
sandbox; similarly for 16 GB.

BUG= 675642 

Change-Id: I8820cd9ffceffb5fbd5332aff1744c2f4926c88f
Reviewed-on: https://chromium-review.googlesource.com/510902
Reviewed-by: Will Harris <wfh@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#474474}
[modify] https://crrev.com/ed8a8cfa1017522995d6be2961e5dc68064e747f/content/common/sandbox_win.cc

This is finally fixed. Sorry for the long delay. Fix will ship in Chrome 60.

Your feedback on whether this is working for you in the next Chrome Canary will be appreciated.

Works beautifully, Thanks a lot !
(tested on win10 with Version 61.0.3114.0 (Official Build) canary (64-bit))

Excellent. Thank you Paul for verifying.