washingtonpost.com uses 53Mib on browser for certificate verification |
|||||||||||||||
Issue description
OS: Linux
What steps will reproduce the problem?
1. Add a trace event in net::CertVerifyProcNSS::VerifyInternal in net/cert/cert_verify_proc_nss.cc as:
TRACE_EVENT0("net", "net::CertVerifyProcNSS::VerifyInternal");
2. Save a json file named "trace_config" with contents:
{
"startup_duration": 60,
"result_file": "/tmp/trace.json",
"trace_config": {
"record_mode": "record-continuously",
"included_categories": ["disabled-by-default-memory-infra", "net"],
"excluded_categories": ["*"],
"memory_dump_config": {
"triggers": [
{ "mode": "detailed", "periodic_interval_ms": 55000 }
]
}
}
}
3. Start chrome as:
out/Release/chrome --enable-heap-profiling --trace-config-file=path/to/trace_config
4. Open https://www.washingtonpost.com/ and scroll down. Wait for a minute.
5. Open chrome://tracing, press load, and select file "/tmp/trace.json" which is specified path in the config json.
6. Click on the red dot in the top row, and find the "Overview" section at bottom. Click on the box that is on "browser" row and "malloc" column. You will see the memory allocated in "Heap details" section.
It shows around 50Mib of memory is allocated by net::CertVerifyProcNSS::VerifyInternal.
To get a quick info, just download the trace attached in this bug and follow steps 5 and 6.
Is so much memory really required for certificate verification? Is there a cache that is left uncleared?
,
Mar 16 2016
+primiano Do you think this is an issue? +dskiba I think you were also hitting this issue of PKIXVerifyCert allocating lot of memory.
,
Mar 16 2016
Yes, PKIXVerifyCert calls CERT_PKIXVerifyCert from libnss, which allocates in several places, including sqlite (which it uses under the hood it seems). I'll try to get some stack traces.
,
Mar 16 2016
+rsleevi@ can you help triage this issue? +klobag FYI Context: ssid and dskiba are looking into reducing memory of the browser process, as part of the wider go/trim effort. There seem to be a substantial amount of memory related with certificate verification (see #1) which sticks even after closing the tab. None of us have enough context / knowledge about the network stack, we would really appreciate if somebody some help here, as there seem to be good margin for improvement.
,
Mar 16 2016
(klobag: Since you were CC'd, note that if it's coming from CertVerifyProcNSS, it cannot affect Android. This code does not run in Android.)
,
Mar 16 2016
,
Mar 16 2016
#6 good news for Android. I think that ssid/dskiba are currently looking on Linux Desktop (it's easier to debug).
,
Mar 16 2016
Thanks David. I know this is not for Android. ssid and dskiba are tasked to improve browser process memory profiling and find ways to reduce browser process memory usage. Before we get memory profiler working on Android, they are using Linux as test bed.
,
Mar 16 2016
There are no plans to invest in reducing the NSS memory in the next three quarters. We are working to move away from NSS, where possible. I totally appreciate the concerns with product excellence, but we're better off moving away from it then investing serious engineering effort in trying to change this, given the broader economies. Because of this, I believe WontFix is the appropriate remark.
,
Mar 16 2016
Further comments: 1) NSS uses its own memory allocation strategy with arenas. As such, any form of memory management will misreport, because those arenas are used as free lists. NSS_DISABLE_ARENA_FREE_LIST=1 as an environment variable can more accurately report usage. 2) NSS uses extensive caches internally. These caches live for the duration of the process. There are no plans in NSS to remove these caches. This is one of the reasons we're moving away from NSS.
,
Mar 16 2016
So here is a trace with PKIXVerifyCert stack traces. Open, click on "malloc" in the browser row, go to "Stack frame" bottom pane, expand "net | cert".
Here is the heaviest trace:
▶ƒ net::(anonymous namespace)::PKIXVerifyCert() 78.1 MiB
▶ƒ CERT_PKIXVerifyCert 78.1 MiB
▶ƒ <unknown> 78.1 MiB
▶ƒ <unknown> 77.8 MiB
▶ƒ <unknown> 77.8 MiB
▶ƒ <unknown> 77.8 MiB
▶ƒ <unknown> 77.8 MiB
▶ƒ <unknown> 77.8 MiB
▶ƒ <unknown> 77.8 MiB
▶ƒ SECITEM_CopyItem_Util 11.4 MiB
▶ƒ PORT_ArenaAlloc_Util 11.4 MiB
ƒ PL_ArenaAllocate 11.4 MiB
▶ƒ PORT_ArenaZAlloc_Util 66.3 MiB
▶ƒ PORT_ArenaAlloc_Util 66.3 MiB
ƒ PL_ArenaAllocate 66.3 MiB
I'm fine with marking this as "WontFix" though.
,
Mar 16 2016
re: comment #12 - Did you re-run that with the environment variable set? Having large arena allocations is not surprising, since they will all hang on via free-lists without that variable set.
,
Mar 16 2016
That was just a plain run. Defining NSS_DISABLE_ARENA_FREE_LIST=1 changed things:
1. Total dropped from 96.7 to 27.0 MiB
2. Heaviest trace dropped from 78.1 to 22.9 MiB
3.CERT_PKIXVerifyCert is not the heaviest trace anymore, it allocates just 1 MiB
4. The heaviest trace now is:
▶ƒ <unknown> 22.9 MiB
▶ƒ <unknown> 22.9 MiB
▶ƒ <unknown> 22.9 MiB
▶ƒ <unknown> 22.9 MiB
▶ƒ <unknown> 22.9 MiB
▶ƒ <unknown> 22.9 MiB
▶ƒ <unknown> 22.9 MiB
▶ƒ <unknown> 22.9 MiB
▶ƒ <unknown> 22.9 MiB
▶ƒ SECITEM_CopyItem_Util 3.2 MiB
▶ƒ PORT_ArenaAlloc_Util 3.2 MiB
ƒ PL_ArenaAllocate 3.2 MiB
▶ƒ PORT_ArenaZAlloc_Util 19.7 MiB
▶ƒ PORT_ArenaAlloc_Util 19.7 MiB
ƒ PL_ArenaAllocate 19.7 MiB
<unknown> means that it's some system library without symbols.
,
Mar 16 2016
If you're running on a Corp machine, I can think of a variety of Google-specific libraries that may be the cause for that. Running tests of NSS/NSPR on Corp-enrolled machines = recipe for pain. But it sounds like setting that environment variable was sufficient to address what this bug was originally filed about, and as noted, it doesn't affect Android, so are you comfortable with WontFix on that basis?
,
Mar 16 2016
@primiano, is it possible to test in any non-copy machines? Is NSS_DISABLE_ARENA_FREE_LIST=1 set by default on release builds? Does it have performance impact? Even with the flag, I am seeing 25 Mib of memory allocated at PKIXVerifyCert. I think cbentzel@ was removed by mistake.
,
Mar 16 2016
No, NSS_DISABLE_ARENA_FREE_LIST=1 is not by default. That's an environment variable. The NSS development team (it is a third-party library) feel strongly about the value of the arena free lists. There's renewed debate coming from the two largest consumers (Firefox and Chrome), but the distro maintainers aren't terribly thrilled with removing the freelist. In short, the remarks are about a third-party library. Chrome is transitioning away from that third-party library, but if you're wanting to optimize that third-party library, that's best done in that third-party library.
,
Mar 17 2016
> NSS uses its own memory allocation strategy with arenas. As such, any form of memory management will misreport, because those arenas are used as free lists Well not sure I'd call it misreport. Unless I am missing something, my understanding is that NSS is malloc-ing a large block to do it's own memory internal management within that. If that is the case, regardless on how smart NSS is, those are actually X MB stolen from the system, so not sure if this is really a case of misreporting. This is simply counting memory malloc-ed and not freed. (Unless NSS bothers doing madvise-s to free up the actual unused pages) Anyways, my understanding is that this entire issue is Linux desktop only, and even there there is a plan to move away from NSS. As such I think the most practical thing to do here is to move on and archive this issue. It still sounds sad to me that NSS uses all this memory, but form a realistic viewpoint, given the limited amount of people dealing with chrome-wide analysis, those energies might be better spent looking for more actionable places which affect more platforms, rather than trying to improve a soon-to-be-dead horse.
,
Jun 20 2017
No activity in > 1 year. File again if you still think it's relevant. |
|||||||||||||||
►
Sign in to add a comment |
|||||||||||||||
Comment 1 by ssid@chromium.org
, Mar 16 2016