Do we have metrics for idle socket use? I'm wondering if turning off the predictor in an experiment can show how much of these sockets are the predictors fault.

We close idle sockets that were never used in 10 seconds (Well...when a socket is requested, and they've been unused for 10 seconds), so I don't think the predictor is the main culprit here.  With multiple navigations, we basically only have the predictor's never used sockets from the last navigation hanging around, at worst, assuming the navigations are >= 10 seconds apart, and all make at least one network request (i.e., aren't entirely served from the cache).

Not that it's not worth thinking about predictor sockets, but I don't think that's the first place to focus.

Ah right, if the sockets are used then it isn't really the predictor's fault :)

Could we get "idle time before reconnect" metric? I wonder if we can kill the socket on a 95 percentile of use or something (could even make this dynamic... might be overkill)

That sounds like a good place to start.

We could also consider adding some sort of metrics based on number of idle sockets in the pool.  If I decide to never have more than 20 (30?) idle sockets in the socket pool, and just close the oldest socket when that happens (Or least frequently used?  Or most recent?), how much does that cost us?  HTTP2 does confuse that metric a bit, though at least we only keep around one connection per HTTP2 server.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a9d6c7d3dc9a4a700ab9bf0aa71581540e014cd3

commit a9d6c7d3dc9a4a700ab9bf0aa71581540e014cd3
Author: mmenke <mmenke@chromium.org>
Date: Wed Jun 22 18:48:31 2016

net: Add some metrics on reuse of idle sockets.

We're investigating keeping fewer of them around, and/or timing them out
more aggressively to decrease memory usage.

BUG= 621197 

Review-Url: https://codereview.chromium.org/2086903003
Cr-Commit-Position: refs/heads/master@{#401370}

[modify] https://crrev.com/a9d6c7d3dc9a4a700ab9bf0aa71581540e014cd3/net/socket/client_socket_pool_base.cc
[modify] https://crrev.com/a9d6c7d3dc9a4a700ab9bf0aa71581540e014cd3/tools/metrics/histograms/histograms.xml

https://uma.googleplex.com/p/chrome/histograms?endDate=20161128&dayCount=1&histograms=Net.Socket.IdleSocketFate%2CNet.Socket.IdleSocketReuseTime&fixupData=true&showMax=true&filters=channel%2Ceq%2C1%2Cplatform%2Ceq%2CA%2Cisofficial%2Ceq%2CTrue&implicitFilters=isofficial

We recently also added a Net.Socket.IdleSocketFate histogram. From the data in Canary, idle socket reuse rate is pretty high. Majority of socket reuse happen within 1 second. There is a small difference between Android and Windows. On Android, more sockets are cleaned up because connection is closed or timed out. I will wait for more data to come in since the Canary population is small. Let me know if you have any interpretation from the current data.

Maria - is this something you would track?

mmenke@ and I added some histograms to investigate. Data showed that idle socket reuse rate is pretty high (~50% on Desktop and ~30 on Android). Idle sockets do not really retain much memory, after a few improvements in this area (use CRYPTO_BUFFER  Issue 671420  and lazy buffer initialization  Issue 524258 ).

mmenke@ removed his histogram in r8152dd7e39b668b61890503cfba8af0591464029 and I removed mine in ra46c80057c897f53772701bbde45b551401b691e.

Since idle sockets no longer use 28K of memory, I will mark this as WontFix. 

Worth noting that's just chrome-allocated memory.  Unclear how much memory sockets really take up behind the scenes.

Yes, it's unclear how much memory each idle socket consumes in the OS. We can only improve what we can see from the data we have.