Created this after a long rant with David. I don't have a clear plan on the right approach, but thoughts we had

Problem: Circular dependency
- It seems like HPKP logic (as well as CT requirement, such as Symantec/CNNIC) may belong moreso in the CertVerifier, but that's historically been a complicated relationship (various non-Chrome-but-Chromium things creating their own but wanting policies either same or different, the layering for how integration with the OS was done, the issues with how our caching layer handled)
- Alternatively, splitting out the HPKP/RequireCT stuff into a dedicated class either for //net/cert or //net/socket, which simply offers querying mechanisms for judgements

2) Scope/Mission creep
- HSTS/ExpectCT/ExpectStapling are all 'opportunistic' modes based on headers. Should these live in the //net/http side to evaluate after an HttpStreamSocket, while the HPKP/RequireCT/RequireStapling live in the //net/socket side (since they affect socket pooling logic?)

3) Persistence
- With the greater use of Cronet, and the desire to support Android platform-level behaviours there, it's likely becoming more and more necessary to decouple the notion of 'static' (baked in) and 'dynamic' (observed by the website), because we've clearly got a third-category - those set by the developer. Plus, yanno, supporting enterprise cases (such as exemptions from RequireCT)

4) Propagation
- SpdySession::CanPool may just need to bite the bullet and go async, especially with the Cronet/Android changes. If we centralized all of the Require-logic into CertVerifier, than CanPool could just async CertVerifier for the proposed hostname. This gives the most consistent results.

These are just dumping various ideas being discussed, but they really need a meeting/gvc to flesh out into something actionable. But I don't want to lose state about some of the weirdness when working with it.

I've also been thinking about how to handle the scope creep of the transport_security_state_static.{json, h} files.

For  https://crbug.com/595493 , I need to start generating transport_security_state_static.h dynamically. Since I'll be rewriting the script that converts the JSON into the .h, I have been thinking of proposing a format change to the JSON format, with more straightforward and extensible entries:

"entries": [
  ...,
  {
    "domain": "garron.net",
    "hsts": {"level": "enforce", "include_sub_domains": false},
    "hpkp": {"level": "report", "include_sub_domains": true, "pins": "lgarron", "report_uri": "https://report.garron.net"}
    // "ct" and "staple" not set, defaulting to off.
  },
  ...
]

I was also thinking of possibly moving the transport_security_state* files into their own folder, especially since the HSTS list may need to become a component in the medium-term.

JSON Cleanup
- That's been on the list for 4 years. I'm glad you're thinking about reforming it. Changes to the format are explicitly tricky because of the coupling to users' persisted state. Every attempt at changing the underlying format has resulted in months of review. Please don't underestimate the complexity or the possible pushback. Considering past efforts introduced a number of security bugs (either during development, or during release), please cc me on reviews if you do start doing that.

Moving it out to a folder
- As a net/ OWNER, I'm pretty opposed to that. Also, moving out the list to a component (presumably, component updater, not //component) is fine, but that's something almost orthogonal.
- FWIW, I'm also opposed to moving it out to a component. I think that's bad for users in both the short and long term.

I'm not sure, even if we do end up supporting the weird platform features, that CanPool needs to go async. Depends exactly how/whether we end up implementing them. That state machine is hairy enough as it is (and keeps getting worse what with all this Alt-Svc craziness), so if we at all can, I'd like to keep it synchronous.

But yeah, my strawman proposal for the net/{http,socket} circular dependency is we hide everything into the CertVerifier. As far as SSLClientSocket and friends are concerned, they need a yes/no answer (plus some extra diagnostic output) from the CertVerifier and probably don't need to know that some are TSS and some are other things.

I was also thinking CertVerifier should probably gain a synchronous IsResultAlsoValidForHostname(...) function which replaces the cert/name matching portion of the CanPool check. This way SPDY needn't make assumptions about CertVerifier's behavior. (Right now it assumes the name checks it makes are consistent with doing a new verification from scratch. But only the CertVerifier implementation could know that.)

But, yeah, we should probably GVC since I'm sure my 8pm first-thought-that-came-to-mind ideas will have holes in it somewhere. :-)

(IsResultAlsoValidForHostname obviously needs a better name. But the semantics are "I want to use this certificate for a different hostname. I've still got the CertVerifyResult handy and don't want to do another expensive path-building step. Is this still cool?" Basically what CanPool does.)

HSTS entries "set by the developer" seem a lot like "baked in" entries to me.  I wonder if the baked in entries should really even be baked in at the net/ layer.

Also worth noting that the baked-in pinned certs are deliberately not used by Cronet, making the decision to bake them into net/ even more questionable (Particularly since it's my understanding they're still actually built into Cronet, bloating up binary size).

Most of the binary size comes from the HSTS preloads, not the HPKP preloads, and those are enabled by Cronet and elsewhere. The baked-in pins are loose change in comparison. But it's true that shipping them is a waste. For historical reasons, the HSTS and HPKP preloads are very tied together, which is why they're shipped as a unit. (But they don't have to be.)

Hrm..yea, you're right.  For some reason, was thinking they took up much more space.  Looks like they're probably only a couple k.

@mmenke: The current interface assumes 'static' is truly static (e.g. you can mint a new TransportSecurityState and all static entries are there, but no dynamic entries). This doesn't really match with dev-set expectations. It also plays with how we persist things, which is... weird.

I'm not sure why you suggest not baking into the //net layer - I think the baked in sets we have (HSTS, HPKP, or even CT logs) - are something we want *every* //net consumer to be using (whether they be Remoting, Cronet, Blimp, etc). So I'm not terribly keen on extracting that from //net, other than some of the HPKP set - and again, that's only because Cronet's a special snowflake.

There's also the whole thing about carveouts and enterprises.

I'm dubious about having a magic opt-in list of HSTS domains as part of the network stack.  Baked in lists don't scale, and a baked in HSTS list doesn't really seem to be a basic part of HTTP.

Ah, at least it's clear where we disagree then :)

The security policies belong as close to the //net stack as possible, in my opinion, because that's the only way we ensure that the various embedders of the //net stack get the security right. Much in the same way I wouldn't be supportive of lifting the platform-specific cert verification logic out, the logic for "All connections to this domain must use TLS" or "All connections to this domain must use this set of certs" is a pretty core security feature, independent of the protocol (as both specs mention).

And they are, of course, both IETF specs with defined headers and behaviours, coupled to HTTP. They definitely meet the 'standardization' bar (e.g. they're not part of "The Web Platform", they're part of "The Core Protocols"). But we can always hash that out more offline, but at least it's clear where our philosophical disagreement is.

Just to clarify where we disagree a bit more:  HSTS itself certainly meets the standardization bar, and I'm certainly all for that.  A hard-coded HSTS list, on the other hand...

We'll follow-up offline, but I feel pretty strongly that those core security policies - including the hardcoded list - belong in //net. Just like the public suffix list does.