This was originally reported on uBlock Origin issue tracker:
https://github.com/gorhill/uBlock/issues/3219

While waiting for the test team to kick in, I did some digging.

Bisect info for "prefetch" jsfiddle: 381040 (good) - 381050 (bad)
https://chromium.googlesource.com/chromium/src/+log/005d3f26..3b10a107?pretty=fuller
Suspecting r381046 = 3b2ebae80f0d0985148ae0478ee5dde185019669 = https://crrev.com/1777833003 by pasko@chromium.org
"Unbundle link rel=prefetch from privacy settings"
Landed in 51.0.2679.0
Merged to 50.0.2661.52

Seems intended according to the commit message:
	This change allows <link rel=prefetch> to perform prefetches regardless of the
	setting and network type.

	Reason: web developers should be able prefetch with all possible privacy
	settings and with all possible network types. This would avoid web devs coming
	up with creative ways to prefetch in cases they are not allowed to do so.

	Assumption: all requests arriving through ChromeResourceDispatcherHostDelegate
	are initiated by the renderer, and the only way to issue a prefetch from there
	is via <link rel=prefetch>.

I didn't bisect "preconnect" and "dns-prefetch" jsfiddle as I supposed the above one is sufficient to assign the developer who can handle this bug report.

#2
Thanks for searching for the related change. It seems this was discussed in https://bugs.chromium.org/p/chromium/issues/detail?id=589369

If I understand that correctly, the change involved two things,
1) Switching from "tri-state (always / only on wi-fi / never) to a boolean (on / off)" for the prediction setting
2) "Unbundle link rel=prefetch from privacy settings"

The default setting on mobile devices was apparently "only on wi-fi", which changed to "on" due to this change. This is definitely understandable to speed up mobile experience.

Unbundling prefetch from this privacy setting on the other hand seems a bit too much for me. One reason mentioned was that developers will find a way around using iframes anyway, but when using a blocking solution like uBlock Origin, users can easily block iframes (https://github.com/gorhill/uBlock/wiki/Dynamic-filtering:-Benefits-of-blocking-3rd-party-iframe-tags). The prefetching on the other hand is not blockable at all by an extension, which from my perspective is a serious privacy issue.

> web developers should be able prefetch with all possible privacy settings and with all possible network types.

This is so wrong, it baffles me the argument is being made at all to rationalize working against the user, let alone against user's privacy settings.

+net owners to get this triaged

Thanks for the report, and subsequent analysis!

Closing this issue as "working as intended".

If you have further concerns, please raise them in  Issue 589369 .

The setting disables Chrome's logic for predicting user actions, not predictive directives from websites.

If you visit http://foo.com, for instance, we may notice that from your previous navigations to http://foo.com, you needed connections to images.foo.com, foo.bar.com, and kung.foo.com.  So depending on our confidence and the number of connections you previously needed, we may preemptively open one or more connections to those domains, or do a predictive DNS lookup while we're still requesting the main http://foo.com document.

Separately, it sounds like filing a feature request regarding Comment #3 for extensions' ability to control that may be useful.

There are several web platform features that extensions are not (yet) able to control, typically because it just hasn't been requested yet. Filing/starring such bugs helps the team prioritize such requests. This includes HTTP/2 Push (Issue 773392) and the <link rel=> types you identified.

Note that the <link rel=> behaviours you describe are part of HTML spec itself ( https://html.spec.whatwg.org/multipage/links.html#linkTypes ), and that is consistently implemented across user agents. As noted in Comment #8, the "prediction service" refers to learning model based on past activity, rather than the link-rel types, which are explicit notes from the Web author themselves.

> The setting disables Chrome's logic for predicting user actions, not predictive directives from websites.

The setting is labeled: "Use a prediction service to load pages more quickly".

Why does the setting "Use a prediction service to load pages more quickly" exist at all as a Privacy setting if it can easily be worked around at the whim of web sites? There is no sense for this setting to exist as a Privacy setting if its state is completely dismissed by the browser just because a remote server dictated it.

Source: https://w3c.github.io/resource-hints/

> The user agent SHOULD attempt to initiate a preconnect and perform the full connection handshake (DNS+TCP for HTTP, and DNS+TCP+TLS for HTTPS origins) whenever possible, but is allowed to elect to perform a partial handshake (DNS only for HTTP, and DNS or DNS+TCP for HTTPS origins), or skip it entirely, due to resource constraints or other reasons.

Not playing on words and respecting the obvious wish of users who disable a setting described as "Use a prediction service to load pages more quickly" is quite a good reason to "skip it entirely" -- it's just common sense aside being ethical -- surely the user expressed a wish to opt out of connecting to remote servers out without his express approval. Browsers are USER agent, they should never prioritize what remote web sites want over what user wants.

The whole time I thought I was helping users by disabling a feature I considered hostile to those who willingly and explicitly install a content blocker -- just to find out now that the whole time (since Chromium 51 apparently) we were all misled and were not really protected from the negative side effects of prefetching to privacy.

Sites aren't working around that setting - that setting only affects Chrome's internal prediction service, which is completely tangential to web platform APIs, so there's nothing for them to work around, with respect to that setting.

I'm not sure what content blockers you're referring to, but HTTP prefetch requests are visible to extensions, just like normal requests, and can be blocked.  DNS lookups and preconnects cannot currently be blocked by an extension without a content script, to the extent of my knowledge.  As rsleevi says, an extension API to prevent that would be an entirely reasonable feature request.  There's currently no way for users to indicate they aren't interested in preconnects/DNS prefetches.

> HTTP prefetch requests are visible to extensions, just like normal requests, and can be blocked.  DNS lookups and preconnects cannot currently be blocked

Right, I put prefetch/preconnect in the same category, my mistake. My understanding however was really that the setting disabled preconnection as well, and I am pretty sure it used to be like this in the past -- so this was my expectation and the expectation I have been sharing with users of my extension. Again I want to reiterate the setting is presented as a *privacy* setting.

What information is available to a remote web site with a preconnection? There is obviously the IP address. What else?

I do want to stress that the setting only controls the Chrome-specific behaviour, and is not intended to control the "Web Platform" features exposed.

It would be better filing a feature request for an extension API to evaluate that. It seems a perfectly reasonable request, although with the caveat that it is up to the extension team to evaluate and requestors to help explain what they would use it for.

This setting is documented not to control webpage prefetching, as described at https://www.google.com/chrome/browser/privacy/whitepaper.html#netpredict

I believe that's basically it.  We don't do any authentication.  If you're using an SSL proxy talking to an SSL server, we may send HTTP auth credentials to the proxy if you've previously sent them to the proxy since you last started Chrome, but we won't send auth credentials to the destination server, even if you have them cached.

We do the standard SSL negotiation for HTTPS, so they could recognize that we support HTTP2 and they know what cypher suites we advertise, so they could probably figure out you're using Chrome, and possibly what version you're using.

We also don't do HSTS at that layer (Each HSTS check could give someone one bit of information, in theory).

Of course we don't send cookies, since we don't send any HTTP request.

The site requesting the preconnects could use magic IP addresses so that what IP addresses we preconnect to contains some information that the source site had (We don't even provide the domain name, except to proxies, though I'm not sure about the SSL case).

> Of course we don't send cookies, since we don't send any HTTP request.

Thanks. That's reassuring, my flawed understanding led me to expect the worst.

Note that with Firefox's implementation of browser.privacy.networkPredictionEnabled, disabling the setting also disable preconnection and DNS lookup:

> If true, the browser attempts to speed up web browsing by pre-resolving DNS entries, prerendering sites (using, for example, <link rel='prefetch' ...>), and preemptively opening TCP and SSL connections to servers.

See <https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/privacy/network>.

Thanks for the explanations and sorry that I wasn't able to find  Issue 589369  before opening this ticket.

What personally concerns me the most is that the behaviour for an existing setting changed (with version 51 apparently) while the description almost not changed. I'd suggest to mention prefetch in there and/or maybe link to the whitepaper explanation. Also the underlying chrome.privacy.network.networkPredictionEnabled is still described in the extensions documentation to affect prefetch (https://developer.chrome.com/extensions/privacy - same description as quoted in Comment #15). So I think that should be changed to reflect what's actually happening. Should I open an Issue for that or can it be handled in the scope of this Issue?

Just to clarify regarding HTTP requests and cookies: in case of a prefetch a HTTP request is made and cookies are transmitted, like it is documented in the privacy whitepaper. Thanks for clarifying in Comment #11 that prefetch is visible for and blockable by extensions. This doesn't apply to preconnect/dns-prefetch though as mentioned in Comment #14.

To prevent preconnect/dns-prefetch from opening connections I consider opening an Issue suggesting to introduce another chrome.privacy.network setting and/or introducing hooks in chrome.webRequest (https://developer.chrome.com/extensions/webRequest) for DNS request and Sockets. If you have another idea how to handle this or an addition please let me know.

You're right that the whitepaper is inaccurate, and that it once was accurate (And changing behavior out from under people isn't great).  I'll look into updating the whitepaper to reflect current behavior.

#17
Thanks for looking into it!

Just to clarify:

If you want to make the whitepaper even more accurate as it already is, that's perfectly fine for me. Although it is already mentioned in there that "Webpage prefetching is allowed regardless of whether Chrome’s network prediction service feature is enabled."

What's definitely inaccurate is the Extensions Documentation for chrome.privacy.network.networkPredictionEnabled - since it implicitly still mentions the ability to deactivate prefetch altogether.

And extending the Help Article block for the prediction service with a "Learn more" link (like other descriptions also do) to the netpredict section in the whitepaper would be really helpful for learning more about the actual behaviour.

For reference: I've opened Issue 786038 as feature request to restore full functionality for extensions that aim to block requests/connections.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/74bdb978e324645543b1e2d5925ebfc34ec2a11b

commit 74bdb978e324645543b1e2d5925ebfc34ec2a11b
Author: Matt Menke <mmenke@chromium.org>
Date: Thu Nov 16 21:02:15 2017

Fix inaccuracy in chrome.privacy.network.networkPredictionEnabled docs.

Bug:  785125 
Change-Id: I018a94e0ea980c2971a8d085adf394cf13992fa3
Reviewed-on: https://chromium-review.googlesource.com/775141
Reviewed-by: Devlin <rdevlin.cronin@chromium.org>
Commit-Queue: Matt Menke <mmenke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#517184}
[modify] https://crrev.com/74bdb978e324645543b1e2d5925ebfc34ec2a11b/chrome/common/extensions/api/privacy.json