Added TE-NeedsTriageHelp as it can't be triaged from TE end.

The 1MB restriction has been in place for years, and this is the first report I have seen of problems from it.

* Can you give some additional details on where such a big PAC script is coming from, and why it needs to be this large?

* What size limit would work for your use case?

* Are you aware of other environments that would use such large scripts?

Cheers.

Hello, +eroman.

> Can you give some additional details on where such a big PAC script is coming from, and why it needs to be this large?

In Russia Internet is censored. Internet providers are obliged to censor sites by domain names and IPs.
Full list of sites that must be censored may be found in this updated file: [ https://raw.githubusercontent.com/zapret-info/z-i/master/dump.csv ]. It is 6.64MB in size and contains IPs, domain names, authorities, dates and reasons of blockings.

I develop an anticensorship solution for Russian Internet. The idea is to take this dump.csv, generate PAC script with proxies from it and apply to browser via Web Extension. Internally PAC scripts doesn't work on a simple data structure like array of IPs, but inclines to trie-like structure based on `switch`es. For sites "foo.ru" and "bar.com" trie-like structure looks like:
```js
// `domains` is ["foo", "ru"] or ["bar", "com"].
switch(domains.pop()) {
  case "ru":
    switch(domains.pop()) {
      case "foo"
        return true;
    }
  case "com":
    switch(domains.pop()) {
      case "bar"
        return true;
    return false;
  }
}
return false;
```
You may find a real PAC file we generate here: [ https://github.com/anticensorship-russia/generated-pac-scripts/ ].
It is 2.41MB currently.

> What size limit would work for your use case?

On Jan 06 2016 dump.csv was 2.81MB in size: [ https://github.com/zapret-info/z-i/blob/147004744812e6e53dee80a4ed0b101e3fa92b84/dump.csv ].
The speed of growing is: 6.64MB (06.01.17) - 2.81MB (06.01.16) = 3.83 (MB per last year).

(16MB - 6.64MB)/3.83 = 2.44 years   // 16MB  would be enough for 2.44  years from today*.
(32MB - 6.64MB)/3.83 = 6.62 years   // 32MB  would be enough for 6.62  years from today*.
(64MB - 6.64MB)/3.83 = 14.98 years  // 64MB  would be enough for 14.98 years from today*.
(128MB - 6.64MB)/3.83 = 31.69 years // 128MB would be enough for 31.69 years from today*.

* if speed of growing won't change.

I think about 32MB would be enough for several years.
Maybe such big PAC scripts would be inefficient and slow to use.

> Are you aware of other environments that would use such large scripts?

Firefox works with our 2.41MB PAC script.
I don't know any other use cases except anticensorhsip that would require such big scripts.

Best regards,
Ilya.

Interesting use of PAC script, thanks for those details ilyaigpetrov@ !

For auto-detected PAC script I wouldn't want to change our size policy of 1MB. For explicitly configured PAC scripts there may be some wiggle room, but 32MB still sounds excessive.

My initial recommendation is to reduce the overall size of your PAC script. Based on the predicted rate of growth, the performance of this current approach is not going to scale well, even if the browser allowed it.

A couple ideas:

* Run the existing code through a Javascript minifier -- this can probably compress it by 50-60% without any other changes, and might bring you to under 1MB.

* Encode the rules as data rather than generated code

* Use a bloom filter rather than a trie. Depending on the accuracy you can tolerate, this could be quite compact and fast.

* Use a tool to convert your Javascript to a self-extracting program. Can probably squash it down to 200K-300K with such an approach, and there are tools out there for this.

Cheers

This doesn't seem to be extension specific, rather it is a net/ issue. Removing extensions label.

It seems --proxy-pac-url chrome flag is affected too.

eroman@, we still use huge PAC files and we currently have 45k users ( see https://rebrand.ly/ac-webstore ). The size of the PAC file is about 3.47 MB, and I may get 1.9MB by changing algo/approach. The data (ips + hosts) is 1.524MB for plain lists. I hope you'll reconsider your decision, at least we could write tests with --proxy-pac-url for our scripts if you raised the limit.

Here in Russia we have so many blocked websites that we just hit 1MiB PAC file limit and it stopped working.
Is it planned to increase this limit?

The problem in Russia is with the opening of some sites. Can I increase the size of more than 1 megabyte?

Please, fix the problem with 1Mb limit for automatic proxy configuration .PAC-file size. We are from RF and use automatic proxy configuration for free Internet surfing.

Thanks for the additional feedback.
I will mark this as available (but low priority).

Per comment #6, I think we can address this by:

  (1) Keep the limit for auto-detected PAC URLs as it currently is (1MB)

  (2) Raise the limit for *explicitly* configured PAC URLs (by extension/system settings/chrome policy) to something higher. Bumping it up to 4MB or 8MB wouldn't have too much risk, so I would propose starting there.

Maybe even make the limit configurable through chrome:flags?

For now, I've found a workaround by launching the browser with --winhttp-proxy-resolver key (works on Windows and OSX).

eroman@, I would propose 32MB for the explicitly set PAC-scripts, it should be enough for the next decade.
What risk do you see about it?

Because the PAC script is downloaded in its entirety, and kept in the browser, the risks are around reliability and potential for abuse.

As an example, say the limit were 256MB.
Configuring a bad PAC URL (whether accidental or done maliciously) would now end up downloading the full 256MB of data (which is slow and uses bandwidth). This could be repeated for multiple network contexts, so now maybe we are using 512MB. And the provisioning a JavaScript VM for this could again double the effective memory needed, bringing us to 1GB

This is an extreme example, but illustrates the sorts of concerns with having unbounded growth for PAC scripts.

8MB seems good. Can you please propose the size to be changed in the next Chromium version? I'm struggling to shrink my pac file more, it exceeds the limit every several days.

Any plans to increase PAC file size limit in next browser version? Please put some more attention to this issue if possible.

There is no milestone target for this.

Does anyone have data on what Microsoft Edge's limit is? I seem to recall it was also fairly small.

For the Web Extension case also be aware that using a data: URL would give you some wiggle-room, however that you would run into problems at 2MB (since Chrome limits URLs to that size).

A solution today could be to use a webRequest extension (https://developer.chrome.com/extensions/webRequest) to do the blocking.

https://developer.chrome.com/extensions/webRequest may be used for
blocking, but our use case is applying proxy to sites from a list, I
doubt it can be done with webRequest.

In our extension we use the data field of
https://developer.chrome.com/extensions/proxy#type-PacScript which
doesn't have 1MB restriction and has no relation to data URLs.

Also many requests here pertain to --proxy-pac-url command line option
of Chromium, not to extensions API directly.