> I've seen this bug first on 9th of Jan 2018

On January 8th, uMatrix was updated to 1.2.0.

An important difference with prior version is that with 1.2.0, uMatrix systematically inject a Content-Security-Policy and/or Content-Security-Policy-Report-Only into the response headers of all main_frame/sub_frame requests, whereas before a Content-Security-Policy header was injected only if 1st-party scripts were blocked by the user.

I feel there might be a latent bug on Chromium side when dealing with modified response headers, and now with the systematic injection or modification of response headers on main_frame/sub_frame requests, the latent bug is more likely to be triggered.

The issue seems to out of ET-scope as a IIS hosted site is needed to test the issue. Hence, adding label TE-NeedsTriageFromHYD and forwarding the issue to inhouse team for further investigation.

Thanks...!!

It has been reported that the issue can not be reproduced with uMatrix on Firefox 57:

https://github.com/gorhill/uMatrix/issues/912#issuecomment-357525522

Is there any chance someone having this issue could reproduce the problem with a netlog dump running?  (See https://sites.google.com/a/chromium.org/dev/for-testers/providing-network-details).  That might allow us to substantially narrow down where the problem is occurring in the network stack (if it's in the network stack).

I can't expose the page data or cookies and needed to alter the domain and some IPs. I hope this is useful.

Deleted: chrome-net-export-log.json 313 KB

chrome-net-export-log.json 313 KB View Download

The netlog dump is attached.
Dump was collected as "Strip private information".

Deleted: replicated_issue.zip 234 KB

replicated_issue.zip 234 KB Download

The summary on the uMatrix issue suggested by the uMatrix dev ended up being 

""When using a proxy, an extension modifying the response headers may cause Chrome to parse improperly the response body""

He indicated that the only change done by uMatrix was adding "Content-Security-Policy-Report-Only" in the response headers, and when that was disabled, the problem did not occur.  

I've looked at the netlog dumps above, and there's nothing obvious in them.  We log when extensions change headers, but not what the change is.  Interestingly, in both traces uMatrix is shown as changing the headers for the first couple of URLRequests (in one of the two cases a redirect, then for both a 401 authorization required) but not for the last two (a second 401 authorization challenge--I'm not wise enough in the ways of authentication to know why there are two--and then the actual bytes being delivered).  So this supports the hypothesis that the issue is in the relationship of the authentication challenge to the headers modification--once the authentication has completed, a reload will use the authentication cache (there are cache control headers that differ between the two logs that may or may not result in a network level reload). 

Having said that, I don't have any good ideas as to why headers modification of the type described would lead to successful authentication but to a later garbling of the data.  FWIW, the garbling looks like an encoding issue to me--either gzip (the specified Content-Encoding) is being set in error, or somehow the point at which decoding starts is being moved by a byte, resulting in the gzip decoder going off the rails.  But again I've got no clue as to why that would happen because of a headers modification to a network response which is an authorization challenge.

Adding auth network labels in case someone with that expertise has context, but leaving Internals>Network since I don't think we've narrowed down root cause.

Pretty sure the headers being added are CSP policies:
https://github.com/gorhill/uMatrix/blob/c89a4cfe5e47e3afb60d58c33f9c5bd70f153506/src/js/traffic.js#L332

Not sure if that helps drill down the issue though.

Here is some extension code that modifies the headers:
https://cs.chromium.org/chromium/src/extensions/browser/api/web_request/web_request_api_helpers.cc?rcl=7458fbc6ca2ff2fcf38bb4320183d3304fbea671&l=1075

The issue seems to out of ET-scope as a IIS hosted site is needed to test the issue. Hence, adding label TE-NeedsTriageHelp and forwarding the issue further investigation.

Thanks...!!

I have the same effect with a Java Servlet application running behind Apache and with uMatrix in the browser. It started about 2 weeks ago. On the first page after the HTTP authentication prompt, the Google Chrome browser displays the gzipped binary data in the content area. When I reload the page with F5, the page is dispayed correctly.

The server first sends a HTTP response with status code 401 (unauthorized) and the header field "WWW-Authenticate" set to "Basic realm=...". This causes the browser to display a prompt for username+password. The error only occurs when the authentication challenge response is uncompressed and the following page is compressed. When both the authentication challenge response and the following page are gzip compressed, the error does not occur.

When I set disableCSPReportInjection=true in the uMatrix options ("more" tab), the error does no longer occur.

I have written a test program to reproduce the problem.
Instructions to run the test program are within the source code.
See also: https://github.com/gorhill/uMatrix/issues/931

Deleted: testHttpServer.zip 3.7 KB

testHttpServer.zip 3.7 KB Download

Can anybody confirm that my test program works and can be used to reproduce this bug?

(It's easy to run the test program. You just need Node to run it. Start the test program with "node dist/main.js" and open the browser at "http://localhost:8080".)

Hi,

using that test server I can reproduce the bug: auth, garbled output,
correct output after f5/reload.

HTH

Thanks for the test program. That was extremely helpful! I've confirmed the bug and figured out the cause. The extension is getting called on the 401, replacing the headers, and then when we get the two hundred, we apply that state again to that, which is incorrect.

On the //net side of things, I believe the fix is just to reset the response header override state when we restart the request for auth. That said, the second iteration doesn't appear to trigger the extension at all, so that appears to be additionally be a bug on the extensions side of things.

Splitting the second issue over to issue #809761 for the extensions folks to fix. I have a CL to fix this issue here:
https://chromium-review.googlesource.com/#/c/chromium/src/+/905586

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0bda12fe10d2e8b007394d3ab210cf6d958740a1

commit 0bda12fe10d2e8b007394d3ab210cf6d958740a1
Author: David Benjamin <davidben@chromium.org>
Date: Wed Feb 07 19:58:45 2018

Clear override_response_headers_ when restarting for auth.

Otherwise we get a little confused when extensions add headers when HTTP
auth happens. Notably, if the 401 is not gzipped and the 200 is,
hilarious things happen.

Bug:  801237 
Change-Id: I3c1db55a3840a6c5180a603f74b4c65b57635b67
Reviewed-on: https://chromium-review.googlesource.com/905586
Commit-Queue: David Benjamin <davidben@chromium.org>
Reviewed-by: Matt Menke <mmenke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#535112}
[modify] https://crrev.com/0bda12fe10d2e8b007394d3ab210cf6d958740a1/net/url_request/url_request_http_job.cc
[modify] https://crrev.com/0bda12fe10d2e8b007394d3ab210cf6d958740a1/net/url_request/url_request_test_util.cc
[modify] https://crrev.com/0bda12fe10d2e8b007394d3ab210cf6d958740a1/net/url_request/url_request_test_util.h
[modify] https://crrev.com/0bda12fe10d2e8b007394d3ab210cf6d958740a1/net/url_request/url_request_unittest.cc

That should hopefully fix it.

Cool. Thanks. I don't have the required 100GiB free to compile chromium on this machine here - is there a CI build publicly available? Do we get notified when a fixed canary chrome and/or production chrome is released? I can't even find a detailed changelog anywhere...

The fix is in Chrome 66, which should be released to stable sometime in April. I believe it is in Chrome Canary now. (Specifically the version number needs to be at least 66.0.3343.0.)