Note: on Chrome 54.0.2840.8 dev-m (Win10, x64)I am currently not getting this, but I had been, and others still are.

devlin@ -- could you please take a look at this? It's not clear whether this is a Chrome/Chromium bug.

I don't think there is a vulnerability here, so dropping view restrictions.

The problem here seems to be that HTTPS Everywhere and Privacy Badger aren't getting signed file content hashes generated by the webstore (no _metadata/verified_contents.json file in the .crx file), and when we make a request to the webstore to ask for this file we fail to get a reply, so chrome cannot tell that the files on disk are untampered with. In earlier chrome versions we used to just keep requesting a copy of verified_contents.json from the webstore if it was missing, but we found malware in the wild abusing this behavior to keep their changes to legitimate extensions from being detected, so in chrome 53 we made things stricter by doing one attempt to fetch the verified_contents.json and disabling the extension if that failed.

For most extensions in the webstore, this file gets added to the .crx during the publishing step so no separate request to the webstore is needed. I seem to recall that a while back that the webstore had some special process for the EFF to upload .crx files (instead of unsigned .zip files like most other extensions) because they didn't want Google to have a copy of their private key. When we first rolled out extension content verification I thought we had accounted for this and were going to be able to separately generate verified_contents.json files that could be downloaded, but either that process broke at some point or was never completed. I'll investigate this with the webstore folks. 

BTW, content verification is only turned on by default in Google Chrome branded official builds, and is off by default in chromium source, so depending on where you got your build of chrome/chromium that can account for whether you'd see this or not. 

Thank you for your response and investigating this.

Do you have a timeline for when this issue can be resolved on your end?  If it's a quick fix and no changes are required from us, that's great and we'll block on your updates.

If it requires some extra time from you, we may consider modifying our process to ensure that at least you can be delivered an extension in .zip format that `verified_contents.json` can be generated from.

Please let us know so we can consider how to proceed.

Privacy Badger developer here. Thanks for paying attention to this issue, I just want to reiterate that this is a super urgent issue for us as many of our users now have the extension disabled with no option to turn it back on, which as you might expect, has caused a great deal of concern and negative feedback from our users. We would really like to get this resolved ASAP so that people can use Privacy Badger and HTTPS Everywhere again. 

I've contacted the webstore team asking them to look into the code on their end that handles the special case of uploads of already-packaged .crx files and whether this should be working but has broken, or needs to implemented and somehow slipped through the cracks. I'll post again here as soon as I hear back, hopefully within a couple of hours. 

As far as workarounds, I see a couple of possibilities, none of which is a very good solution:

a) Create new entries in the webstore for each item, following the typical process of uploading the source in a .zip file and letting the webstore assign a private key and create the .crx file (which will include the verified_contents.json file). 
If I understand the history correctly, we developed the special "directly upload a .crx file" process for the EFF because there were concerns around Google having a copy of the extension's private key, and it was desirable to be able to assert that Google could not deliver a modified version of the extension to everyone or specific users, so we'd lose that property.

b) Have users download the source themselves, enable developer mode on chrome://extensions, and use the "Load unpacked extension" button load it in developer mode. This is unappealing for various reasons: the steps are arcane for non-developer users, users doing this won't get future autoupdates, and also we issue an annoying warning about developer-mode on every startup (due to malware using it as a vector for force-installing unwanted extensions). 

c) [This may or may not even work at all] See if you can upload a .zip file copy of your extensions containing a copy of the existing private key following the little-documented process here: https://groups.google.com/forum/#!msg/chromium-extensions/3vvygtEajMQ/qcA03E81jq8J . This again would have the downside of needing to share the private key with Google. 

Ok, I just got an update from the webstore folks while typing my previous comment. It looks like they are generating the verified_contents.json file for your extensions, but something is going wrong when we attempt to download them. I'm going to investigate if this is a client bug or something going wrong in our download infrastructure. 

Update: it turns out this is a bug in chrome's code, not the webstore. I've found where the bug is and the fix is fairly simple, so I should be able to get it checked in within a day or two and then can get it included in the next stable push - I'm not sure off the top of my head when that will be, but usually we push stability/security updates on the stable channel every week or two. 

I'm also going to see if I can add something to help automatically re-enable the disabled extensions so that users won't have to do so manually; I'm not sure off the top of my head if that will turn out to be tricky or not. 

My apologies again for the disruption this has caused your users. 

Thanks asargent, that is very helpful! Perhaps you could offer some insight into this question, if it is going to potentially be a week or two until a fix is out it might make sense for us to push a zip file to the chrome store to temporarily fix this problem. Obviously this will change the key that the extension is signed with. And then when chrome is fixed and we push our signed version again the key will change back again to EFFs private key. How likely is this to cause problems for our users? 

I don't think we could seamlessly push an update to existing users since the extension id is based on the public key, and the autoupdate process relies on the new version being signed with the corresponding private key. So I think the best we could do is have you create new entries in the webstore using your extension's source that would end up with new extension id's, and then users would have to manually go and install those. So not a great user experience. 

However, I just had a thought about something we might be able to do to manually work around the chrome bug, which concerns how the client code handles the download of the verified_contents.json file. It might just work for you to take a copy of the verified_contents.json file that the webstore has generated, add that as the path "_metadata/verified_contents.json" in an unzipped copy of your extension source, re-sign that with your private key, and then upload the resulting .crx file to the webstore. Then users could hit the "Repair" button on chrome-extensions://, which would download and install this new .crx file, bypassing the buggy code that handles the separately fetched verified_contents.json file. 

Because the extension version is part of the information in verified_contents.json that gets signed, it's important that you do *not* increment the version number in the manifest.json file. One problem we might run into is that under normal circumstances the webstore insists that new uploads of an extension have an incremented version number, and even if the special case .crx upload process you folks use bypasses this, it's possible that the download-serving infrastructure is set up not to ever allow changing what "version X" of a particular extension is. If we do run into this problem, it's possible that I can take a copy of your source, increment the version number, and have the webstore folks generate a valid verified_contents.json file based on that which we can send to you, and then you can add it to your source, sign with your private key, and upload the .crx. 

BTW, if you're curious about the format and meaning of data in the verified_contents.json file, there is a bunch of documentation in the comments in this file:
https://cs.chromium.org/chromium/src/extensions/browser/verified_contents.cc

This makes sense as a solution to me.  Will you be able to provide the "_metadata/verified_contents.json" file for HTTPS Everywhere and Privacy Badger tomorrow for us to attempt this?

I've run in to the error where a new version is required as part of the normal extension upgrade process.  Unless there's a way for your Web Store team to manually turn this off, I propose we increment the version number, send the generated crx to you for you to manually generate the required json file which you can then send back to us for us to sign and upload to the CWS.  Does this workflow sound correct to you?

Will a timeline of tomorrow work on your end?

Your proposed set of steps sounds correct. 

I'll touch base again with the webstore engineers today to see if this is feasible.

Ping asargent - sorry to be a bother but we're awaiting any news you might have from them.

Unfortunately I haven't heard back yet, so we don't have an answer one way or the other yet. 

(In the meantime I've been working on getting my patch ready to fix the bug)

Hey asargent,

Any update on the patch and when it may be deployed?

I'm still waiting to hear back from the webstore team - I think the delay is that the one engineer who is most familiar with this area of their code was out of the office. I've pinged the email thread again. 

I ran into a tricky-to-diagnose bug in writing the tests for my fix that took longer to figure out than I would have hoped, but figured out what was going wrong yesterday and should be getting the change up for review today. Hopefully it can be submitted later today or tomorrow. 

I'm not sure if there are any more planned stable releases of the chrome 53 branch beyond one they finalized last week and are pushing today, so the next stable release wouldn't be for ~4 weeks until chrome 54 becomes stable. But if the webstore workaround doesn't pan out, I'll try making the case for this bug being important enough to spin up another release and see what the release managers decide. 

Fix is now up for codereview at https://codereview.chromium.org/2336403002/

asargent, thanks for the updates.  I certainly hope we can get this resolved before Chrome 54 hits stable!  Our extension has been disabled already for these users for nearly two weeks, due to no fault of our own.  I appreciate if you can make the case for a release fixing this, otherwise we will be disabled for nearly 6 weeks for Chrome stable users.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e4de9f973eae513486d2912e1793e943fce70014

commit e4de9f973eae513486d2912e1793e943fce70014
Author: asargent <asargent@chromium.org>
Date: Thu Sep 15 01:40:06 2016

Fix extension content verification out-of-band hash fetching

When we changed content verification to strict mode (crbug.com/601512),
it turns out that the code for downloading missing
verified_contents.json files from the webstore had two longstanding bugs
that we never noticed due to insufficient tests:

a) Returning the results of a boolean expression as an integer
b) Use after free of an object

The end result is that when the patch for crbug.com/601512 hit stable,
many users became unable to run the EFF's HTTPS Everywhere and Privacy
Badger extensions because those are uploaded via a special process where
the webstore does not have a copy of their private key and cannot
include the verified_contents.json file in the .crx file; rather we rely
on being able to download it after install but fail due to the above
mentioned bugs.

This patch fixes the bugs and also introduces a change which clears the
corruption "disable reason" on extension autoupdate, so that users who
had the extension disabled because of this bug can get it re-enabled
automatically by an extension autoupdate.

BUG= 643814 

Review-Url: https://codereview.chromium.org/2336403002
Cr-Commit-Position: refs/heads/master@{#418748}

[modify] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/chrome/browser/extensions/chrome_content_verifier_delegate.cc
[modify] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/chrome/browser/extensions/extension_service.cc
[modify] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/chrome/browser/extensions/extension_service_unittest.cc
[modify] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/chrome/common/extensions/extension_constants.cc
[modify] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/chrome/common/extensions/extension_constants.h
[modify] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/extensions/browser/BUILD.gn
[modify] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/extensions/browser/content_hash_fetcher.cc
[modify] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/extensions/browser/content_hash_fetcher.h
[add] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/extensions/browser/content_hash_fetcher_unittest.cc
[modify] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/extensions/browser/content_verifier.cc
[modify] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/extensions/common/constants.cc
[modify] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/extensions/common/constants.h
[add] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/extensions/test/data/content_hash_fetcher/missing_verified_contents/README.txt
[add] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/extensions/test/data/content_hash_fetcher/missing_verified_contents/source.zip
[add] https://crrev.com/e4de9f973eae513486d2912e1793e943fce70014/extensions/test/data/content_hash_fetcher/missing_verified_contents/verified_contents.json

asargent: Any update on the release schedule?

This didn't make it into Friday's canary build, so I had to wait until today to make sure there was at least a couple of days of canary use with no problem reports before I could request a merge into the M54 beta branch. I've also communicated with the M53 stable branch release managers and it is sounding pretty unlikely that they'll be willing to take this patch, but having some time in the M54 beta branch would be a prerequisite for that anyway. 

I understand that this is probably very frustrating for you and your affected users - the release process is conservative because they generally don't like to risk the stability of hundreds of millions of users to fast track fixes to problems that only affect several thousand. 

FYI, I'm also still investigating the out-of-band signing approach I mentioned in comment 13 - I've been talking with the webstore engineers but there are some subtleties and it's still not clear if we'll be able to make it work or not. 

Your change meets the bar and is auto-approved for M54 (branch: 2840)

And whoops, "several thousand" actually isn't quite right - it looks like these both have between several hundred thousand to a bit over a million users. 
 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec

commit 8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec
Author: Antony Sargent <asargent@chromium.org>
Date: Tue Sep 20 05:19:45 2016

Fix extension content verification out-of-band hash fetching

When we changed content verification to strict mode (crbug.com/601512),
it turns out that the code for downloading missing
verified_contents.json files from the webstore had two longstanding bugs
that we never noticed due to insufficient tests:

a) Returning the results of a boolean expression as an integer
b) Use after free of an object

The end result is that when the patch for crbug.com/601512 hit stable,
many users became unable to run the EFF's HTTPS Everywhere and Privacy
Badger extensions because those are uploaded via a special process where
the webstore does not have a copy of their private key and cannot
include the verified_contents.json file in the .crx file; rather we rely
on being able to download it after install but fail due to the above
mentioned bugs.

This patch fixes the bugs and also introduces a change which clears the
corruption "disable reason" on extension autoupdate, so that users who
had the extension disabled because of this bug can get it re-enabled
automatically by an extension autoupdate.

BUG= 643814 

Review-Url: https://codereview.chromium.org/2336403002
Cr-Commit-Position: refs/heads/master@{#418748}
(cherry picked from commit e4de9f973eae513486d2912e1793e943fce70014)

Review URL: https://codereview.chromium.org/2350403002 .

Cr-Commit-Position: refs/branch-heads/2840@{#434}
Cr-Branched-From: 1ae106dbab4bddd85132d5b75c670794311f4c57-refs/heads/master@{#414607}

[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/chrome/browser/extensions/chrome_content_verifier_delegate.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/chrome/browser/extensions/extension_service.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/chrome/browser/extensions/extension_service_unittest.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/chrome/common/extensions/extension_constants.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/chrome/common/extensions/extension_constants.h
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/browser/content_hash_fetcher.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/browser/content_hash_fetcher.h
[add] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/browser/content_hash_fetcher_unittest.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/browser/content_verifier.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/common/constants.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/common/constants.h
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/extensions_tests.gypi
[add] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/test/data/content_hash_fetcher/missing_verified_contents/README.txt
[add] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/test/data/content_hash_fetcher/missing_verified_contents/source.zip
[add] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/test/data/content_hash_fetcher/missing_verified_contents/verified_contents.json

Removing from the blocker list since the change has already been merged.

This is a pretty nasty bug, that's been affecting huge numbers of our users for a long time. Was it considered for Chrome 53 stable channel updates? Did we make a mistake in not asking for that sooner?

I see that Chrome 54 has been pushed.  How do I test the following:

> This patch fixes the bugs and also introduces a change which clears the
corruption "disable reason" on extension autoupdate, so that users who
had the extension disabled because of this bug can get it re-enabled
automatically by an extension autoupdate.

Do I have to cut a new release of the extension?

@30 - I'm very sorry about the impact; I understand it has been really bad for users of the 2 extensions affected. I did talk with the release managers for the chrome 53 release but they were uncomfortable taking the patch (they were sympathetic to the impact of the bug, but there were also a number of unrelated bugs that destabilized the release and they were feeling particularly conservative about taking non-critical patches). 

@31: Yes, now would be a good time to release a new version, because the fix I landed should clear the corruption bit on an extension and re-enable it during autoupdate. Note that it will take several days to a week or two for a majority of the userbase to be updated (we throttle things from the server side and watch for things like upticks in crash reports). It's probably a good idea to release a new version every 2 days or so for about a week to catch people newly upgraded to 54, and then add a minimum_chrome_version value of 54 to the manifest (https://developer.chrome.com/extensions/manifest/minimum_chrome_version). We want to add the minimum_chrome_version for two reasons: (1) so that existing users still on chrome 53  won't take the update, and (2) so that new users still on chrome 53 won't try installing the extension only to have it get immediately disasbled. 

asargent, another way this has affected us negatively is that there have been a flurry of negative reviews for the extension:

https://chrome.google.com/webstore/detail/https-everywhere/gcbommkclmclpchllfjekcdonpmejbdp/reviews?hl=en

The first negative review mentioning the "extension corrupted" issue was on June 10th.  As this issue was caused by Chrome, can the Chrome Web Store remove the reviews that rate us badly due to this issue?  It seems that would be a minimum of redress.

@33 - Can you first try contacting webstore developer support at:

https://support.google.com/chrome_webstore/contact/developer_support/

I'm not sure what their policies around this kind of thing are, but you can explain the situation and point to this bug.

In the meantime, you might want to update your description to mention this was a bug in chrome and out of your control, and include a link to this bug (you can use  https://crbug.com/643814  as a shortlink). You could also try replying to a few of the negative reviews letting people know we fixed the bug in chrome 53 onwards. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec

commit 8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec
Author: Antony Sargent <asargent@chromium.org>
Date: Tue Sep 20 05:19:45 2016

Fix extension content verification out-of-band hash fetching

When we changed content verification to strict mode (crbug.com/601512),
it turns out that the code for downloading missing
verified_contents.json files from the webstore had two longstanding bugs
that we never noticed due to insufficient tests:

a) Returning the results of a boolean expression as an integer
b) Use after free of an object

The end result is that when the patch for crbug.com/601512 hit stable,
many users became unable to run the EFF's HTTPS Everywhere and Privacy
Badger extensions because those are uploaded via a special process where
the webstore does not have a copy of their private key and cannot
include the verified_contents.json file in the .crx file; rather we rely
on being able to download it after install but fail due to the above
mentioned bugs.

This patch fixes the bugs and also introduces a change which clears the
corruption "disable reason" on extension autoupdate, so that users who
had the extension disabled because of this bug can get it re-enabled
automatically by an extension autoupdate.

BUG= 643814 

Review-Url: https://codereview.chromium.org/2336403002
Cr-Commit-Position: refs/heads/master@{#418748}
(cherry picked from commit e4de9f973eae513486d2912e1793e943fce70014)

Review URL: https://codereview.chromium.org/2350403002 .

Cr-Commit-Position: refs/branch-heads/2840@{#434}
Cr-Branched-From: 1ae106dbab4bddd85132d5b75c670794311f4c57-refs/heads/master@{#414607}

[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/chrome/browser/extensions/chrome_content_verifier_delegate.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/chrome/browser/extensions/extension_service.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/chrome/browser/extensions/extension_service_unittest.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/chrome/common/extensions/extension_constants.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/chrome/common/extensions/extension_constants.h
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/browser/content_hash_fetcher.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/browser/content_hash_fetcher.h
[add] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/browser/content_hash_fetcher_unittest.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/browser/content_verifier.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/common/constants.cc
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/common/constants.h
[modify] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/extensions_tests.gypi
[add] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/test/data/content_hash_fetcher/missing_verified_contents/README.txt
[add] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/test/data/content_hash_fetcher/missing_verified_contents/source.zip
[add] https://crrev.com/8ad1b10e02b6fa6eb4f7dd0d89518d33c6a0a6ec/extensions/test/data/content_hash_fetcher/missing_verified_contents/verified_contents.json

Even after clicking "repair" on Chrome 54, many users are reporting that the issue is not resolved.  Do you have advice for these users?

I just tried it myself a few times on windows 10 and it seems to work ok. Can you get any details from the users experiencing this? Knowing what OS they're running would be helpful, as would knowing if they have any other extensions installed that might be interfering with the network request to fetch the verified_contents.json file. 

Another thing that would be very helpful would be to have the affected users open up a tab to chrome://net-internals and leave it open, then trying the repair, and then going to the net-internals tab and opening the "Events" section and searching for "installsource%3Dsignature" which is the request for the verified_contents.json - they should see a request for an URL that looks like:

https://clients2.google.com/service/update2/crx?response=redirect&x=uc%26installsource%3Dsignature%26id%3Dgcbommkclmclpchllfjekcdonpmejbdp%26v%3D2016.10.20

and the details should show a 302 redirect to an url something like this:

https://clients2.googleusercontent.com/crx/blobs/QgAAAC6zw0qH2DJtnXe8Z7rUJP3-dOM73Sdm5mbyLgo7LoOWDCmrx7ShfhraweMq2yE08dGgR9aAydz0UgSf3CN6JQRfwaZL0PFkQmTlEqcxpHm-AMZSmuW_uNmoQCrgr5kOaj72Ffby2e6MHA/extension_2016_10_20.crx

and the fetch of that should look like it succeeded - the bottom of that event should look something like:

t=14543 [st=193]     -HTTP_TRANSACTION_READ_HEADERS
t=14543 [st=193]      URL_REQUEST_DELEGATE  [dt=0]
t=14543 [st=193]   -URL_REQUEST_START_JOB
t=14543 [st=193]    URL_REQUEST_DELEGATE  [dt=0]
t=14543 [st=193]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=14543 [st=193]    URL_REQUEST_JOB_BYTES_READ
                    --> byte_count = 4096
t=14543 [st=193]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=14543 [st=193]    URL_REQUEST_JOB_BYTES_READ
                    --> byte_count = 4096
t=14543 [st=193]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=14543 [st=193]    URL_REQUEST_JOB_BYTES_READ
                    --> byte_count = 4096
t=14543 [st=193]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=14543 [st=193]    URL_REQUEST_JOB_BYTES_READ
                    --> byte_count = 4096
t=14543 [st=193]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=14543 [st=193]    URL_REQUEST_JOB_BYTES_READ
                    --> byte_count = 4096
t=14543 [st=193]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=14543 [st=193]    URL_REQUEST_JOB_BYTES_READ
                    --> byte_count = 4096
t=14543 [st=193]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=14543 [st=193]    URL_REQUEST_JOB_BYTES_READ
                    --> byte_count = 323
t=14543 [st=193]    HTTP_TRANSACTION_READ_BODY  [dt=2]
t=14545 [st=195]    URL_REQUEST_JOB_BYTES_READ
                    --> byte_count = 4096
t=14545 [st=195]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=14545 [st=195]    URL_REQUEST_JOB_BYTES_READ
                    --> byte_count = 4096
t=14545 [st=195]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=14545 [st=195]    URL_REQUEST_JOB_BYTES_READ
                    --> byte_count = 2464
t=14545 [st=195]    HTTP_TRANSACTION_READ_BODY  [dt=2]
t=14547 [st=197]    URL_REQUEST_JOB_BYTES_READ
                    --> byte_count = 4096
t=14547 [st=197]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=14547 [st=197]    URL_REQUEST_JOB_BYTES_READ
                    --> byte_count = 4096
t=14547 [st=197]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=14547 [st=197]    URL_REQUEST_JOB_BYTES_READ
                    --> byte_count = 586
t=14547 [st=197]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=14547 [st=197] -REQUEST_ALIVE

Another thought I just had about this - occasionally we've seen windows antivirus programs interfering with the download or writing to the local disk of json files - it would also be helpful to know what antivirus software, if any, the users still experiencing problems are using.  

This is what I see in net-internals upon repair:

t=15916 [st= 0] +REQUEST_ALIVE  [dt=68]
t=15916 [st= 0]    URL_REQUEST_DELEGATE  [dt=0]
t=15916 [st= 0]   +URL_REQUEST_START_JOB  [dt=68]
                   --> load_flags = 592 (DISABLE_CACHE | DO_NOT_SAVE_COOKIES | DO_NOT_SEND_COOKIES)
                   --> method = "GET"
                   --> priority = "LOWEST"
                   --> url = "https://clients2.google.com/service/update2/crx?response=redirect&x=uc%26installsource%3Dsignature%26id%3Dgcbommkclmclpchllfjekcdonpmejbdp%26v%3D2016.11.8"
t=15916 [st= 0]      URL_REQUEST_DELEGATE  [dt=0]
t=15916 [st= 0]      HTTP_CACHE_GET_BACKEND  [dt=0]
t=15916 [st= 0]     +HTTP_STREAM_REQUEST  [dt=1]
t=15916 [st= 0]        HTTP_STREAM_REQUEST_STARTED_JOB
                       --> source_dependency = 1045 (HTTP_STREAM_JOB)
t=15917 [st= 1]        HTTP_STREAM_REQUEST_STARTED_JOB
                       --> source_dependency = 1044 (HTTP_STREAM_JOB)
t=15917 [st= 1]        HTTP_STREAM_REQUEST_BOUND_TO_JOB
                       --> source_dependency = 1045 (HTTP_STREAM_JOB)
t=15917 [st= 1]     -HTTP_STREAM_REQUEST
t=15917 [st= 1]      HTTP_STREAM_REQUEST_BOUND_TO_QUIC_SESSION
                     --> source_dependency = 651 (QUIC_SESSION)
t=15917 [st= 1]     +HTTP_TRANSACTION_SEND_REQUEST  [dt=0]
t=15917 [st= 1]        HTTP_TRANSACTION_QUIC_SEND_REQUEST_HEADERS
                       --> :authority: clients2.google.com
                           :method: GET
                           :path: /service/update2/crx?response=redirect&x=uc%26installsource%3Dsignature%26id%3Dgcbommkclmclpchllfjekcdonpmejbdp%26v%3D2016.11.8
                           :scheme: https
                           accept-encoding: gzip, deflate, sdch, br
                           accept-language: en-US,en;q=0.8
                           user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.100 Safari/537.36
                       --> quic_priority = 1
                       --> quic_stream_id = 15
t=15917 [st= 1]     -HTTP_TRANSACTION_SEND_REQUEST
t=15917 [st= 1]     +HTTP_TRANSACTION_READ_HEADERS  [dt=67]
t=15983 [st=67]        HTTP_TRANSACTION_READ_RESPONSE_HEADERS
                       --> HTTP/1.1 204
                           status: 204
                           cache-control: no-cache, no-store, max-age=0, must-revalidate
                           pragma: no-cache
                           expires: Mon, 01 Jan 1990 00:00:00 GMT
                           date: Fri, 18 Nov 2016 19:52:58 GMT
                           server: GSE
                           alt-svc: quic=":443"; ma=2592000; v="36,35,34"
t=15984 [st=68]     -HTTP_TRANSACTION_READ_HEADERS
t=15984 [st=68]      URL_REQUEST_DELEGATE  [dt=0]
t=15984 [st=68]   -URL_REQUEST_START_JOB
t=15984 [st=68]    URL_REQUEST_DELEGATE  [dt=0]
t=15984 [st=68]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=15984 [st=68] -REQUEST_ALIVE

@39 that's very interesting - looks like we served up an http 204 (No Content) instead of the 302 redirect. 

waffles: can you think of any reason we might be returning an http 204 for these  installsource=signature requests (see comments 36,37,39)? Perhaps throttling? Or maybe the underlying .crx file was just updated to a new version, but we don't have the verified_contents.json to serve quite yet (or it hasn't propagated to all the download servers)?

Maybe it is just throttling -- I just tried to repair again and it worked.

I seem to be having the problem, and unable to get rid of it -- so some details as follows:

Running Windows 10 Pro x64 with Avira Antivirus Free (includes Firewall and Real-time Protection) as well as Malwarebytes Anti-Malware Premium.

Going with that line of thinking that it was, indeed, perhaps my antivirus as I have known it to be quite intrusive in the past (though nowhere near as intrusive, so far, as BitDefender formerly I disabled all assets in the Avira protection panel and exit/closed out of Malwarebytes entirely however, this did not resolve the issue as it occurred even still with them disabled. Without going so far as to fully uninstall either source of protection, I'm not sure how else I could circumvent them.

Currently have the following extensions running in Chrome: DuckDuckGo for Chrome, HTTPS Everywhere, LastPass: Password Manager, Privacy Badger, Reddit Enhancement Suite, uBlock Origin, and uMatrix.

Installed HTTPS Everywhere through the Chrome Store.
Build of HTTPS Everywhere, appears to be the latest, listed as 2016.11.30
Chrome is listed on Version 55.0.2883.75 m (64-bit)

and the following, is what I pull from my net internals.

2097: URL_REQUEST
https://clients2.google.com/service/update2/crx?response=redirect&x=uc%26installsource%3Dsignature%26id%3Dgcbommkclmclpchllfjekcdonpmejbdp%26v%3D2016.11.30
Start Time: 2016-12-06 08:50:42.086

t=43422 [st= 0] +REQUEST_ALIVE  [dt=72]
t=43422 [st= 0]    URL_REQUEST_DELEGATE  [dt=0]
t=43422 [st= 0]   +URL_REQUEST_START_JOB  [dt=72]
                   --> load_flags = 592 (DISABLE_CACHE | DO_NOT_SAVE_COOKIES | DO_NOT_SEND_COOKIES)
                   --> method = "GET"
                   --> priority = "LOWEST"
                   --> url = "https://clients2.google.com/service/update2/crx?response=redirect&x=uc%26installsource%3Dsignature%26id%3Dgcbommkclmclpchllfjekcdonpmejbdp%26v%3D2016.11.30"
t=43422 [st= 0]      URL_REQUEST_DELEGATE  [dt=0]
t=43422 [st= 0]      HTTP_CACHE_GET_BACKEND  [dt=0]
t=43422 [st= 0]     +HTTP_STREAM_REQUEST  [dt=1]
t=43423 [st= 1]        HTTP_STREAM_REQUEST_STARTED_JOB
                       --> source_dependency = 2099 (HTTP_STREAM_JOB)
t=43423 [st= 1]        HTTP_STREAM_REQUEST_STARTED_JOB
                       --> source_dependency = 2098 (HTTP_STREAM_JOB)
t=43423 [st= 1]        HTTP_STREAM_REQUEST_BOUND_TO_JOB
                       --> source_dependency = 2099 (HTTP_STREAM_JOB)
t=43423 [st= 1]     -HTTP_STREAM_REQUEST
t=43423 [st= 1]      HTTP_STREAM_REQUEST_BOUND_TO_QUIC_SESSION
                     --> source_dependency = 262 (QUIC_SESSION)
t=43423 [st= 1]     +HTTP_TRANSACTION_SEND_REQUEST  [dt=0]
t=43423 [st= 1]        HTTP_TRANSACTION_QUIC_SEND_REQUEST_HEADERS
                       --> :authority: clients2.google.com
                           :method: GET
                           :path: /service/update2/crx?response=redirect&x=uc%26installsource%3Dsignature%26id%3Dgcbommkclmclpchllfjekcdonpmejbdp%26v%3D2016.11.30
                           :scheme: https
                           accept-encoding: gzip, deflate, sdch, br
                           accept-language: en-US,en;q=0.8
                           user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.75 Safari/537.36
                       --> quic_priority = 1
                       --> quic_stream_id = 21
t=43423 [st= 1]     -HTTP_TRANSACTION_SEND_REQUEST
t=43423 [st= 1]     +HTTP_TRANSACTION_READ_HEADERS  [dt=71]
t=43494 [st=72]        HTTP_TRANSACTION_READ_RESPONSE_HEADERS
                       --> HTTP/1.1 204
                           status: 204
                           cache-control: no-cache, no-store, max-age=0, must-revalidate
                           pragma: no-cache
                           expires: Mon, 01 Jan 1990 00:00:00 GMT
                           date: Tue, 06 Dec 2016 14:50:48 GMT
                           server: GSE
                           alt-svc: quic=":443"; ma=2592000; v="35,34"
t=43494 [st=72]     -HTTP_TRANSACTION_READ_HEADERS
t=43494 [st=72]      URL_REQUEST_DELEGATE  [dt=0]
t=43494 [st=72]   -URL_REQUEST_START_JOB
t=43494 [st=72]    URL_REQUEST_DELEGATE  [dt=0]
t=43494 [st=72]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=43494 [st=72] -REQUEST_ALIVE


All help would be greatly appreciated.

This issue has cropped up again with Privacy Badger. I'm concerned it will start impacting their users again the way it did late last year. 

Sorry I missed #40. Inspecting the logs, I concur this is predominantly due to server-side update throttling. Long-term, these requests should mark themselves as being foreground-priority if and only if the client is in ENFORE_STRICT mode. In the short-term, I will make a server-side adjustment to treat all such requests with foreground priority.

I will update this bug when the server-side change ships later this week.

The change has just finished rolling out globally and this should now be resolved for the majority of users.

(This will still be an issue in some locations due to OFAC restrictions; i.e. Cuba, Iran, North Korea, Sudan, Syria, and the Crimean contested region. This is unactionable for those users at this time.)

Please re-open if you are still seeing this issue, in the meantime I will confirm the fix using our server-side logs.

That list seems too long. For instance, there are clear OFAC exemptions in place for Iran http://www.steptoe.com/resources-detail-8103.html and I believe some of the other countries too. If there's any uncertainty about that, please have your legal team reach out to our colleague Nate Cardozo at EFF.

You're right, that list was too long. This will still be an issue in North Korea.