New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 5 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 2
Type: Bug



Sign in to add a comment

Extension update leaving extension with only HTML and CSS files

Reported by erik.rot...@gmail.com, Nov 28 2012 Back to list

Issue description

UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.91 Safari/537.11

Steps to reproduce the problem:
1. Use Chrome extension as normal
2. New update is released
3. Suddenly, non-functioning extension, without any JavaScript files or images in file bundle

What is the expected behavior?
Extension should work as normal, JavaScript files and images should not be missing

What went wrong?
A lot of users have had their extension suddenly stop working, usually after I release new updates to it. The symptoms are the same: The users start seeing the error HTML in my popup, which will appear if a JavaScript error occurs on load of the popup.

I ask the user to send a screenshot of their error console. I have attached an example of that. I have been trying to get the users to send the extension bundle they have on disk, and finally got one user to do this. It appears that all the JavaScript files are missing, as well as images. I have attached the bundle that is broken, and a reference bundle, the way it should be.

It appears that all JavaScript files are gone, but also all folders that only contain JavaScript files. No image files are present either. The HTML files and CSS files appear to be intakt.

As this does not happen to all users, I have no real idea how to reproduce it.

WebStore page: http://chrome.google.com/webstore/detail/pnjaodmkngahhkoihejjehlcdlnohgmp?hl=en

Did this work before? N/A 

Chrome version: 23.0.1271.91  Channel: stable
OS Version: Windows 7
 
FaultyExtension.zip
15.2 KB Download
The error messages.png
257 KB View Download
ReferenceExtension.zip
1.1 MB Download

Comment 1 by kalman@chromium.org, Nov 28 2012

Cc: kalman@chromium.org mpcomplete@chromium.org
Do you know whether this correlates to the versions of Chrome that your users are running?

Matt, any idea?
Cc: asargent@chromium.org
No idea... Antony?
Cc: mek@chromium.org keiger@chromium.org
+cc keiger, mek

I haven't heard of this problem before, but I can think of a few possible causes:

1) Maybe local antivirus software is interfering with the unpack/install process? It would be interesting to see if all those users happened to run the same AV software.

2) The webstore could have generated a broken .crx file that's missing these files. However, if this were the case, I'd sort of expect this to affect both all new installs and everyone getting the update, not just some users.

3) We might have introduced some bug in our unpacking/installing code.


Comment 4 by keiger@chromium.org, Nov 28 2012

I checked the published version on the Web Store, and the CRX looks good. I installed it successfully, so I doubt the problem lies with the downloaded archive. Still, I'll keep watching this issue.
They always have the latest version. I'm not sure if I can check if they recently upgraded? The first time I got an e-mail about this, that I know was this very issue, was in July. So it has been going on for quite some time. I have asked a couple of users which antivirus they have, will update with more information.
One of the users had AVG Anti-Virus. That same user sent me his broken bundle, and it appears to have images in it, still no JS files though. I have attached it here. It's almost as if it's just making a partial download. I don't know if extension updates are differential, maybe it's not merging the files correctly to the new directory?


FaultyExtension2WithImages.zip
961 KB Download

Comment 7 by kalman@chromium.org, Nov 29 2012

Cc: jamescook@chromium.org joaodasilva@chromium.org
I think we validate the crx files when they are downloaded to ensure that they're not corrupted? That would imply a problem in the unpacking process, so either (1) or (3) in asargent's list.

We have made some changes to the unpacking code btw, I saw a change go past e.g. https://chromiumcodereview.appspot.com/11198067 as one example. Author CC'ed (and joaodasilva for good measure...).

How many users are affected? What percentage is this of your users?
I'm not familiar with this code, but while fixing a regression introduced by the temp-dir-in-extensions-dir CL I noticed that sandboxed_unpacker.cc relies on unpacker.cc to convert all images and catalogs. The bug I saw was that the converted images were dumped, but once the utility process was done the main process couldn't read the converted images back because they were garbage collected by GarbageCollectExtensions. Maybe this is the same bug? Or something else is messing with the temp-dir while a CrxInstaller is in progress?

FWIW the fix was https://codereview.chromium.org/11411227/. It could be merged back to M23 if it also solves this issue.

Comment 9 by kalman@chromium.org, Nov 30 2012

jamescook: ping
There doesn't seem to be a correlation between antiviruses, since everyone I asked have different antivirus software. One thing that I'm noticing though, is that their Chrome version always appear to be multi-version, or have the "m" in the version string. Like so:

23.0.1271.91 (Official Build 167501) m

Could this be something?
I asked one of our release managers about this "m" suffix on the user agent, and it sounds like it probably isn't relevant. The "m" stands for the "multi user installer" which the vast majority of users get when installing chrome these days (as opposed to the single user only installer, which installs the binaries just into the user's OS profile directory instead of a shared location). 
Is there anything else I can check? I have a lot of examples of extension directories now that I've checked through, and they all have stopped the download at different places and contain a different set of files. Would it help if I uploaded some more examples?
Is there anything else I can check? I have a lot of examples of extension directories now that I've checked through, and they all have stopped the download at different places and contain a different set of files. Would it help if I uploaded some more examples?
Sorry, just saw this. This could be the issue joaodasilva mentions in #8.  It will be hard to prove that, since the symptom would be a Temp directory that was unexpectedly empty during an unpack, which will be hard to catch.  I would prefer to keep my change in place, as it makes a big difference for us on Chrome OS, and slow extension installs at shutdown are a major cause of shutdown hangs/crashes for us.

It looks like the fix in #8 missed the M24 branch point, so I think it would be worth backporting to M24.  I can handle that if you like, since I caused that regression.

@Erik.rothoff - have you already collected the chrome version numbers from the reports you've gotten about this? It would be useful to know if this has happened to anyone running Chrome 25 dev or canary channel.
It appears that it only occurs in Chrome 23.0.1271.95 and other 23.0.1271.* variants. If I get any higher versions than that, I'll report back.
Labels: Action-FeedbackNeeded
Status: Available
I have still been getting error reports about this in Chrome 24, and now recently got one of the same about Chrome 25.

The attached screenshot shows a user with the chrome://version 
25.0.1364.97 (Official Build 183676) m
Screen Shot 2013-02-27 at 2.02.27 AM.png
370 KB View Download
Project Member

Comment 19 by bugdroid1@chromium.org, Mar 9 2013

Labels: -Action-FeedbackNeeded Needs-Feedback
Project Member

Comment 20 by bugdroid1@chromium.org, Mar 10 2013

Labels: -Feature-Extensions Cr-Platform-Extensions

Comment 21 by j...@penguin.ws, Nov 8 2014

Hi everyone,

I also experienced this issue in two occasions. I have published my own extension called 'startuptabs' and the extension is installed on 2k+ users and works fine for the most part. However, a few users and I've actually experienced this myself as well, run into an issue where a certain file in the package is missing upon installation.

When this issue occured for me, there was no going around it. I've tried reinstalling multiple times without any luck. Release a new version also did not solve the problem. I initially thought my package may have been too large but even after cutting down the size, I am experiencing this problem. 

One thing common when this occured is that the issue was reproduced on brand new systems. I have a chrome book and a new ubuntu install. The extensions were installed while I logged into sync my bookmark/pw/extensions to the new machine. Upon installation, I noticed that my 'startuptabs' was throwing javascript errors due to the some js files being missing.
Is there a pattern? I notice that in your extension there are a lot of unusual files like .less files.
Cc: -joaodasilva@chromium.org
Good day. I've got same problem this days. Any ideas about what is it and why it's happened?
Any chance you did a shutdown or restart around the time the extension was installed? If so this might be the problem of not flushing all files to disk, a fix for which is being worked on for bug 463987

Hi. I can't view this bug. If i correctly understand - problem is in how NTFS works. So, why only *.js files dissapear? Nor html or png, only js. We had about 1 mln. users and about 30 000 of them catch this problem. Looks like not problem with flushing. May be here be a problem with a lot of unpacking action at the same time? (bug appear on first run after installing, when pack of extensions install simultaneously)
Sorry about the referenced bug being restricted - I didn't realize it was and have asked if we can open it up since there doesn't seem to be anything confidential there. You can see the CL that has already landed for it at:

https://codereview.chromium.org/998663002

The status is that we have an experiment we are running that does extra work to try and flush file writes and a directory move to disk immediately upon install. 

It is very interesting though that its only .js files that you've noticed disappearing, and that a large number of users observed the same problem. Were the .js files in question all content scripts (eg files that are referenced in the "content_scripts" { "js" : [...] } section of your manifest.json?

Another possibility is that this problem is due to antivirus software interfering with the writes to disk. The way extension install works is:

1. Unzip .crx file in a temporary directory using a sandboxed helper process
2. Helper process reads manifest.json, messages.json files for i18n, content scripts, and images used in browser process (any of the "icons" referenced in the manifest, browser action icons, etc.), and sends binary serializations of those objects representing that content over to the non-sandboxed browser process
3. The regular browser process overwrites those files in the temp directory with versions created with our own json/image libraries. 
4. Move the temp directory into the profile dir

What we've sometimes seen in the past is that the file writes during step 3 can sometimes  be hung up by antivirus software scanning them (even though the OS has reported that the writes succeeded), and the move in step 4 will fail due to the files being still opened. But usually that causes the entire install to fail, not just some files to go missing. 

If you have contact with some users that can reproduce the problem, you might ask them what AV software they are running and see if they are all running the same things, and have them temporarily turn it off and see if that makes the installs succeed. 





Good day.

About the problem: we detect that one of Antivirus software silently remove our *.js files from extension (this was false positive). But in some cases files was remove without antivirus. Does Chromium project planning to create somekind of checking extension files integrity and automatically reinstall on problem?
Cc: -jamescook@chromium.org
Cc: -mpcomplete@chromium.org
Project Member

Comment 31 by sheriffbot@chromium.org, Mar 29 2017

Labels: Hotlist-Recharge-Cold
Status: Untriaged
This issue has been available for more than 365 days, and should be re-evaluated. Please re-triage this issue.
The Hotlist-Recharge-Cold label is applied for tracking purposes, and should not be removed after re-triaging the issue.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Cc: -kalman@chromium.org -asargent@chromium.org rdevlin....@chromium.org
Owner: lazyboy@chromium.org
Status: Assigned
There's only so much that chrome can do to *prevent* this from happening.  If antivirus (or any other program on the machine) decides to eliminate the files from the extension, chrome can't stop it (they have equal privilege).

That said, lazyboy@ has been doing some great work on content verification that should help us to *detect* this case, and allow users to reinstall the extension when this corruption is detected.  Note that this still might not fix everything perfectly; there's a good chance that if it's a program on the machine removing the files, it would do it again, but at least it wouldn't be as silent.

lazyboy@, should this be merged into any of the content verification bugs?

Sign in to add a comment