Hi, thank you for reporting this.

tldr; are the computers allowed to download and update Chrome and Chrome components? If the answer is 'no' because updating Chrome is not desired, then let's discuss what is the best way to achieve it.

I understand this happens in the context of a managed computer network for an organization. The output of running bitadmin shows two Chrome component updater jobs and one Google Update job corresponding to a Chrome delta update from M66 to M67.

To begin with, is BITS allowed to download the files Chrome and Google Update are trying to download. It looks like we have two jobs in "Connecting" state, and one job in "Queued" state. For the moment, I can't explain the later, but having the jobs in the "Connecting" state may indicate that BITS can't reach the Google backend. Is this intentional?

If the BITS queue is cleared, then new jobs may be created in the future, because updates are available, and update payloads must be downloaded. This part works as intended.

Second, Chrome and Google Update are using BITS as a service. Our code does not do anything to interfere with the execution of other BITS jobs, the service itself, or make bitsadmin hang.

Tested the issue on chrome reported version 67.0.3396.62 using Windows-10 with steps mentioned below":
1) Launched chrome reported version and waited for around 6 minutes
2) Opened CMD with administrator and given "bitsadmin /list /allusers"
3) It shows output as shown below:
BITSADMIN version 3.0
BITS administration utility.
(C) Copyright Microsoft Corp.

{4CB52538-A213-4650-85F3-F66782BD8141} '29.0.0.171_win64_PepperFlashPlayer.crx3' TRANSFERRED 1 / 1 13038611 / 13038611
{9CC03793-0664-4D71-802A-71717F7714EC} '29.0.0.171_win64_PepperFlashPlayer.crx3' TRANSFERRED 1 / 1 13038611 / 13038611
{C224F961-C57C-4A77-9A94-6FE09AAA4CA4} 'Chrome Component Updater' TRANSIENT_ERROR 0 / 1 0 / 352250
Listed 3 job(s).

@Reporter: Please find the attached screencast for your reference and provide your feedback on it which in further triaging it.

Thanks!

Deleted: 847818.mp4 4.2 MB

	847818.mp4 4.2 MB View Download

Thanks for the comments.  I suspect the Chrome Component Updater may be stuck in connecting status due to URL filtering on our firewall, though our settings are rather conservative.  Is there a list available containing the URLs that Chrome uses to update itself and its' associated components?  I would like to whitelist these URLs and test further.  Thanks.

Yes, the list is maintained at https://support.google.com/chrome/a/answer/7334577?hl=en

The current list is:

www.google.com/dl/*
dl.google.com/*
google.com/dl/*
*.gvt1.com
tools.google.com/service/update2
clients2.google.com
update.googleapis.com/service/update2

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

@Reporter: Could you please let us know if this issue is still reproducible? If so could you please have a look at comment#8 and let us know if this is helpful.

Thanks!

Thank you for your time.  I have added all of the URLs to our URL filter as safe, and after clearing out the BITS queue, the issue appears again.

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

BITS queue now contains Three instances of "Chrome Component Updater" with one in "Connecting" status and the other two in "queued" status for the past two days.

For #13, would it be possible to do the following:
1. Please paste here the output of BITS admin command, just like the original post did.
2. Please link a screenshot with the content of chrome://components on the system that exhibits this behavior.
3. Please try to manually download the URL that BITS is stuck in connecting state using BITS admin to see if the content at that URL can in fact be downloaded through the URL filter.

Basically, we are trying to find out of the component updater can update its Chrome components at all, and if there is a programming problem with the code that is using BITS in Chrome.

Thank you!

Good day, I had to forcibly kill the stuck jobs as it was interfering with my daily functions.  I did find a BITS queue error this morning though, so below is the verbose output from that error.  It should also be noted that I was able to manually download the link specified in the output of the error.  Since the error seems to point to the local destination on my machine, I have since deleted the folder in question and will monitor the BITS queue for when the issue pops up again

BITSADMIN version 3.0
BITS administration utility.
(C) Copyright Microsoft Corp.

GUID: {5437207E-D74B-4EA0-95DD-8F8E019C5E8D} DISPLAY: 'Chrome Component Updater'
TYPE: DOWNLOAD STATE: TRANSIENT_ERROR OWNER: HAASFORMULA\jcatino
PRIORITY: NORMAL FILES: 0 / 1 BYTES: 0 / 14496
CREATION TIME: 6/18/2018 7:56:11 AM MODIFICATION TIME: 6/18/2018 8:02:25 AM
COMPLETION TIME: UNKNOWN ACL FLAGS: 
NOTIFY INTERFACE: UNREGISTERED NOTIFICATION FLAGS: 3
RETRY DELAY: 60 NO PROGRESS TIMEOUT: 86400 ERROR COUNT: 42
PROXY USAGE: PRECONFIG PROXY LIST: NULL PROXY BYPASS LIST: NULL
ERROR FILE:    http://redirector.gvt1.com/edgedl/release2/chrome_component/Z-62ajQ9JHE_4525/4525_all_crl-set-6833967478491770133.data.crx3 -> C:\Users\jcatino\AppData\Local\Temp\chrome_BITS_9144_25431\4525_all_crl-set-6833967478491770133.data.crx3
ERROR CODE:    0x80200024 - The job is not making progress.  The server may be misconfigured.  Background Intelligent Transfer Service (BITS) will try again later.

ERROR CONTEXT: 0x00000004 - The error occurred while the local file was being processed. Verify that the file is not in use, and retry.

DESCRIPTION: 4525_all_crl-set-6833967478491770133.data.crx3
JOB FILES: 
	0 / 14496 WORKING http://redirector.gvt1.com/edgedl/release2/chrome_component/Z-62ajQ9JHE_4525/4525_all_crl-set-6833967478491770133.data.crx3 -> C:\Users\jcatino\AppData\Local\Temp\chrome_BITS_9144_25431\4525_all_crl-set-6833967478491770133.data.crx3
NOTIFICATION COMMAND LINE: none
owner MIC integrity level: MEDIUM
owner elevated ?           false

Peercaching flags
	 Enable download from peers      :false
	 Enable serving to peers         :false

CUSTOM HEADERS: NULL

Listed 1 job(s).

Well after clearing local files from temp folder used by the chrome updater job, After a day the same error above showed up again and it looks to be the same or very similar to the last error I posted.  Again I can download the link manually and when I go to chrome://components and click on each component they all get stuck "checking for status".  Below is the output from BITS and screenshot of components page attached.  Thanks.


BITSADMIN version 3.0
BITS administration utility.
(C) Copyright Microsoft Corp.

GUID: {55BC4C78-2D17-403D-873F-6C9B57944A36} DISPLAY: 'Chrome Component Updater'
TYPE: DOWNLOAD STATE: TRANSIENT_ERROR OWNER: HAASFORMULA\jcatino
PRIORITY: NORMAL FILES: 0 / 1 BYTES: 0 / 14510
CREATION TIME: 6/19/2018 8:40:14 AM MODIFICATION TIME: 6/19/2018 8:41:18 AM
COMPLETION TIME: UNKNOWN ACL FLAGS: 
NOTIFY INTERFACE: UNREGISTERED NOTIFICATION FLAGS: 3
RETRY DELAY: 60 NO PROGRESS TIMEOUT: 86400 ERROR COUNT: 12
PROXY USAGE: PRECONFIG PROXY LIST: NULL PROXY BYPASS LIST: NULL
ERROR FILE:    http://redirector.gvt1.com/edgedl/release2/chrome_component/XWmeUffllpM_4527/4527_all_crl-set-11009839644233083909.data.crx3 -> C:\Users\jcatino\AppData\Local\Temp\chrome_BITS_14344_27198\4527_all_crl-set-11009839644233083909.data.crx3
ERROR CODE:    0x80200024 - The job is not making progress.  The server may be misconfigured.  Background Intelligent Transfer Service (BITS) will try again later.

ERROR CONTEXT: 0x00000004 - The error occurred while the local file was being processed. Verify that the file is not in use, and retry.

DESCRIPTION: 4527_all_crl-set-11009839644233083909.data.crx3
JOB FILES: 
	0 / 14510 WORKING http://redirector.gvt1.com/edgedl/release2/chrome_component/XWmeUffllpM_4527/4527_all_crl-set-11009839644233083909.data.crx3 -> C:\Users\jcatino\AppData\Local\Temp\chrome_BITS_14344_27198\4527_all_crl-set-11009839644233083909.data.crx3
NOTIFICATION COMMAND LINE: none
owner MIC integrity level: MEDIUM
owner elevated ?           false

Peercaching flags
	 Enable download from peers      :false
	 Enable serving to peers         :false

CUSTOM HEADERS: NULL

Listed 1 job(s). 

Deleted: chrome biits.PNG 67.7 KB

	chrome biits.PNG 67.7 KB View Download

Thank you for doing this. When you said "Again I can download the link manually", did you mean that you had used bitsadmin to download it or was it the browser that you had used?

If you did not use bitsadmin, could you please try the following two commands from a non-admin command window, to download that CRX as if component updater had download it.

mkdir %LOCALAPPDATA%\Temp\chrome_BITS_14344_27198\

bitsadmin /transfer testcrx http://redirector.gvt1.com/edgedl/release2/chrome_component/XWmeUffllpM_4527/4527_all_crl-set-11009839644233083909.data.crx3 %LOCALAPPDATA%\Temp\chrome_BITS_14344_27198\4527_all_crl-set-11009839644233083909.data.crx3

I meant that I could paste the link in a browser and it would initiate a download.  I just tried the commands above and received another Transient error.  See attached screenshot.

Deleted: Chrome att.PNG 19.4 KB

	Chrome att.PNG 19.4 KB View Download

Thank you.

This is what we learned so far. The browser is able to to download the content but BITS itself is not. The bitsadmin tool itself encounters a similar error when attempting to download the update payload. 

A couple of ideas to further discuss.
* Is it possible that there is some local cybersecurity, antivirus, or content filtering  product, which may interfere with the BITS service trying to download this content?
The error from BITS indicates that the server might be misconfigured and/or there is a an error processing the local file (which is the CRX we want to download). We know the Google backend that is serving this content is not misconfigured but it could appear it is to this instance of BITS if there is some transparent proxy or some other type of enterprise content filtering solution in between BITS and Google. Also, cybersecurity products such as Bit9 are known to monitor the file system I/O, keep files in use, etc. That could be a reason why BITS thinks there is a sharing violation on the file it is trying to download.

* Could be please try to download the URL above using BITS and specify another destination for the download. There might be an issue with the the local temp directory, not sure, it could be worth trying.

* This issue could be further debugged by capturing network traces and/or using filemon to get a trace of file system I/O. I am not suggesting we do this now but mention it as a possible way to further understand what is going on.

In summary, Chrome won't be able to download its component updates in this configuration if the bitsadmin utility can't. There is something either on the local host or in between Google that is affecting the ability of BITS to download such content from Google, and it
results in creating such jobs in the BITS queue.

So I have done some more testing on my end and the issues persist.  I have tested on an outside network and interestingly had a similar transient error however this time it stated that the remote file was being processed.  I have performed a Wireshark capture while running the bitsadmin commands listed above, and have attached it here, along with a few screenshots from Procmon (filemon doesn't exist anymore) showing BITS actively interesting with the appdata\temp folder, and all seems to be well.  The only issue is there are some actions being taken on registry keys related to BITS which return "Name not found" however some searching reveals that this may not be a show-stopping issue.

Deleted: Chrome BITS.pcapng 121 KB

Chrome BITS.pcapng 121 KB Download

Deleted: filemon.PNG 79.3 KB

	filemon.PNG 79.3 KB View Download

Deleted: filemon2.PNG 47.3 KB

	filemon2.PNG 47.3 KB View Download

Deleted: filemon3.PNG 254 KB

	filemon3.PNG 254 KB View Download

I also forgot to mention that the machine I'm testing with has no third party AV installed and is relying on Windows Defender.  Additionally, Windows 10 BITS will not work with the firewall disabled, so even with explicitly allowing BITS traffic through Windows Firewall, the issue remains.

So I took the test job provided to me above and modified it to run the same test except downloading from Microsoft instead of Google, and it works without issue.  This would rule out a network issue or a BITS issue as we can see the mechanisms clearly work when we use an address other than Google.  For reference I used the following info

mkdir %LOCALAPPDATA%\Temp\chrome_BITS_OtherTest\
bitsadmin /transfer myDownloadJob /download /priority normal http://go.microsoft.com/fwlink/?LinkID=18922 %LOCALAPPDATA%\Temp\chrome_BITS_OtherTest\MSSecure.cab 

Thanks. Sorin and I have been investigating this in detail, it seems like the edge cache node BITS is striking is behaving incorrectly.

To test this hypothesis, let's try downloading directly from dl.google.com, bypassing edge cache:

mkdir %LOCALAPPDATA%\Temp\chrome_BITS_test\

bitsadmin /transfer testcrx http://dl.google.com/edgedl/release2/chrome_component/XWmeUffllpM_4527/4527_all_crl-set-11009839644233083909.data.crx3 %LOCALAPPDATA%\Temp\chrome_BITS_test\4527_all_crl-set-11009839644233083909.data.crx3

Thanks for the reply.  That example did the trick!  Downloaded instantly.

Thank you for sending us the trace. We found something in there which indicates that Google backend is responding with an error when such content is requested. We will look into this further.

HEAD /edgedl/release2/chrome_component/XWmeUffllpM_4527/4527_all_crl-set-11009839644233083909.data.crx3?cms_redirect=yes&mip=192.235.88.250&mm=28&mn=sn-ab5sznlk&ms=nvh&mt=1529592743&mv=m&pl=22&shardbypass=yes HTTP/1.1
Connection: Keep-Alive
Accept: */*
Accept-Encoding: identity
User-Agent: Microsoft BITS/7.8
Host: r4---sn-ab5sznlk.gvt1.com

HTTP/1.1 200 OK
Accept-Ranges:  none
Content-Length: 14510
Content-Type: application/octet-stream
Etag: "24a46f"
Server: downloads
Vary: *
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
Date: Thu, 21 Jun 2018 07:34:51 GMT
Alt-Svc: quic=":443"; ma=2592000; v="43,42,41,39,35"
Last-Modified: Tue, 19 Jun 2018 10:28:28 GMT
Connection: keep-alive

GET /edgedl/release2/chrome_component/XWmeUffllpM_4527/4527_all_crl-set-11009839644233083909.data.crx3?cms_redirect=yes&mip=192.235.88.250&mm=28&mn=sn-ab5sznlk&ms=nvh&mt=1529592743&mv=m&pl=22&shardbypass=yes HTTP/1.1
Connection: Keep-Alive
Accept: */*
Accept-Encoding: identity
If-Unmodified-Since: Tue, 19 Jun 2018 10:28:28 GMT
Range: bytes=0-1119
User-Agent: Microsoft BITS/7.8
Host: r4---sn-ab5sznlk.gvt1.com

HTTP/1.1 416 Requested Range Not Satisfiable
Content-Length: 0
Connection: close
Accept-Ranges: none

We are trying to understand more about the response "HTTP/1.1 416 Requested Range Not Satisfiable" given back to the client.

As a follow up to the comment #18, can we try an HTTPS download. Everything should stay the same, but please change the URL from http to https, like this:

bitsadmin /transfer testcrx https://redirector.gvt1.com/edgedl/release2/chrome_component/XWmeUffllpM_4527/4527_all_crl-set-11009839644233083909.data.crx3 %LOCALAPPDATA%\Temp\chrome_BITS_14344_27198\4527_all_crl-set-11009839644233083909.data.crx3

Thank you!

Thank you for the reply.  I just tried the link above with HTTPS and it successfully downloaded.  

Is there any agent (such as content filtering, anti-virus, cybersecurity solutions, etc) in between this machine and Google that can intercept the network request and deny range requests? We know that the client is receiving a "HTTP/1.1 416 Requested Range Not Satisfiable" but based on what we see so far, it is not the Google backend that is responding with 416. A third party entity might be transparently modifying the response BITS is getting.

In fact, assuming your computer's clock is correct, correlating the wireshark trace with our server logs I don't see evidence that the server is receiving the request at all - for example one of the HTTP 416s is sent at time 1529592964.809994 and a response received at 1529592964.860416, but there is no matching request in the server logs at that time.

Oops, hit "save changes" too early.

I know you already said no AV besides Windows Defender. Can you check your network connection's proxy settings to see if there's a system proxy?

I believe my colleague and I found the issue.  We adjusted a few settings on our firewall and our test case found in Comment 18 began to transfer and complete without issue.

Thank you! I am glad it worked out.