ERR_CONTENT_DECODING_FAILED while accessing mp3 on cloudfront
Reported by
glenjung...@gmail.com,
Jul 19 2017
|
||||||||||||||
Issue description
Chrome Version : 59.0.3071.115
OS Version: 6.1 (Windows 7, Windows Server 2008 R2)
URLs (if applicable) :
Other browsers tested:
Add OK or FAIL after other browsers where you have tested this issue:
Safari 5:
Firefox 4.x:
IE 7/8/9:
What steps will reproduce the problem?
1. Use Chrome build 59 to open this mp3 on cloudfront cdn: https://dhqbrvplips7x.cloudfront.net/crm-cic/2120/assets/sound/RingIn.mp3 (it fails and others likely will as well) (this worked, and has worked for a long time in builds prior to 59)
2. Open same file in incognito mode and it will work.
3. Open same file in incognito mode and disable caching via developer tools and this will fail.
What is the expected result?
In Chrome builds prior to 59, this file played just fine (has played for quite some time. File has not changed)
What happens instead of that?
Chrome appears to open the mp3, but stops. Console logs indicate ERROR_CONTENT_DECODING_FAILED
Please provide any additional information below. Attach a screenshot if
possible.
UserAgentString: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
My employer has a product that runs as a addin for Salesforce. Chrome is Salesforce's recommended browser, and thus ours when using Saleforce and our product. Since build 59 was released and our customers are slowly getting the upgrade, this mp3 file (perhaps others) will not play via Chrome. This file is important because our softphone product uses it to indicate an incoming phone call to the user.
I've tried playing this file on my home PCs (to eliminate some setting being added at work) and get similar/mixed results. My primary laptop running 64 bit Chrome on 64 bit Mint 17 fails to open the file, but my old Atom based 32bit (also Mint 17) will play the file fine.
Here is my ongoing conversation with a Chrome expert on the Chrome help forums:
https://productforums.google.com/forum/?utm_medium=email&utm_source=footer#!msg/chrome/KWk3C0OVSbI/rdypTF_cAQAJ
,
Jul 19 2017
I can add a video of this. This is happening in PURE Chrome. Our product does not even need to be involved. Our customers (again w/o even using our software product) can reproduce the same issue. It is not our product. This has worked 100% up until build 59 was released.
,
Jul 19 2017
I have also been able to reproduce this issue. From what I can tell looking at the chrome networking an internal events two requests were issued to play this file 1 download and 1 stream attempt. From what I have been able to gather the issue comes in when attempting to stream the content (range: 0-). Related events from chrome://net-internals/#events 629275: URL_REQUEST https://dhqbrvplips7x.cloudfront.net/crm-cic/2120/assets/sound/RingIn.mp3 Start Time: 2017-07-19 17:46:02.497 t=199963334 [st= 0] +REQUEST_ALIVE [dt=15] --> priority = "HIGHEST" --> url = "https://dhqbrvplips7x.cloudfront.net/crm-cic/2120/assets/sound/RingIn.mp3" t=199963334 [st= 0] DELEGATE_INFO [dt=0] --> delegate_blocked_by = "NavigationResourceThrottle" t=199963334 [st= 0] URL_REQUEST_DELEGATE [dt=0] t=199963334 [st= 0] URL_REQUEST_START_JOB [dt=0] --> load_flags = 37122 (BYPASS_CACHE | MAIN_FRAME_DEPRECATED | MAYBE_USER_GESTURE | VERIFY_EV_CERT) --> method = "GET" --> url = "https://dhqbrvplips7x.cloudfront.net/crm-cic/2120/assets/sound/RingIn.mp3" t=199963334 [st= 0] +URL_REQUEST_START_JOB [dt=13] --> load_flags = 37122 (BYPASS_CACHE | MAIN_FRAME_DEPRECATED | MAYBE_USER_GESTURE | VERIFY_EV_CERT) --> method = "GET" --> url = "https://dhqbrvplips7x.cloudfront.net/crm-cic/2120/assets/sound/RingIn.mp3" t=199963335 [st= 1] URL_REQUEST_DELEGATE [dt=0] t=199963335 [st= 1] HTTP_CACHE_GET_BACKEND [dt=0] t=199963335 [st= 1] HTTP_CACHE_DOOM_ENTRY [dt=0] t=199963335 [st= 1] HTTP_CACHE_CREATE_ENTRY [dt=0] t=199963335 [st= 1] HTTP_CACHE_ADD_TO_ENTRY [dt=0] t=199963335 [st= 1] +HTTP_STREAM_REQUEST [dt=1] t=199963335 [st= 1] HTTP_STREAM_JOB_CONTROLLER_BOUND --> source_dependency = 629279 (HTTP_STREAM_JOB_CONTROLLER) t=199963336 [st= 2] HTTP_STREAM_REQUEST_BOUND_TO_JOB --> source_dependency = 629280 (HTTP_STREAM_JOB) t=199963336 [st= 2] -HTTP_STREAM_REQUEST t=199963336 [st= 2] +HTTP_TRANSACTION_SEND_REQUEST [dt=0] t=199963336 [st= 2] HTTP_TRANSACTION_SEND_REQUEST_HEADERS --> GET /crm-cic/2120/assets/sound/RingIn.mp3 HTTP/1.1 Host: dhqbrvplips7x.cloudfront.net Connection: keep-alive Pragma: no-cache Cache-Control: no-cache Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8 Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.8 t=199963336 [st= 2] -HTTP_TRANSACTION_SEND_REQUEST t=199963336 [st= 2] +HTTP_TRANSACTION_READ_HEADERS [dt=11] t=199963336 [st= 2] HTTP_STREAM_PARSER_READ_HEADERS [dt=11] t=199963347 [st=13] HTTP_TRANSACTION_READ_RESPONSE_HEADERS --> HTTP/1.1 200 OK Content-Type: audio/mpeg Content-Length: 13474 Connection: keep-alive Date: Mon, 17 Jul 2017 11:07:34 GMT x-amz-replication-status: COMPLETED Last-Modified: Mon, 26 Jun 2017 11:57:28 GMT ETag: "9f71779e9ca6826b68b1badc85cc2793" Cache-Control: max-age=31556926 Content-Encoding: gzip x-amz-version-id: rQ.1P0gdYAdaWXoQLvTcCt3DBJ.fQxpM Accept-Ranges: bytes Server: AmazonS3 Age: 211109 X-Cache: Hit from cloudfront Via: 1.1 b7fec677fffcdd47588ce3a6a9e6d5e1.cloudfront.net (CloudFront) X-Amz-Cf-Id: u2659Ys81C94WakvJjkWgXEc8eah_P4jIbgNEka1LiAb5AoMs7lWuw== t=199963347 [st=13] -HTTP_TRANSACTION_READ_HEADERS t=199963347 [st=13] HTTP_CACHE_WRITE_INFO [dt=0] t=199963347 [st=13] HTTP_CACHE_WRITE_DATA [dt=0] t=199963347 [st=13] HTTP_CACHE_WRITE_INFO [dt=0] t=199963347 [st=13] URL_REQUEST_DELEGATE [dt=0] t=199963347 [st=13] URL_REQUEST_FILTERS_SET --> filters = "GZIP" t=199963347 [st=13] -URL_REQUEST_START_JOB t=199963347 [st=13] +URL_REQUEST_DELEGATE [dt=2] t=199963348 [st=14] DELEGATE_INFO [dt=0] --> delegate_blocked_by = "NavigationResourceThrottle" t=199963349 [st=15] -URL_REQUEST_DELEGATE t=199963349 [st=15] HTTP_TRANSACTION_READ_BODY [dt=0] t=199963349 [st=15] HTTP_CACHE_WRITE_DATA [dt=0] t=199963349 [st=15] URL_REQUEST_JOB_BYTES_READ --> byte_count = 3508 t=199963349 [st=15] URL_REQUEST_JOB_FILTERED_BYTES_READ --> byte_count = 3747 t=199963349 [st=15] HTTP_TRANSACTION_READ_BODY [dt=0] t=199963349 [st=15] HTTP_CACHE_WRITE_DATA [dt=0] t=199963349 [st=15] URL_REQUEST_JOB_BYTES_READ --> byte_count = 9966 t=199963349 [st=15] URL_REQUEST_JOB_FILTERED_BYTES_READ --> byte_count = 32768 t=199963349 [st=15] URL_REQUEST_JOB_FILTERED_BYTES_READ --> byte_count = 13725 t=199963349 [st=15] HTTP_TRANSACTION_READ_BODY [dt=0] t=199963349 [st=15] HTTP_CACHE_WRITE_DATA [dt=0] t=199963349 [st=15] -REQUEST_ALIVE 629289: URL_REQUEST https://dhqbrvplips7x.cloudfront.net/crm-cic/2120/assets/sound/RingIn.mp3 Start Time: 2017-07-19 17:46:02.623 t=199963460 [st= 0] +REQUEST_ALIVE [dt=13] --> priority = "LOWEST" --> url = "https://dhqbrvplips7x.cloudfront.net/crm-cic/2120/assets/sound/RingIn.mp3" t=199963460 [st= 0] URL_REQUEST_DELEGATE [dt=0] t=199963460 [st= 0] +URL_REQUEST_START_JOB [dt=13] --> load_flags = 33026 (BYPASS_CACHE | MAYBE_USER_GESTURE | VERIFY_EV_CERT) --> method = "GET" --> url = "https://dhqbrvplips7x.cloudfront.net/crm-cic/2120/assets/sound/RingIn.mp3" t=199963460 [st= 0] URL_REQUEST_DELEGATE [dt=0] t=199963460 [st= 0] HTTP_CACHE_CALLER_REQUEST_HEADERS --> X-DevTools-Emulate-Network-Conditions-Client-Id: 499ef854-a36f-44fb-88f2-633a8a9683c4 Accept-Encoding: identity;q=1, *;q=0 X-DevTools-Request-Id: 16632.2 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36 Range: bytes=0- Accept: */* Referer: https://dhqbrvplips7x.cloudfront.net/crm-cic/2120/assets/sound/RingIn.mp3 Accept-Language: en-US,en;q=0.8 --> line = "" t=199963460 [st= 0] HTTP_CACHE_GET_BACKEND [dt=0] t=199963460 [st= 0] HTTP_CACHE_DOOM_ENTRY [dt=1] --> net_error = -2 (ERR_FAILED) t=199963461 [st= 1] HTTP_CACHE_CREATE_ENTRY [dt=0] t=199963461 [st= 1] HTTP_CACHE_ADD_TO_ENTRY [dt=0] t=199963461 [st= 1] +HTTP_STREAM_REQUEST [dt=0] t=199963461 [st= 1] HTTP_STREAM_JOB_CONTROLLER_BOUND --> source_dependency = 629292 (HTTP_STREAM_JOB_CONTROLLER) t=199963461 [st= 1] HTTP_STREAM_REQUEST_BOUND_TO_JOB --> source_dependency = 629293 (HTTP_STREAM_JOB) t=199963461 [st= 1] -HTTP_STREAM_REQUEST t=199963461 [st= 1] +HTTP_TRANSACTION_SEND_REQUEST [dt=0] t=199963461 [st= 1] HTTP_TRANSACTION_SEND_REQUEST_HEADERS --> GET /crm-cic/2120/assets/sound/RingIn.mp3 HTTP/1.1 Host: dhqbrvplips7x.cloudfront.net Connection: keep-alive Pragma: no-cache Cache-Control: no-cache Accept-Encoding: identity;q=1, *;q=0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36 Accept: */* Referer: https://dhqbrvplips7x.cloudfront.net/crm-cic/2120/assets/sound/RingIn.mp3 Accept-Language: en-US,en;q=0.8 Range: bytes=0- t=199963461 [st= 1] -HTTP_TRANSACTION_SEND_REQUEST t=199963461 [st= 1] +HTTP_TRANSACTION_READ_HEADERS [dt=12] t=199963461 [st= 1] HTTP_STREAM_PARSER_READ_HEADERS [dt=12] t=199963473 [st=13] -HTTP_TRANSACTION_READ_HEADERS --> net_error = -330 (ERR_CONTENT_DECODING_FAILED) t=199963473 [st=13] -URL_REQUEST_START_JOB --> net_error = -330 (ERR_CONTENT_DECODING_FAILED) t=199963473 [st=13] URL_REQUEST_DELEGATE [dt=0] t=199963473 [st=13] -REQUEST_ALIVE --> net_error = -330 (ERR_CONTENT_DECODING_FAILED)
,
Jul 19 2017
I am the TC of the Chrome forum thread above. I was able to reproduce this on Chrome Stable 59.0.3071.115 and Chrome Canary 61.0.3161.0 on Windows 10 x64. They worked under incognito mode.
,
Jul 20 2017
This issue appears to stem from the fact that this audio file is zlib compressed at level 9. From what I can tell chrome is having an issue with the decompression of this mp3 audio file. I was able to reproduce this issue by hosting this same audio file zlib compress with the (Content-Encoding: gzip) in an S3 bucket. Another interesting observation is playing this same file using html5 audio and video does not work, but when attempting to play this audio through a third party js library (howler js) it plays just fine. I am assuming it has something to do with the fact that the audio is not immediately streamed in that case.
,
Jul 21 2017
Able to reproduce the issue on the latest canary(61.0.3163.0), stable(59.0.3071.115) on Windows-10, Mac OS 10.12.5 and Linux Ubuntu 14.04. However this shows very inconsistent behavior on the same chrome version. Since this is inconsistent hence unable to provide the tool bisect result here. Marking this as Untriaged and adding proper component for someone from the team to have a look into this and help in further debugging.
,
Jul 21 2017
Could someone who is reproducing this collect a netlog and attach it to this issue? See http://dev.chromium.org/for-testers/providing-network-details. A couple of notes pending that, based on c#4: The second request is being sent with 'Accept-Encoding: identity;q=1, *;q=0', but is resulting in a ERR_CONTENT_DECODING_FAILED. Though the response headers aren't recorded, this suggests that the response isn't being sent respecting that cache control header, i.e. the response is being sent encoded. We recently made a change to Chrome to reject Content-Encoding that violate sent Accept-Encodings; that could be resulting in this problem. Of course, if cloudfront is ignoring Accept-Encoding headers, that could be a very large problem. -> Internals>Network so that this is on the triager's radar screen, +eustas FHI, Pri 1 for the possibility a major CDN is ignoring Accept-Encoding.
,
Jul 21 2017
Yup, fetching entry with curl confirms that CDN ignores Accept-Encoding request. What will happen, if we, at last, remove "A-E" override in the media library?
,
Jul 21 2017
@eustas: I think everything would Just Work, though it would make doing anything funky with range requests and the cache harder (though since we don't have any plans to do anything funky with range requests and the cache, I probably shouldn't worry about that :-}). But I'm concerned in the abstract that Cloudfront is ignoring Accept-Encoding constraints. Not sure what to do about it, though (other than to dig in and figure out why we haven't been seeing this in our UMA, which is on my plate for today).
,
Jul 21 2017
Can't check the effect of it, my default Chromium build downloads media file instead of playing it. I guess that if there is no way clients can get raw data (bypassing filters), then removing A-E override won't break media player.
,
Jul 21 2017
Is that a change you're thinking it's reasonable to do for other reasons? I've flip-flopped on whether I think it's reasonable that the media player specified the Accept-Encoding that it does, but I don't think it makes sense to change the media player to solve this problem alone.
,
Jul 21 2017
We should at least update the logging code to log the received headers in this case, to make the failure more visible. I think maybe the logging was moved up to the cache layer, breaking logging when we decide at lower layers we don't like the headers?
,
Jul 21 2017
I don't remember the logging being moved; I presumed the logging problem was that we were returning an error from the HttpNetworkTransaction that we haven't before (ERR_CONTENT_DECODING_FAILED used to be returned only from the filter level, and now HttpNetworkTransaction::DoReadHeadersComplete can return it. Having said that, I've finally figured out what's broken about the UMA that we're not seeing any signs of this problem in it (though not the path through which it was broken): Histogram name mismatch. Fix coming.
,
Jul 21 2017
I've submitted https://chromium-review.googlesource.com/c/581507/ for review. After that lands, we should start getting real data within a few days on how often this is happening. My current thought is to wait to decide on any Chromium changes until after we have that data. Arguably, though, we should reach out to Cloudfront in parallel. I'll see if I can scare up a contact with them. Also possibly in parallel we should add logging code to log the headers. Matt, I'm willing to do the grunt work for that, but I'd value your opinion as to how to do it--just log the headers in this particular failure case, or try and figure out if we should just move header logging down to this level? I'm inclined to think just in the error case; if we log here, arguably we should kill header logging at higher levels, which probably means we won't see headers when they come from the cache (though I actually thought we *didn't* see headers when they come from the case, so maybe that's a red herring?).
,
Jul 21 2017
At least for this moment, this bug's pretty solidly mine. I'll put Internals>Network back on if I drop it on the floor.
,
Jul 22 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/74b18cd26d60a7379713ad0c1bce3444f9c7fd8e commit 74b18cd26d60a7379713ad0c1bce3444f9c7fd8e Author: Randy Smith <rdsmith@chromium.org> Date: Sat Jul 22 05:19:10 2017 Bring code and histograms.xml back into agreement. https://codereview.chromium.org/2753453003 changed the histogram named referred to in the code from Net.ContentDecodingFailed2.FilterType to just Net.ContentDecodingFailed2, but didn't update histograms.xml. This CL updates histograms.xml and marks as obsolete histograms related to that one. Bug: 746421 Change-Id: I7401e8516a2a00dc255ccb2bdef171dc30f21387 Reviewed-on: https://chromium-review.googlesource.com/581507 Commit-Queue: Randy Smith <rdsmith@chromium.org> Reviewed-by: Robert Kaplow <rkaplow@chromium.org> Cr-Commit-Position: refs/heads/master@{#488856} [modify] https://crrev.com/74b18cd26d60a7379713ad0c1bce3444f9c7fd8e/tools/metrics/histograms/enums.xml [modify] https://crrev.com/74b18cd26d60a7379713ad0c1bce3444f9c7fd8e/tools/metrics/histograms/histograms.xml
,
Jul 25 2017
The results from the above UMA change have come back, and the answer is that requests in number of about 0.02% of web page loads are being rejected. That's low enough that I'm inclined to say "let's go after the servers to be spec compliant" rather than "Chrome should adapt to the lack of spec compliance". I've tossed a note about this to net-dev@ (https://groups.google.com/a/chromium.org/forum/#!topic/net-dev/GeXCJifwJUY), and am trying to get someone from Cloudfront to take a look at this bug. (rsleevi@, now cc'd, has been pinging his contacts). As such I'm going to take myself off this bug and hand it over to the triager to drive as needed. As such, I'm going to take myself off the bug
,
Jul 28 2017
,
Sep 6 2017
I reproduced error. https://web.opendrive.com/api/v1/download/file.json/122136872_9GiUP?inline=1 Chrome 58 -plays this file Chrome 60 - fails with error ERR_CONTENT_DECODING_FAILED Firefox 55 - plays It seems the chrome is unable to decompress files that compressed with deflate algorithm (1st bytes 0x78 0x9c and latest CRC 4bytes are removed when this file is streamed) I attached original file
,
Sep 6 2017
Re #22: This doesn't look like a problem with deflate. The ERR_CONTENT_DECODING_FAILED is emitted because Chrome advertised "Accept-Encoding: identity;q=1, *;q=0" but server sent back an encoded response. Randy: has there been any followup to log the received headers on error? I used M62 Dev. We are not yet logging received headers for this error case.
,
Sep 8 2017
Sorry, Helen, I've been snowed and this bug hasn't made it to near the top of my priority list for a while. It's still on my plate, and I suspect I'll get to it within the next week or two, but if we need faster response than that we may need to find someone else to work on it (or reprioritize other stuff on my plate, but I don't think that's going to happen--several crashers and a sheriff rotation).
,
Sep 8 2017
I don't see anything obviously wrong with the headers returned from CloudFront. What I don't understand is why decoding/playback *works* in an incognito window, but not in a regular window. What's different about incognito? I used to work on CloudFront so let me know if you're still looking for a contact.
,
Sep 8 2017
maniscalco: I have no idea about the incognito, but the fact is we're telling them we don't support gzip (We don't support gzipped range responses, so on some media requests, we claim not to support gzip), and they're apparently sending a gzip response. The logic not to accept unadvertised encodings was added for brotli - servers were sniffing for browser instead of accept-encoding, which has implications for compatibility with other browser, and prevents us from disabling brotli in the future (For field trials, to save RAM, or, as we're currently doing, disabling it over HTTP connections due to broken middleboxes).
,
Sep 8 2017
mmenke: I see, thanks. The request I was looking at in devtools was for the full object (i.e. not a Range request). I understand the problem now. For other folks reading this bug, this curl command illustrates the behavior: $ curl -v https://dhqbrvplips7x.cloudfront.net/crm-cic/2120/assets/sound/RingIn.mp3 -H'Accept-Encoding:identity;q=1, gzip;q=0' -H'Range:bytes=0-' > /dev/null (some lines omitted) > GET /crm-cic/2120/assets/sound/RingIn.mp3 HTTP/1.1 > User-Agent: curl/7.35.0 > Host: dhqbrvplips7x.cloudfront.net > Accept: */* > Accept-Encoding:identity;q=1, gzip;q=0 > Range:bytes=0- > < HTTP/1.1 206 Partial Content < Content-Type: audio/mpeg < Content-Length: 13474 < Connection: keep-alive < Date: Fri, 08 Sep 2017 16:07:25 GMT < x-amz-replication-status: COMPLETED < Last-Modified: Mon, 26 Jun 2017 11:57:28 GMT < ETag: "9f71779e9ca6826b68b1badc85cc2793" < Cache-Control: max-age=31556926 < Content-Encoding: gzip < x-amz-version-id: rQ.1P0gdYAdaWXoQLvTcCt3DBJ.fQxpM < Accept-Ranges: bytes * Server AmazonS3 is not blacklisted < Server: AmazonS3 < Content-Range: bytes 0-13473/13474 < Age: 3495 < X-Cache: Hit from cloudfront < Via: 1.1 28fd9501206a11a06951e60ad8a996fd.cloudfront.net (CloudFront) < X-Amz-Cf-Id: FqVaWS158jZNAwOs98zg70MuNzctuUmL4-7s2jGGgAbyKfEeLQtBLA== I'll talk with someone who works on CloudFront.
,
Sep 8 2017
It may be that cloud front respects the header, but at some point we sent the same request allowing gzipped responses, and are using the cached response (Which would imply all compresseed responses should perhaps have a varies: accept-encoding header).
,
Sep 8 2017
Sorry, missed that comment 27 was from CURL. So not a cache issue. Also generally, 206 responses + content-encoding is also a bad idea. I don't think any browser supports partial responses with content-encodings.
,
Nov 13 2017
FYI I'm seeing a similar issue with Yahoo's servers ignoring the "Accept-Encoding" value when the "User-Agent" value specifies Chrome. I've filed issue #784601 for that.
,
Feb 16 2018
,
Mar 28 2018
Closing, since it's a server issue.
,
Mar 29 2018
I don't agree to close. Please check https://bugs.chromium.org/p/chromium/issues/detail?id=746421#c22 In firefox link is playable, but chrome wouldn't to play file
,
Mar 29 2018
We're telling the server we don't accept gzipped responses. The server sends a gzipped response. That's pretty clearly broken.
,
Mar 29 2018
While I can agree that the server side should behave correctly, I think the overall user experience should also be kept in mind here. Unfortunately, all the user sees is that functionality is broken that works fine in other browsers. In almost all cases, telling the user to go get the server fixed is not going to result in anything positive. |
||||||||||||||
►
Sign in to add a comment |
||||||||||||||
Comment 1 by mef@chromium.org
, Jul 19 2017