New issue
Advanced search Search tips

Issue 630680 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Jul 2016
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 3
Type: Feature



Sign in to add a comment

When using chunked encoding, each chunk size is limited to 2GB

Reported by romain.f...@gmail.com, Jul 22 2016

Issue description

UserAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36

Example URL:

Steps to reproduce the problem:
1. run the attached ./test_server.py
2. open http://127.0.0.1:8080/
3. the download will fail

What is the expected behavior?
the download should work

What went wrong?
rfc7230 does not specify any limit on the chunk-size, it just defines:

chunk-size     = 1*HEXDIG

in http_chunked_decoder.cc HttpChunkedDecoder::ParseChunkSize uses HexStringToInt to parse the chunk-size.

This restricts the chunk size to 2^31-1

Using an uint64_t and HexStringToUInt64 would be better
Firefox seems to be using a uint and be limited to 2^32-1

Did this work before? No 

Chrome version: 49.0.2623.75  Channel: stable
OS Version: 
Flash Version: Shockwave Flash 22.0 r0
 
net-internals-log (1).json
154 KB View Download
test_server.py
1.1 KB View Download

Comment 1 by mef@chromium.org, Jul 22 2016

Components: -Internals>Network Internals>Network>HTTP
Labels: -OS-Linux -Type-Bug -Pri-2 OS-All Pri-3 Type-Feature
Status: Untriaged (was: Unconfirmed)

Comment 2 by mmenke@chromium.org, Jul 22 2016

Owner: mmenke@chromium.org
Status: Assigned (was: Untriaged)
Thanks for the high quality bug report, much appreciated!  You're absolutely correct about there being no specced max chunk size.

I'll take this on myself.  Not high priority, but it's a simple enough thing to fix, and a silly reason to fail requests.
Project Member

Comment 3 by bugdroid1@chromium.org, Jul 25 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8ea249f8b70f154f9995ed538fc853fe9cf46503

commit 8ea249f8b70f154f9995ed538fc853fe9cf46503
Author: mmenke <mmenke@chromium.org>
Date: Mon Jul 25 19:49:16 2016

HttpChunkedDecoder: Support chunks longer than 2^31-1 bytes.

We were using HexStringToInt to parse chunk size, which returns a
32-bit int.  This CL switches to using HexStringToInt64, which uses
64-bit ints, so we can now support chunks up to 2^63-1 bytes.

That should be enough for anybody. [Cue dramatic music]

BUG= 630680 

Review-Url: https://codereview.chromium.org/2170133004
Cr-Commit-Position: refs/heads/master@{#407549}

[modify] https://crrev.com/8ea249f8b70f154f9995ed538fc853fe9cf46503/net/http/http_chunked_decoder.cc
[modify] https://crrev.com/8ea249f8b70f154f9995ed538fc853fe9cf46503/net/http/http_chunked_decoder.h
[modify] https://crrev.com/8ea249f8b70f154f9995ed538fc853fe9cf46503/net/http/http_chunked_decoder_unittest.cc

Comment 4 by mmenke@chromium.org, Jul 25 2016

Status: Fixed (was: Assigned)
Thanks for the fix, but wouldn't it be better to use uint64_t instead ?

Comment 6 by mmenke@chromium.org, Jul 25 2016

I kept it signed so that the existing logic just worked - mixing ints and uints when not needed is just asking for trouble, and most place in net/, we use ints.

I don't think there's much gain from being able to handle chunks from 8 to 16 exabytes in size, anyways - doubling the size just doesn't get us much.  There are reasonable use cases for chunk sizes from 2 GB to 8 exabytes, however.
Project Member

Comment 7 by bugdroid1@chromium.org, Jul 26 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2321e8a45603ca595296b67e4d064c070f2b3591

commit 2321e8a45603ca595296b67e4d064c070f2b3591
Author: mpearson <mpearson@chromium.org>
Date: Tue Jul 26 00:33:06 2016

Revert of HttpChunkedDecoder: Support chunks longer than 2^31-1 bytes. (patchset #4 id:60001 of https://codereview.chromium.org/2170133004/ )

Reason for revert:
Speculative revert for bug 631246.  Have no better ideas.

Apologies if this wasn't the cause.

Original issue's description:
> HttpChunkedDecoder: Support chunks longer than 2^31-1 bytes.
>
> We were using HexStringToInt to parse chunk size, which returns a
> 32-bit int.  This CL switches to using HexStringToInt64, which uses
> 64-bit ints, so we can now support chunks up to 2^63-1 bytes.
>
> That should be enough for anybody. [Cue dramatic music]
>
> BUG= 630680 
>
> Committed: https://crrev.com/8ea249f8b70f154f9995ed538fc853fe9cf46503
> Cr-Commit-Position: refs/heads/master@{#407549}

TBR=eroman@chromium.org,mmenke@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 630680 

Review-Url: https://codereview.chromium.org/2183433003
Cr-Commit-Position: refs/heads/master@{#407651}

[modify] https://crrev.com/2321e8a45603ca595296b67e4d064c070f2b3591/net/http/http_chunked_decoder.cc
[modify] https://crrev.com/2321e8a45603ca595296b67e4d064c070f2b3591/net/http/http_chunked_decoder.h
[modify] https://crrev.com/2321e8a45603ca595296b67e4d064c070f2b3591/net/http/http_chunked_decoder_unittest.cc

Project Member

Comment 8 by bugdroid1@chromium.org, Jul 26 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3fca2b4fdb391fa3c14386af8b72fc2092657029

commit 3fca2b4fdb391fa3c14386af8b72fc2092657029
Author: mmenke <mmenke@chromium.org>
Date: Tue Jul 26 04:01:41 2016

Reland of HttpChunkedDecoder: Support chunks longer than 2^31-1 bytes. (patchset #1 id:1 of https://codereview.chromium.org/2183433003/ )

Reason for revert:
The CL was blamed for compile failures it didn't cause, re-landing.

Original issue's description:
> Revert of HttpChunkedDecoder: Support chunks longer than 2^31-1 bytes. (patchset #4 id:60001 of https://codereview.chromium.org/2170133004/ )
>
> Reason for revert:
> Speculative revert for bug 631246.  Have no better ideas.
>
> Apologies if this wasn't the cause.
>
> Original issue's description:
> > HttpChunkedDecoder: Support chunks longer than 2^31-1 bytes.
> >
> > We were using HexStringToInt to parse chunk size, which returns a
> > 32-bit int.  This CL switches to using HexStringToInt64, which uses
> > 64-bit ints, so we can now support chunks up to 2^63-1 bytes.
> >
> > That should be enough for anybody. [Cue dramatic music]
> >
> > BUG= 630680 
> >
> > Committed: https://crrev.com/8ea249f8b70f154f9995ed538fc853fe9cf46503
> > Cr-Commit-Position: refs/heads/master@{#407549}
>
> TBR=eroman@chromium.org,mmenke@chromium.org
> # Skipping CQ checks because original CL landed less than 1 days ago.
> NOPRESUBMIT=true
> NOTREECHECKS=true
> NOTRY=true
> BUG= 630680 
>
> Committed: https://crrev.com/2321e8a45603ca595296b67e4d064c070f2b3591
> Cr-Commit-Position: refs/heads/master@{#407651}

TBR=eroman@chromium.org,mpearson@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 630680 

Review-Url: https://codereview.chromium.org/2180063002
Cr-Commit-Position: refs/heads/master@{#407700}

[modify] https://crrev.com/3fca2b4fdb391fa3c14386af8b72fc2092657029/net/http/http_chunked_decoder.cc
[modify] https://crrev.com/3fca2b4fdb391fa3c14386af8b72fc2092657029/net/http/http_chunked_decoder.h
[modify] https://crrev.com/3fca2b4fdb391fa3c14386af8b72fc2092657029/net/http/http_chunked_decoder_unittest.cc

Project Member

Comment 9 by bugdroid1@chromium.org, Jul 26 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3fca2b4fdb391fa3c14386af8b72fc2092657029

commit 3fca2b4fdb391fa3c14386af8b72fc2092657029
Author: mmenke <mmenke@chromium.org>
Date: Tue Jul 26 04:01:41 2016

Reland of HttpChunkedDecoder: Support chunks longer than 2^31-1 bytes. (patchset #1 id:1 of https://codereview.chromium.org/2183433003/ )

Reason for revert:
The CL was blamed for compile failures it didn't cause, re-landing.

Original issue's description:
> Revert of HttpChunkedDecoder: Support chunks longer than 2^31-1 bytes. (patchset #4 id:60001 of https://codereview.chromium.org/2170133004/ )
>
> Reason for revert:
> Speculative revert for bug 631246.  Have no better ideas.
>
> Apologies if this wasn't the cause.
>
> Original issue's description:
> > HttpChunkedDecoder: Support chunks longer than 2^31-1 bytes.
> >
> > We were using HexStringToInt to parse chunk size, which returns a
> > 32-bit int.  This CL switches to using HexStringToInt64, which uses
> > 64-bit ints, so we can now support chunks up to 2^63-1 bytes.
> >
> > That should be enough for anybody. [Cue dramatic music]
> >
> > BUG= 630680 
> >
> > Committed: https://crrev.com/8ea249f8b70f154f9995ed538fc853fe9cf46503
> > Cr-Commit-Position: refs/heads/master@{#407549}
>
> TBR=eroman@chromium.org,mmenke@chromium.org
> # Skipping CQ checks because original CL landed less than 1 days ago.
> NOPRESUBMIT=true
> NOTREECHECKS=true
> NOTRY=true
> BUG= 630680 
>
> Committed: https://crrev.com/2321e8a45603ca595296b67e4d064c070f2b3591
> Cr-Commit-Position: refs/heads/master@{#407651}

TBR=eroman@chromium.org,mpearson@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 630680 

Review-Url: https://codereview.chromium.org/2180063002
Cr-Commit-Position: refs/heads/master@{#407700}

[modify] https://crrev.com/3fca2b4fdb391fa3c14386af8b72fc2092657029/net/http/http_chunked_decoder.cc
[modify] https://crrev.com/3fca2b4fdb391fa3c14386af8b72fc2092657029/net/http/http_chunked_decoder.h
[modify] https://crrev.com/3fca2b4fdb391fa3c14386af8b72fc2092657029/net/http/http_chunked_decoder_unittest.cc

Components: Internals>Network
Components: -Internals>Network>HTTP

Sign in to add a comment