It might help to have a net-log dump during the speedtest in both cases.
https://dev.chromium.org/for-testers/providing-network-details

My gut is this could just be latency incurred by writing to the HTTP disk cache, which is in-memory in incognito profiles. +morlovich to check me though.

FWIW I can't reproduce this on my workstation + network. I consistently get 940mbit throughput.

Is this a recent regression for you?

As per comment#1 adding Needs-Feedback label.

@Reporter: Please provide net-log for further investigation from dev team.

Thanks!

Will get you the capture that's what I was hoping to know exactly what to collect.

FYI, in my tests the underlying disk is very fast SSD, with simple cache flag enabled.  Not disregarding your first guess, just adding that info I should have originally.

Should have the capture uploaded here shortly.  TIA!

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

I just did two side by side caps using the remove private data setting.  I pre-loaded the speedtest.net site, set both windows to utilize the same test server, waited for idle and then began the cap and immediately stopped it when the test completed for each - capturing each run (NORMAL) and (INCOGNITO).

Note, since I seem to be able to consistently reproduce this on multiple systems, for sake of getting this to you quickly just took a DEV workstation I run that as you will see does, unlike the others, have numerous extensions installed but you will note that I did disable them all.  I was glad to see I still was able to capture exactly what I keep seeing - roughly 200mbit on normal and almost 500 on incognito on this system (although unlike the other, this system does in fact have a mechanical HDD).

I loaded the caps myself to verify you can see the socket levels I stated above in each - but that is where my knowledge ends as to reading the diagnostics and capture to determine more.  Interestingly, I do NOT see the graph line for disk cache, is that because I am using the simple caching method?

I would really love to learn a bit more on how to drill down better in cases like this to determine what is happening so I would love any process info you care to share as to how you are going about the dissection!  TIA!

NORMAL MODE CAPTURE FILE:
https://1drv.ms/u/s!AhSqjTYUuHPKg7VZ15ikDrYKP88AZA

INCOGNITO MODE CAPTURE FILE:
https://1drv.ms/u/s!AhSqjTYUuHPKg7VaoESS5b7NX_3KpQ

Let me know once you have retrieved these and I will flush the links.  TIA!

Thank you for the logs. I copied them to an internal drive link [1]. Some differences I noticed in the logs:
1. The incognito reads often did _not_ fill up the 32k read buffer, while the normal reads did (aka _more_ reads in incognito). This indicates to me that in the "normal" case, we are never bottlenecked on the network, since we fill up the read buffer fully every time. The bottleneck is in Chrome.

2. Incognito finished requests quicker

3. Incognito triggered something in the application which caused it to issue larger requests. Both tests started off fetching 25mb resources, but the incoginto test ramped up to 50mb.

I'm adding the cache component because I think it's the most likely thing to be affected by incognito. Even if you have a fast SSD, it's hard to beat synchronous writes to memory. I wouldn't be surprised if an SSD is limited to 500mbps.

[1]: https://drive.google.com/open?id=10Vi_E7VwqJB5d76E-yorCnfbEK5mrQ9M

Interesting!  I had never loaded caps up in chrome so this was interesting and I can follow what you are saying but I'm not sure I agree about the SSD limit (remember we're talking mBIT here it may be limited to 400-500 megaBYTE like you're saying (it is, I just checked), but that equals 4000mbit and we're certainly not pushing that through Chrome on these tests!). :)  We're only pushing 500mbit/8 = 62.5 mBYTE which is no problem even for a crummy mechanical SATA drive.

However, I too believe I have found (at least in my test case) a possible root cause - CPU.  Is that not shown anywhere in the captures?  I ask because what I had not been watching was the CPU and in this test case I ran quick for you, it was on a much older test system that only had an older Core2 cpu  which in any other case has no issue utilizing a gigabit NIC....until you throw in all the apparent processing overhead required for Chrome internals these days.

Watching via taskmgr in each mode (and of course if I ENABLE all the extensions) shows the CPU hitting consistent 100% during the speed test at that high of throughput.

I also just re-ran using a much newer system that I ensured had a combination of a much newer, LOW UTILIZED CPU as well as a SSD, set the flag for simple cache method and for the first time was able to do THIS with a dozen extensions enabled, no less (543mbit):

https://i.imgur.com/nebx8a4.png

So, this does look like a sliding scale and like you said, incognito showed it mysteriously "triggered something in the application which caused it to issue larger requests".....could this be because in incognito the CPU requirements were simply less, allowing it to step "up" to the next level in the speedtest to request the 50mb chunks?  That's my theory.

This does probably look more like a possible opportunity to identify perhaps inefficiencies in one or more Chrome processes causing such high CPU requirements.....or you just chalk it up to "if you want to go THAT fast, be sure you're running a FAST enough system". :)  (which BTW the real issue could just be that folks like me may THINK they are...but need to have "FAST system" requirements better detailed.  (AKA: if you expect to pump gigabit through Chrome, i5 or higher required)).

Thoughts?  I'll be happy to collect more if you want to investigate further.  Otherwise as the reporter - I've now finally with your help been able to at least see it work properly in one case on one system, so other than wishing Chrome perhaps didn't require so much horsepower, am satisfied.

Thanks again for the help!

Ah you're right, I was mixing up my units. In any case CPU bottleneck is also very plausible. It's hard to tell CPU bottlenecking from the netlog, but one thing you can try is an about:tracing log which measures similar things but is more geared towards performance.

To log one, go to about:tracing and hit "record". Then, manually select the categories (only these): net, netlog, toplevel, toplevel.flow

Careful: this logging system uses a lot of memory, you might want to stop recording _before_ the upload stage of speedtest to keep the log small. If it's still too big, just record the toplevel category.


It's plausible for the simple cache to use more CPU than the in-memory one. For one thing the in-memory cache has a synchronous API, so that means we don't need any logic or overhead of queuing tasks (which often involves copying metadata around).

A CPU profile of the benchmark could also help, but that is not something you can provide. We have to do it ourselves :)

For anyone following along, new CPU profiling docs have recently been published (kudos to szager):
https://chromium.googlesource.com/chromium/src/+/master/docs/profiling.md

There is definitely a lot more overhead in simple cache, yes, like a whole bunch of data structures, cross-thread hops, computation of CRC, and, well, also system calls to write out the data.

It's also probably worth pointing out that the in-memory cache will never store a resource bigger than 6.25MiB anyway.

Adding label TE-NeedsTriageHelp as the issue is already been taken up by 	
Internals>Network>Cache team and making the issue out of the testing team bucket.

Thanks...!!