Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Issue 505048 Chrome makes more conditional re-validation requests than other browsers
Starred by 23 users Reported by ben.mau...@gmail.com, Jun 26 2015 Back to list
Status: Fixed
Owner:
Closed: Apr 1
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 2
Type: Bug


Sign in to add a comment
UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36

Example URL:
http://www.facebook.com

Steps to reproduce the problem:
A while back, we noticed that a significant percentage of requests to Facebook's CDN for javascript & css resources were If-Modified-Since requests that resulted in a 304 Not Modified response. Interestingly, Chrome had a substantially higher fraction of such requests than other browsers. We found one substantial cause of this, that Chrome had a non-standard behavior that re-validated resources on a page that was redirected to after a POST (eg, when a user made a post to log in to the site and we redirected them to the home page).

Since this fix was applied (see issue 294030), we ran the same analysis again. Chrome seems to have improved substantially -- in the past 60% of the requests Chrome sent for JS/CSS resources were conditional requests resulting in a 304 not modified. That number has dropped to 24%. However, Chrome still sends us nearly 2X the ratio of conditional requests.

Chrome     24%
Firefox    12%
IE         10%
Safari     14%

What is the expected behavior?
Chrome should have the same level of revalidation requests as other browsers

What went wrong?
Chrome had a higher level

Did this work before? N/A 

Chrome version: 43.0.2357.130  Channel: stable
OS Version: OS X 10.10.3
Flash Version: Shockwave Flash 18.0 r0
 
Cc: davidben@chromium.org japhet@chromium.org
Additional context: https://groups.google.com/a/chromium.org/forum/#!topic/net-dev/ZQjvaV4ogTM

One thing that was mentioned was the FrameLoadTypeSame behavior possibly causing this. (But that was just something that came to mind on a whim. Enumerating everything that sets LOAD_VALIDATE_CACHE is probably not going to be worth anyone's time.)

At a glance, Firefox does not seem to have this same-URL behavior. We could maybe just get rid of it. This link also seems to suggest we and Safari are alone in it. +japhet
https://www.stevesouders.com/blog/2011/06/27/unexpected-reloads-in-webkit/

Of course, given how old this is, I'm sure we'll discover tons of tiny subtle things everywhere rely on it and will have to chase down bugs. :-) And by old, I mean nearly six years older than the public release of Chrome:
https://chromium.googlesource.com/chromium/blink/+/71a896d186d9e937f9ce794a39686311d294cc1b 

(That was quite a git traversal. FrameLoader got renamed so many times...)
Comment 2 by japhet@chromium.org, Jun 26 2015
I tried to rip out FrameLoadTypeSame a year or two ago and it got reverted. We could certainly take another stab at it.
Could we make FrameLoadTypeSame so that it does a revalidation request on the root document, but that it does not revalidate sub resources? That might be a reasonable compromise between not revalidating at all and revalidating everything.
Cc: igrigo...@chromium.org
Ben: in the report you indicated "OS-Mac". Is this data for Mac users only, or does it apply to all platforms?
Our statistics show that this happens to users across all platforms.
Labels: -OS-Mac OS-All
Thanks, updated the bug.
japhet: Do you remember why it got reverted?
Alternatively, I could have just looked:
https://codereview.chromium.org/14672016
https://codereview.chromium.org/14610003

Although, that turns Same into Reload. I was thinking turning Same into Standard.
#3: That's an option... there's some logic there for POST I think. But before we think about adding complexity for compromises, we should probably confirm we even want the top-level to revalidate. :-P
I dug up the patch. It says it broke an android test, but links to the bot results instead of listing the failing tests. Since bot results get purged, the test list is gone.

@ben.maurer, I'm open to the idea of revalidating the main resource of the main frame only, especially if removing the FrameLoadTypeSame concept is infeasible for some reason. Do you prefer that we explicitly keep the main resource revalidation behavior?
Personally, I don't really care one way or another if we revalidate the main resource. For Facebook, our main resources are always no-cache (since they are dynamic content). My main concern is to avoid revalidating sub resources
Labels: -Cr-Internals-Network Cr-Internals-Network-Cache
I think the cache is the appropriate network sub-bucket for this, though I understand we may be driving the behavior in question from outside the cache.
Status: Untriaged
Comment 14 by y...@yoav.ws, Jun 30 2015
Cc: y...@yoav.ws
Ben - do you have a simple test case that shows that behavior? (conditional requests being sent when nothing should have been sent)
Unfortunately, we don't have a specific test case here -- though, the same page navigation seems like a clear case where chrome differs from other browsers.
Labels: Cr-Blink-Loader Cr-UI-Browser-Navigation
re #1: If by "same-url" behavior you mean that hitting enter into the omnibox without navigating away triggers a reload, that was an intentional UI decision (I'm not saying it cannot change). The background was that most users don't really mess with URLs (as opposed to devs) and if there is a URL there and the user hits enter, the real desired action is to display again the current page, possibly fixing any error or "hang" that may be happening (effectively, what reload means) (I don't think there's data about it). I don't know if that matters at this point, but I still think that enter==reload is a natural assumption.

As far as I understand, that is also part of the meaning of pressing F5: if there is something missing (or wrong?) somewhere in the current page, it should be fixed by asking the server again. I'm not sure we want to limit that bahavior to the main resource on a page, but maybe I misunderstand the proposal. Reload (or force reload) is not a regular navigation by any means.

Now, if we end up using the flag without a clear user interaction, we should fix that. And if users are reloading a lot we should also fix that.

On another note, as a user I'd like to see a better separation of use cases:
1. I just want the latest version of a page (there's nothing that I'm trying to fix).
2. There is something wrong or the page is stuck, so try again
3. This is completely broken start from scratch (don't even ask for revalidation, and force all intermediary to also go back to the server).

Right now 1 & 2 mean Reload (F5 or enter), but I don't need special flags for #1.
3 means force reload (ctrl-F5).

Of course, most people don't know that force-reload even exists. (again, no data to back this up).
Ben: do you have any data on number of same-page reloads for your use case? Can it explain some of the difference between Chrome and other browsers?
Not sure how we'd get that data -- there's no way for us to tell if a navigation was caused by the refresh button or by the URL bar.
Cc: kenjibaheux@chromium.org
Facepalm, right.. Checking our UMA metrics, I don't see anything tracking this either. Perhaps this is a good place to start on this end. 
We looked at some data about what % of FB loads are reloads.

We see about 2.2% of page loads from Chrome being reloads, 1.9% being back-forward the rest being normal navigation. IE has roughly the same ratios, firefox is lower on reloads (1%ish). Given that many navigations to FB happen with a warm cache it's not unreasonable that 2% of page loads being reloads result in 20% of the misses in the browser cache (and thus hits on our CDN).

We aren't sure if these reloads are explicitly triggered by the user or are via same page navigation. 
Comment 21 Deleted
Additional data confirming similar Chrome behavior from one of our Google products (with lots of traffic):

+----------------+---------+
| browser        | 304     |
+----------------+---------+
| Chrome Desktop |   38.8% |
| Opera Mini     |   15.6% |
| Opera          |   13.4% |
| Chrome Android |   12.3% |
| Safari Mac     |   12.3% |
| iOS            |   4.8%  |
| IE             |   4.2%  |
| Android        |   1.5%  |
| FF             |   1.2%  |
| other          |   0.1%  |
+----------------+---------+

For desktop, ~39% of Chrome requests are revalidations as compared to ~4% of IE requests - that's a very big difference. On mobile things are looking a bit better, but we're still 3x the revalidation rate of mobile Safari.

Unfortunately this data does shed any light on the root cause.. need to keep digging.
In no particular order, brainstorming some mechanisms that can trigger this:

- Lots of reload extensions: https://chrome.google.com/webstore/search/reload?hl=en&_category=extensions
- DevTools: we know that we have many developers using DevTools, can that skew data? 
- Offline auto-reload: chrome://flags/#enable-offline-auto-reload, chrome://flags/#enable-offline-auto-reload-visible-only
- Restart / tab restore?
- Reload button
- URL bar + enter (we revalidate)
- "hard refresh" (cmd-r and equivalent)

Anything else? An "initiator" histogram on reloads would be awesome to have.
I think it makes most sense to start by getting rid of the same-URL thing. That's a concrete thing we know of where we behave different from other browsers and probably can just get rid of it.

There's enough navigation entry points right now and places where the relevant bits can get set that plumbing through an "initiator" enum would likely be rather a mess.
Project Member Comment 25 by bugdroid1@chromium.org, Jul 18 2015
The following revision refers to this bug:
  http://src.chromium.org/viewvc/blink?view=rev&rev=199143

------------------------------------------------------------------
r199143 | japhet@chromium.org | 2015-07-18T00:24:40.633597Z

Changed paths:
   M http://src.chromium.org/viewvc/blink/trunk/Source/core/loader/FrameLoader.cpp?r1=199143&r2=199142&pathrev=199143
   M http://src.chromium.org/viewvc/blink/trunk/Source/core/loader/FrameFetchContext.cpp?r1=199143&r2=199142&pathrev=199143
   M http://src.chromium.org/viewvc/blink/trunk/Source/core/loader/FrameFetchContextTest.cpp?r1=199143&r2=199142&pathrev=199143

Don't revalidate all resources for FrameLoadTypeSame

The main resource of the top-loading frame probably need to be revalidated, but the rest of the page can probably prefer the cache.

Also, simplify setting of load type FrameLoadTypeSame.

BUG= 505048 

Review URL: https://codereview.chromium.org/1213913003
-----------------------------------------------------------------
Project Member Comment 26 by bugdroid1@chromium.org, Jul 18 2015
The following revision refers to this bug:
  http://src.chromium.org/viewvc/blink?view=rev&rev=199155

------------------------------------------------------------------
r199155 | sigbjornf@opera.com | 2015-07-18T07:48:08.981112Z

Changed paths:
   M http://src.chromium.org/viewvc/blink/trunk/Source/core/loader/FrameFetchContextTest.cpp?r1=199155&r2=199154&pathrev=199155

Oilpan: fix webkit unit tests following r199143.

Need to promptly detach the local frame created in
FrameFetchContextCachePolicyTest.MainResource.

TBR=oilpan-reviews
BUG= 505048 
NOTRY=true

Review URL: https://codereview.chromium.org/1245513003
-----------------------------------------------------------------
See issue 511699 for one particular case where we send revalidation requests when we apparently shouldn't.  The repro is extension-related, but may happen at other times as well, not sure.
> I think it makes most sense to start by getting rid of the same-URL thing. That's a concrete thing we know of where we behave different from other browsers and probably can just get rid of it.

+1. At a minimum I'd like to see a counter and an experiment to see how frequently this is triggered. For reference, background on this behavior: https://code.google.com/p/chromium/issues/detail?id=505048#c16. 


Cc: kinuko@chromium.org
On mobile, there is another popular* flow which is similar to the same-URL navigation pattern: pull to refresh. From a quick test, it appears that we conditionally revalidate everything for this flow.

The gesture comes from native apps where the expectation is one of refreshing the content of the app not reloading the app (i.e. re-installing/fixing errors). Can we also apply the same tweak here?


*: used by 24% of our uma users (7 days); the reload button in the menu is used by 6% of our uma users (7 days).
Cc: gavinp@chromium.org cbentzel@chromium.org
Fb stats suggest that we see roughly the same number of revalidations across all chrome platforms:

Windows 19% 
Android	21%
Mac OS X 21%
Chrome OS 18%
Blockedon: chromium:558829
Talked to our UX team, they would like to play with the new behavior for pull to refresh.
Filed issue 558829
OK, the change to navigation to same URL from Omnibox didn't have a huge impact on the numbers as seen from Facebook. I'm assuming that it's not popular enough. Higher hopes for pull to refresh.

However, pull to refresh, while a huge opportunity, is only relevant to mobile. This issue being cross platform, this suggests that there is definitely something else at play.
Comment 35 by bmau...@fb.com, Jan 11 2016
Labels: DevRel-Faceboook
Labels: -DevRel-Faceboook DevRel-Facebook
Owner: kenjibaheux@chromium.org
Status: Started
I looked into Tab evictions. Here is what I'm seeing:
 1. On Android, a call to setNeedsReload is made for tabs that activated after an eviction. This flags triggers a NO_RELOAD load (i.e. loads without revalidating the cache). See [1] below for details.

 2. On other platforms, evicted tabs are reloaded with type=RELOAD which triggers a cache revalidation. See [2] for details. 

I also found an existing bug for this that was scoped to Chrome OS: issue 309693


//Wall of urls//

[1]: bread crumbs: 
 - setNeedsReload called upon tab eviction at: https://code.google.com/p/chromium/codesearch#chromium/src/chrome/android/java/src/org/chromium/chrome/browser/tab/TabWebContentsObserver.java&l=123

 - restoreIfNeeded calling requestRestoreLoad if needsReload is set: https://code.google.com/p/chromium/codesearch#chromium/src/chrome/android/java/src/org/chromium/chrome/browser/tab/Tab.java&l=1897

 - requestRestoreLoad calling native function : https://code.google.com/p/chromium/codesearch#chromium/src/chrome/android/java/src/org/chromium/chrome/browser/tab/Tab.java&l=703

 - native setNeedsReload called: https://code.google.com/p/chromium/codesearch#chromium/src/content/browser/frame_host/navigation_controller_android.cc&l=176

 - loadIfNecessary checking for needsreload and using NO_RELOAD (i.e. without revalidating the cache): https://code.google.com/p/chromium/codesearch#chromium/src/content/browser/frame_host/navigation_controller_impl.cc&l=1921

 [2]: bread crumbs
 - reload called for discarded tabs: https://code.google.com/p/chromium/codesearch#chromium/src/chrome/browser/ui/browser.cc&l=1062
 
 - reloadInternal called: https://code.google.com/p/chromium/codesearch#chromium/src/chrome/browser/ui/browser_commands.cc&rcl=1453163556&l=427

 - call to reload(true): https://code.google.com/p/chromium/codesearch#chromium/src/chrome/browser/ui/browser_commands.cc&rcl=1453163556&l=235

 - reloadinternal called with type set to RELOAD: https://code.google.com/p/chromium/codesearch#chromium/src/content/browser/frame_host/navigation_controller_impl.cc&rcl=1453163556&l=295

 - call to NavigateToPendingEntry with Type set to RELOAD https://code.google.com/p/chromium/codesearch#chromium/src/content/browser/frame_host/navigation_controller_impl.cc&rcl=1453163556&l=406

 - call to NavigateToPendingEntryInternal, type = RELOAD: https://code.google.com/p/chromium/codesearch#chromium/src/content/browser/frame_host/navigation_controller_impl.cc&rcl=1453163556&l=1789
 
 ...
Blockedon: chromium:309693
Nice find!

We saw high numbers of reloads across all platforms. Do tab evictions apply in windows and mac?
Not historically.  That may have changed at some point, though.
It's currently off by default on Windows and Mac OS X.

However, there are Tab Evictions experiments running for Windows and Mac. It's unclear how much 304s we can attribute to evictions though.

Regardless, it's still worth considering fixing issue 309693 because Tab Evictions will eventually ship (guess) and make the problem even worse.

Comment #22 indicates that revalidations on Chrome desktop happens far more often than Chrome mobile, hence the hope that the reload behavior differences on tab restore would be an important reason.
Cc: chrisha@chromium.org georgesak@chromium.org
Adding myself and chrisha@ on CC, as we're actively working on tab discarding/eviction for desktop.
Was wondering if we managed to work out anything new here in the last couple of weeks, or if this is part of some bigger plan to fix issues? Thanks
This still is definitely on our radar among many other things.
We have identified 2 causes (see blocked on) so far but clearly they do not explain it all and we are still looking for more.

Update:
 The initial patch for 558829 didn't WAI. Toyoshim had a working version a week or so ago but the reloading paths are overly complex which lead to the desire to clean up the mess. Hopefully, this would help us tame the beast without having to track individual triggers.
Just off the press: a working patch for 558829 has been submitted and *should* be reflected in the Dev channel for Android behind chrome://flags#enable-non-validating-reload-on-refresh-content
Great to know, I've enabled the flag.
May I ask why it's not on by default in Dev (or later beta)? I don't think
many users turn that on just out of curiosity. (as I think the localization
of the flag is a bit confusing)
Cc: toyoshim@chromium.org
Labels: Reload
Comment 48 by bmau...@fb.com, Feb 25 2016
For what it's worth at fb we're seeing this on desktop and mobile. So the pull to refresh fix is great, but there also seems to be a substantial number of revalidations coming from desktop refreshes.
#48: I know that there is more, hence this meta-bug still being "started".

#46: Definitely, we don't expect that a lot of folks know about nor should care about chrome://flags. This is where our experiments live. Provided nothing surprising shows up, we'll eventually turn this ON for everyone.
Comment 50 by bmau...@fb.com, Feb 28 2016
We started logging window.performance.navigation.type.

On Chrome/Windows, 4.3% of navigations to the FB home page are TYPE_RELOAD -- compare to 5% for Firefox, 2.7% for Edge.

On Chrome/Android,  14.3% of navigations to the FB home page are TYPE_RELOAD.

This suggests that pull to refresh does result in a much higher number of reload operations on mobile. Removing revalidations on PTR will likely be a big win in the mobile experience.

That said, we're still seeing almost 5% of desktop navigations be reloads. This seems to not be out of the range of other desktop browsers. 
Comment 51 by bmau...@fb.com, Feb 28 2016
Keep in mind that it wouldn't be unusual for 5% of our navigations to result in 20% of our CDN requests being 304s. Many users navigate to FB and have their resources cached. So the 5% may account for a high percentage of the uncached page loads.
Thanks Ben!

So, if the type=Reload is inline with other browsers, it could mean 2 things:
 1. the behavior of our type=reload over-triggers revalidations.
 2. something else is driving the extra revalidations.

Is there anything odd going on with the other values (esp. TYPE_BACK_FORWARD) ?
I'll do some tests and report back.
Comment 53 by bmau...@fb.com, Feb 29 2016
We didn't see anything unusual with other values. I'll point out a 3rd possibility is that chrome is just generally more cache efficient than other browsers and so has fewer non-reload requests.

That said, if 5% of our navigations are reloads, and these reloads are actually being requested by real users we really want to find a way to avoid the excessive revalidations in these cases. This is especially true if reloads are driven by user frustrations on slow connections.
>This is especially true if reloads are driven by user frustrations on slow connections.

Ben, can  you segment this out in your analytics? If not, Chrome for Android supports downlinkMax: http://w3c.github.io/netinfo/#widl-NetworkInformation-downlinkMax. 
Worth noting that slow connections and spotty connections may both have problems here.  Not sure spotty connections are reflected downlinkMax, or ones that are technically fast but in practice slow.
Comment 56 by bmau...@fb.com, Mar 1 2016
there looks to be at least some correlation between location and number of refreshes. For example on desktop we see this percentage of requests being reloads:

north america 3.8%
europe 4%
africa 5.7%
oceania 4.7%
asia 4.2%

That said, the pattern doesn't hold on mobile (in fact africa gets more reloads than north america). It's possible that the PTR behavior on android is skewing the stats from people who reload out of frustration.


Blockedon: 600636
As part of this issue, here are our thoughts on Reload behaviors
 https://groups.google.com/a/chromium.org/d/msg/loading-dev/LpgNPCMEg1I/zVtAg6ujBQAJ
Blocking: 519042
Just checking in - is this issue now resolved?
Should we consider this fixed by now due to all the reload-reloaded work?
Status: Fixed
Yes, I believe that we are in a much better place now.
Sign in to add a comment