I suspect the point where our request throttle triggers shouldInterceptRequest is just before the network stack has added these, if this is happening?

Toby, maybe something to look at with the new request APIs?

> For example the "Referer" header is missing. 

I always see the Referer header, no issues.

What version of chrome is this (Settings > Apps > Chrome)?

FWIW we have tests which explicitly check for the Referer header: https://cs.chromium.org/chromium/src/android_webview/javatests/src/org/chromium/android_webview/test/AwContentsClientShouldInterceptRequestTest.java?l=249

Is it maybe disabled for iframes or something?

I stepped through the code and you are right, referrer is passed but not on the case I was testing when I filed this issue. 

I'm attaching a screenshot of what the Chrome developer tools shows me while Android Studio is paused on a breakpoint on shouldInterceptRequest() and I'm attaching a screenshot of what Android Studio shows me as the passed headers. 

Chrome 59.0.3071.125 

Deleted: Screenshot from 2017-07-17 18-38-43.png 50.6 KB

	Screenshot from 2017-07-17 18-38-43.png 50.6 KB View Download

Deleted: Screenshot from 2017-07-17 18-38-32.png 22.0 KB

	Screenshot from 2017-07-17 18-38-32.png 22.0 KB View Download

Thank you for providing more feedback. Adding requester "ntfschr@chromium.org" to the cc list and removing "Needs-Feedback" label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

I still can't reproduce (tested on http://testsafebrowsing.appspot.com/s/malware_in_iframe.html).

What URL are you trying to load?

What happens if you log each header? Are you really missing the headers, or is Android Studio wrong?

```
public WebResourceResponse shouldInterceptRequest(WebView view, WebResourceRequest request) {
    Map<String, String> headers = request.getRequestHeaders();
    for (Map.Entry<String, String> entry : headers.entrySet()) {
        String key = entry.getKey();
        Object value = entry.getValue();
        Log.w("debug-header", key + ": " + value);
    }
    return null;
}
```

Nope, printing them out still didn't show the referrer. This is what Chrome has after the request is completed as request headers:

:authority:server.repelisplus.com
:method:GET
:path:/player/?iframe=nJufXn8FLMIdvOIJ7x1YqUBM0VqtVJb1IN9E8jR9bXjyvQGfqveAStLfXY8kcqKVnd--__--ulymyCVvJyP8CxhnuY79qMNt3o0O0mvyR3z7oyaFQwgUGy17FqKyDLs2jcXUI
:scheme:https
accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
accept-encoding:gzip, deflate
accept-language:en-US,es-EC;q=0.8
cookie:__cfduid=d67b00bdfe0d34f0b57464539777d46091500083693; __utmt=1; __utma=49954084.251348429.1500083692.1500334748.1500338826.4; __utmb=49954084.4.10.1500338826; __utmc=49954084; __utmz=49954084.1500338826.4.4.utmcsr=repelisplus.com|utmccn=(referral)|utmcmd=referral|utmcct=/ver/power-rangers/; _ga=GA1.2.251348429.1500083692; _gid=GA1.2.698831518.1500334628; __utma=164050492.251348429.1500083692.1500334627.1500338803.4; __utmb=164050492.12.10.1500338803; __utmc=164050492; __utmz=164050492.1500334627.3.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)
referer:https://www.repelisplus.com/ver/power-rangers/?m=play
upgrade-insecure-requests:1
user-agent:Mozilla/5.0 (Linux; Android 7.1.1; Nexus 6 Build/N6F26U; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/59.0.3071.125 Mobile Safari/537.36
x-devtools-emulate-network-conditions-client-id:c3251786-8a66-41ed-910d-224f041bbf11
x-devtools-request-id:30045.267
x-requested-with:mypackage

This is all that my code got (formatted as header:value plus the word "for" followed by the url):

07-17 19:56:22.772 30045-30278/mypackage I/mypackage.WebClient: shouldInterceptRequest header Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8 for https://server.repelisplus.com/player/?iframe=nJufXn8FLMIdvOIJ7x1YqUBM0VqtVJb1IN9E8jR9bXjyvQGfqveAStLfXY8kcqKVnd--__--ulymyCVvJyP8CxhnuY79qMNt3o0O0mvyR3z7oyaFQwgUGy17FqKyDLs2jcXUI

07-17 19:56:22.772 30045-30278/mypackage I/mypackage.WebClient: shouldInterceptRequest header User-Agent:Mozilla/5.0 (Linux; Android 7.1.1; Nexus 6 Build/N6F26U; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/59.0.3071.125 Mobile Safari/537.36 for https://server.repelisplus.com/player/?iframe=nJufXn8FLMIdvOIJ7x1YqUBM0VqtVJb1IN9E8jR9bXjyvQGfqveAStLfXY8kcqKVnd--__--ulymyCVvJyP8CxhnuY79qMNt3o0O0mvyR3z7oyaFQwgUGy17FqKyDLs2jcXUI

07-17 19:56:22.772 30045-30278/mypackage I/mypackage.WebClient: shouldInterceptRequest header X-DevTools-Emulate-Network-Conditions-Client-Id:c3251786-8a66-41ed-910d-224f041bbf11 for https://server.repelisplus.com/player/?iframe=nJufXn8FLMIdvOIJ7x1YqUBM0VqtVJb1IN9E8jR9bXjyvQGfqveAStLfXY8kcqKVnd--__--ulymyCVvJyP8CxhnuY79qMNt3o0O0mvyR3z7oyaFQwgUGy17FqKyDLs2jcXUI

07-17 19:56:22.772 30045-30278/mypackage I/mypackage.WebClient: shouldInterceptRequest header Upgrade-Insecure-Requests:1 for https://server.repelisplus.com/player/?iframe=nJufXn8FLMIdvOIJ7x1YqUBM0VqtVJb1IN9E8jR9bXjyvQGfqveAStLfXY8kcqKVnd--__--ulymyCVvJyP8CxhnuY79qMNt3o0O0mvyR3z7oyaFQwgUGy17FqKyDLs2jcXUI

07-17 19:56:22.772 30045-30278/mypackage I/mypackage.WebClient: shouldInterceptRequest header X-DevTools-Request-Id:30045.267 for https://server.repelisplus.com/player/?iframe=nJufXn8FLMIdvOIJ7x1YqUBM0VqtVJb1IN9E8jR9bXjyvQGfqveAStLfXY8kcqKVnd--__--ulymyCVvJyP8CxhnuY79qMNt3o0O0mvyR3z7oyaFQwgUGy17FqKyDLs2jcXUI

07-17 19:56:22.773 30045-30278/mypackage I/AppUtils: Timing - webclient shouldIntercept: 1 - https://server.repelisplus.com/player/?iframe=nJufXn8FLMIdvOIJ7x1YqUBM0VqtVJb1IN9E8jR9bXjyvQGfqveAStLfXY8kcqKVnd--__--ulymyCVvJyP8CxhnuY79qMNt3o0O0mvyR3z7oyaFQwgUGy17FqKyDLs2jcXUI


I'm wondering if maybe they are loading the iframe in some weird way? 

The steps to reproduce are below, hopefully you won't get many ads. 

Go to https://www.repelisplus.com/ver/power-rangers/?m=played

Scroll down maybe one page, you'll see a large red button that says "ver ahora", press it.

Then you'll see a list of a few options, press on streamango or streamcherry or openload. That last click is the one that loads the iframe that doesn't get the referrer header. 

Thank you for providing more feedback. Adding requester "ntfschr@chromium.org" to the cc list and removing "Needs-Feedback" label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

I get tons of log output (42 lines just from the last keypress). I've filtered to show only the Referer lines below.

Carlos, are my results consistent with what you see? Can you attach a minimal repro apk (with compiled apk and zipped source code)?

---
Referer:https://www.repelisplus.com/ver/power-rangers/?m=play for https://server.repelisplus.com/player/?iframe=nJufXn8FLMIdvOIJ7x1YqUBM0VqtVJb1IN9E8jR9bXjyvQGfqveAStLfXY8kcqKVnd--__--ulymyCVvJyP8CxhnuY79qMNt3o0O0mvyR3z7oyaFQwgUGy17FqKyDLs2jcXUI
Referer:https://www.repelisplus.com/ver/power-rangers/?m=played for https://ssl.google-analytics.com/__utm.gif?utmwv=5.6.7&utms=10&utmn=1961192823&utmhn=www.repelisplus.com&utmcs=UTF-8&utmsr=412x732&utmvp=412x604&utmsc=32-bit&utmul=en-us&utmje=0&utmfl=-&utmdt=Ver%20Power%20Rangers%20Online%2C%20Latino%2C%20Espa%C3%B1ol%2C%20Subtitulado%20-%20RepelisPlus&utmhid=811952994&utmr=-&utmp=%2Fver%2Fpower-rangers%2F%3Fm%3Dplayed&utmht=1500345004111&utmac=UA-87487512-1&utmcc=__utma%3D164050492.2038803514.1500344522.1500344522.1500344522.1%3B%2B__utmz%3D164050492.1500344522.1.1.utmcsr%3D(direct)%7Cutmccn%3D(direct)%7Cutmcmd%3D(none)%3B&utmjid=&utmu=qAAAAAAAAAAAAAAAAAAAAAAE~
Referer:https://www.repelisplus.com/ver/power-rangers/?m=played for https://www.repelisplus.com/favicon.ico
Referer:https://server.repelisplus.com/player/?iframe=nJufXn8FLMIdvOIJ7x1YqUBM0VqtVJb1IN9E8jR9bXjyvQGfqveAStLfXY8kcqKVnd--__--ulymyCVvJyP8CxhnuY79qMNt3o0O0mvyR3z7oyaFQwgUGy17FqKyDLs2jcXUI for https://ssl.google-analytics.com/ga.js
Referer:https://server.repelisplus.com/player/?iframe=nJufXn8FLMIdvOIJ7x1YqUBM0VqtVJb1IN9E8jR9bXjyvQGfqveAStLfXY8kcqKVnd--__--ulymyCVvJyP8CxhnuY79qMNt3o0O0mvyR3z7oyaFQwgUGy17FqKyDLs2jcXUI for https://ajax.cloudflare.com/cdn-cgi/nexp/dok3v=85b614c0f6/cloudflare.min.js
Referer:https://server.repelisplus.com/player/?iframe=nJufXn8FLMIdvOIJ7x1YqUBM0VqtVJb1IN9E8jR9bXjyvQGfqveAStLfXY8kcqKVnd--__--ulymyCVvJyP8CxhnuY79qMNt3o0O0mvyR3z7oyaFQwgUGy17FqKyDLs2jcXUI for https://server.repelisplus.com/player/?iframe=nJufXn8FLMIdvOIJ7x1YqUBM0VqtVJb1IN9E8jR9bXjyvQGfqveAStLfXY8kcqKVnd--__--ulymyCVvJyP8CxhnuY79qMNt3o0O0mvyR3z7oyaFQwgUGy17FqKyDLs2jcXUI&
Referer:https://server.repelisplus.com/player/?iframe=nJufXn8FLMIdvOIJ7x1YqUBM0VqtVJb1IN9E8jR9bXjyvQGfqveAStLfXY8kcqKVnd--__--ulymyCVvJyP8CxhnuY79qMNt3o0O0mvyR3z7oyaFQwgUGy17FqKyDLs2jcXUI for https://ssl.google-analytics.com/__utm.gif?utmwv=5.6.7&utms=2&utmn=710337349&utmhn=server.repelisplus.com&utmcs=UTF-8&utmsr=412x732&utmvp=401x278&utmsc=32-bit&utmul=en-us&utmje=0&utmfl=-&utmdt=PLAYER%20REPELIS&utmhid=508629750&utmr=https%3A%2F%2Fwww.repelisplus.com%2Fver%2Fpower-rangers%2F%3Fm%3Dplay&utmp=%2Fplayer%2F%3Fiframe%3DnJufXn8FLMIdvOIJ7x1YqUBM0VqtVJb1IN9E8jR9bXjyvQGfqveAStLfXY8kcqKVnd--__--ulymyCVvJyP8CxhnuY79qMNt3o0O0mvyR3z7oyaFQwgUGy17FqKyDLs2jcXUI&utmht=1500345004500&utmac=UA-87487512-1&utmcc=__utma%3D49954084.2038803514.1500344522.1500344850.1500344850.1%3B%2B__utmz%3D49954084.1500344850.1.1.utmcsr%3Drepelisplus.com%7Cutmccn%3D(referral)%7Cutmcmd%3Dreferral%7Cutmcct%3D%2Fver%2Fpower-rangers%2F%3B&utmjid=&utmu=qAAAAAAAAAAAAAAAAAAAAAAE~
Referer:https://streamango.com/embed/nbepcadrneldpsfp/pwr_-_Lat_1080p_avi_mp4 for https://cdnjs.cloudflare.com/ajax/libs/video.js/5.19.2/video-js.min.css
Referer:https://streamango.com/embed/nbepcadrneldpsfp/pwr_-_Lat_1080p_avi_mp4 for https://streamango.com/assets/css/video.js/mgvideo.css
Referer:https://streamango.com/embed/nbepcadrneldpsfp/pwr_-_Lat_1080p_avi_mp4 for https://content.fruithosted.net/splash/nbepcadrneldpsfp/bnqfcfkbmeplbsso.jpg
Referer:https://streamango.com/embed/nbepcadrneldpsfp/pwr_-_Lat_1080p_avi_mp4 for https://t1.streamango.com/log
---

The first two on your output is what I don't get. I will try to make a quick sample tomorrow. Thanks. 

Thank you for providing more feedback. Adding requester "ntfschr@chromium.org" to the cc list and removing "Needs-Feedback" label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Thanks. Adding back the needs-feedback label until we have the apk.

Once we have a sample apk we can determine if this is device specific or if the bug only repros under particular configuration.

I made a quick sample and it didn't have the issue. 

I don't understand what my app could be doing different to cause it to not receive the referrer header, doesn't seem right that something the app does could affect the webview on something like that. Plus the Chrome dev tools shows that the referrer is being sent. 

Any suggestions on what to do next?

Thanks. 

Thank you for providing more feedback. Adding requester "ntfschr@chromium.org" to the cc list and removing "Needs-Feedback" label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

I went as far as to replace my WebClient with the one from the test and still having the issue. I'm going to continue removing parts I guess until I figure out which does it. 

Alright, figured out the issue. I'm attaching a sample. 

This is what causes it:

        WebSettings settings = webView.getSettings();
        File cacheDir = getCacheDir();
        settings.setAppCachePath(cacheDir.getAbsolutePath());

I don't remember why I have that cache dir stuff, looks like I've had it since I started working on this app, I probably copied it from some sample. 

Deleted: webviewreferrer.zip 21.4 MB

webviewreferrer.zip 21.4 MB Download

That's really weird; I have no idea why appcache would affect it. Does your site actually use AppCache?

Not my site :)

Whatever site you're loading when you test it, I mean.

How can I figure out if they are using AppCache? 

I should mention I have seen this issue before with other sites, but I never really focused on it, I had just assumed referrer wasn't supposed to work on some cases.

The referrer header is not sent in all cases (there's a lot of different conditions on this), but if you see it being sent in devtools with the actual request but it's sometimes missing in shouldInterceptRequest that seems like a bug.

Ok, I'll keep an eye out for that. So the lines that I removed to get it to work (setting the AppCache directory), is that something my app needs or is enabling AppCache enough?

So despite figuring out the issue on the sample I created, on my own app I still have the issue even when I remove that piece of code. 

I did figure something else out though. On my own app, if I reload the webview by doing this:

 Bundle bundle = new Bundle();
 bundle.putByteArray("WEBVIEW_CHROMIUM_STATE", savedWebView.getData());
 view.restoreState(bundle);

Then the issue doesn't happen. I have no idea what this means, I just found it interesting. 

savedWebView.getData() is where I keep the bytes from this:

Bundle b = new Bundle();
webView.saveState(b);
byte[] bs = b.getByteArray("WEBVIEW_CHROMIUM_STATE");

Finally figured it out on my app, but I can't reproduce on the sample. 

Aside from removing the AppCache directory, on my app I had to comment out the setAppCacheEnabled(true) line. On the sample it doesn't matter if it is commented out or not. 

So my question now is, is this a bug (seems like one to me) and will it be fixed? For now I think I'll disable it on my code and let the user enable it if they need it, and will try it out with beta users. 

Does the site actually use appcache? If it doesn't then there's no reason why that should make a difference and that's definitely a bug; if it does, then it's possible that the cache refresh requests don't apply referer in the same way, but it'd still be surprising if shouldInterceptRequest differed from devtools..

Did you try the sample? 

I do not know if the site uses AppCache. Is there a way for me to check? 

Do I need to submit anything else for this issue? 
Thanks. 

Torne, any idea how we might verify if AppCache is used?

Poke around in devtools? Sorry, I have no idea :)

This may seem like a dumb question, but does it really matter whether the site is using AppCache? If enabling that on that site breaks the referrer header on shouldIntercept(), then it must be a bug right? Or are you saying the site is somehow breaking it? 

> then it must be a bug right? Or are you saying the site is somehow breaking it? 

This will narrow down our search, and we can verify if it's the site's bug or ours.

I had a look at the spec, and AppCache-originated network requests have no referrer - there's no sensible value for the header to have, so it doesn't have one. 

If the actual network requests in devtools show different behaviour to shouldInterceptRequest for the same header then that would be a bug; although, while devtools is open it's entirely possible the cache behaviour is different. There's definitely options in devtools to change caching logic, so you'd have to show that shouldInterceptRequest disagrees for the same actual network request, while devtools is observing it, not just the same resource being fetched at different times.

So, if the requests with no referrer stop when you disable appcache, I'm inclined to guess that actually everything here is working correctly and you were mistaken when you saw a difference between devtools and shouldInterceptRequest - if you can reproduce there being a difference then please let us know how so we can look into it, but for now I'm going to close this.

Here are screenshots of the requests side by side using the sample I sent (with a line added to output all headers). 

Deleted: Screenshot from 2017-08-08 17-02-39.png 1.2 MB

	Screenshot from 2017-08-08 17-02-39.png 1.2 MB View Download

Deleted: Screenshot from 2017-08-08 17-02-31.png 1.2 MB

	Screenshot from 2017-08-08 17-02-31.png 1.2 MB View Download

And if you disable appcache, the missing headers come back? If so then it would be really helpful to figure out whether this site is using it or not..

Here is a screenshot with this change:
          //  settings.setAppCachePath(cacheDir.getAbsolutePath());

Deleted: Screenshot from 2017-08-08 17-14-31.png 1.2 MB

	Screenshot from 2017-08-08 17-14-31.png 1.2 MB View Download

It seemed that we don't have issue on this, the refer header appears after disable the app cache.