Note that this is an issue for any extension using chrome.webRequest because it makes determining the origin of a request unreliable. chrome.webRequest only delivers a tabID/frameID combination and it is up to the extension to know the page URL - normally by listening for chrome.webNavigation events. The usual assumption is that navigation only occurs if chrome.webNavigation.onCommitted event fires - that's the point where the old page is replaced by the new one and when extensions should update the URL in their data. However, it seems that for requests made between chrome.webNavigation.onBeforeNavigation and chrome.webNavigation.onCommitted distinguishing between old and new page currently isn't possible.

The documentation warns about this:[1]

=== start of blockquote ===
   There is no defined ordering between events of the webRequest API and the events of the webNavigation API. It is possible that webRequest events are still received for frames that already started a new navigation, or that a navigation only proceeds after the network resources are already fully loaded.

   In general, the webNavigation events are closely related to the navigation state that is displayed in the UI, while the webRequest events correspond to the state of the network stack which is generally opaque to the user.
=== end of blockquote ===

[1] https://developer.chrome.com/extensions/webNavigation#relation_to_webRequest

Adding label TE-NeedsTriageHelp for further investigation.

Thanks...!!

Yes, the current implementation matches the documentation. So it is not a bug per se. However, with these semantics, webNavigation.onComitted is mostly useless, as there is no event that indicates when a tab/frame switches to a new document. This, however, is necessary to reliably determine the context of a request (in Adblock Plus and other extensions). Since we cannot rely on onCommitted because it is fired too late, we now have to evaluate the headers of each document ourselves, simulating Chrome's behavior. This behavior is quite complex, and potentially inconsistent between browsers and browser versions. So this is a rather complicated and error-prone hack, which onCommitted seems to exists for, to avoid, in the first place.

>What is the expected behavior?
>onBeforeRequest events for requests made by pages and frames should never be fired before the onCommitted event was fired.

Why? Prefetching is enabled by default in the browser so the expected behavior is for the requests to fire before onCommitted.

>there is no event that indicates when a tab/frame switches to a new document

chrome.tabs.onReplaced is fired when a tab is replaced with another tab due to prerendering or instant.

This has nothing to do with prefetching. The prefetched document just gets a new tabId, and for that tabId, onComitted ideally would be fired before any onBeforeRequest event. But in the case we discussed so far, prefetching isn't even involved, but still requests are reported before onComitted.

>But in the case we discussed so far, prefetching isn't even involved

Does it mean you've disabled "Use a prediction service to load pages more quickly" option?

>The prefetched document just gets a new tabId, and for that tabId, onComitted ideally would be fired before any onBeforeRequest event.

Just to clarify, for prefetched tabs onCommitted can't be fired before webRequest.onBeforeRequest event:

1. A prefetched tab's id initially is -1 
2. it starts loading the page so webRequest events like onBeforeRequest are fired
3. webNavigation events aren't yet fired because the hidden tab isn't yet in the tabstrip [1]
4. the browser decides to show the tab (e.g. user confirms the url input in the address bar)
5. the hidden tab is added to the tab strip, gets a new valid id, replaces the currently displayed tab, webNavigation.onBeforeNavigate and tab.onReplaced events are fired

  [1]: https://cs.chromium.org/chromium/src/chrome/browser/extensions/api/web_navigation/web_navigation_api.cc?l=267&rcl=ad993476e691b6352ae99e62df1ef2f01013d89b

Prefetching only occurs when you type a URL into the omnibox, which then gets auto-completed. Chrome doesn't randomly prefetches links or alike. That is why it's not relevant to the observations above.

Moreover, prefeteched tabs do have a proper ID, which is used by the webRequest API at least. Otherwise, tab.onReplaced would have no purpose, as it reports that ID along the previous ID of the tab.

You might be right though that webNavigation events aren't fired for prefetched/hidden tabs. I didn't check that. But if this the case, this would be yet another (huge) problem.

As pointed out above, it's an essential requirement for a wide range of extensions, to associate documents with requests. This requires an event that is fired for each new document (prefetched or not) before any request is sent.

Okay, let's look at chrome://tracing while clicking a random link on wikipedia:

1. a link was clicked
2. webNavigation.onBeforeNavigate is fired = navigation is initialized: "StartToCommit" [1]
3. webRequest.onBeforeRequest is fired for the clicked link request: "NotifyBeforeURLRequest"
4. data is received, header and data events are fired accordingly
5. webNavigation.onCommitted is fired in DidNavigate after the URL is checked by the safe web browsing filters and so on: "OnDidCommitProvisionalLoad"

So non-prefetched pages have at least one request to the page URL before onCommitted. If I understand the bug report correctly, the problem is not this initial request, but the requests for the stuff inside that page. At this point I could only guess that sometimes the page is parsed so fast by the background html parser (it runs in a different process as you can see on the tracing page) that it manages to initiate new requests before the main process sends onCommitted to an extension. I agree it'd be nice to have an extensions API to prevent that from happening. There are many users that would enable such option in an extension for privacy.

  [1]: https://cs.chromium.org/chromium/src/content/browser/frame_host/navigation_handle_impl.cc?l=75&rcl=7a4e3a4e6fa17894c3a4cdc0f03ed3380fe468f9

Deleted: tracing.png 55.0 KB

	tracing.png 55.0 KB View Download

> If I understand the bug report correctly, the problem is not this initial request, but the requests for the stuff inside that page. At this point I could only guess that sometimes the page is parsed so fast by the background html parser (it runs in a different process as you can see on the tracing page) that it manages to initiate new requests before the main process sends onCommitted to an extension.

Yes, that is the issue. I think cached secondary resources are more likely to load before onCommitted is fired.

I also had to deal with it for uBlock Origin:
https://github.com/gorhill/uBlock/issues/2053#issuecomment-252241737

The fix was mainly to rely on onBeforeRequest/main_frame to guess that a new top document is being loaded, and using onCommitted only to *confirm* that guess -- which is almost always true.