Correction : This issue is broken in ‘M-62’ and below is the bisect info.

Using the per-revision bisect providing the bisect results,
Good build : 61.0.3163.0 (Revision:488529).
Bad build : 62.0.3164.0 (Revision:488823).

You are probably looking for a change made after 488591 (known good), but no later than 488592 (first known bad).

CHANGE LOG URL: 
https://chromium.googlesource.com/chromium/src/+log/14d17b4600f147a673a69f120fbf0f07349f7677..0646d7fcc6e6823408a067e07df1d7888f5fd698

Suspect : https://chromium.googlesource.com/chromium/src/+/0646d7fcc6e6823408a067e07df1d7888f5fd698

@yhirano : Could you please look into the issue, pardon me if it has nothing to do with your changes and if possible please assign it to concern owner.

Thank you!!

I can reproduce the issue with 488591.

I've confirmed this is due to PlzNavigate (--enable-browser-side-navigation).  That's currently enabled on Stable channel as well, so I can repro on 61.0.3163.79 on Chrome Stable for Windows.

The bug seems to affect navigations from any sad tab to any chrome:// URL.  The spinner starts and the tab goes white, but nothing ever seems to commit.  If you hit enter again on the URL while it's spinning, it loads successfully.

clamy@: Can you help investigate or find an owner?  I'm not sure this is a launch blocker since it's a bit of a corner case, but it is a regression that affects stable.

Ok so here is where I am at.

In terms of RFH, we're doing the following:
1) when we navigate to chrome://settings, we create a speculative RFH, along with its WebUI. However since the current RFH is not live, we immediately commit the speculative RFH along with its WebUI (see https://cs.chromium.org/chromium/src/content/browser/frame_host/render_frame_host_manager.cc?q=renderframehostmanager&dr=CSs&l=792). So the RFH + WebUI become the current one.
2) when the navigation is ready for commit, we look for a RFH to commit it. So we go through RFHM::GetFrameHostForNavigation a second time. There we mistakenly compute that we need to use a speculative RFH instead of the current one and try to commit in this speculative RFH. This happens because the code to determine whether we should swap RFH uses the current effective URL computed from the NavigationEntry (https://cs.chromium.org/chromium/src/content/browser/frame_host/render_frame_host_manager.cc?dr=CSs&l=1162). But in that case, it is not the effective URL of the current RFH, because the RFH is an empty speculative RFH and not the RFH that committed the effective URL (which in that case is the NTP).
3) We send the navigation to commit in this speculative RFH, except it never commits.

I suspect this is because at some point in this sequence we delete the WebUI  data source for the navigation to the WebUI. Therefore, when the renderer wants to access the body of the WebUI, it can't access it. I'm still tracking exactly what is happening with WebUIs along this sequence of RFH swaps.

I've confirmed that if during the SiteInstance computation we use the RenderFrameHost site instance URL when the RFH hasn't committed any URL instead of the URL of the last committed NavigationEntry, then we don't create a new speculative RFH at commit time and the navigation to the WebUI commits and completes properly.

Honestly, I'm not quite sure why we use the URL of the last committed NavigationEntry here, instead of the SiteInstance URL or the last committed URL for the RFH. I think we should change that. Wdyt?

Nice find!  Using the last committed NavigationEntry there predated RenderFrameHost having a last committed URL of its own (indeed, predated RFH entirely and I think goes back to when Chrome launched).  :)  At first glance, I agree that we should probably use the RFH's last committed URL if it exists, and if it has none, fall back to the SiteInstance's site URL as we do today when there's no entry.

(I suppose there's questions about the about:blank case and its effective origin, but we don't handle that today anyway in this case.)

FYI: I'm trying out a draft of the change here, just to see if it passes try jobs:
https://chromium-review.googlesource.com/c/chromium/src/+/667405

I can see about adding tests unless you've got one in progress already.

FYI: https://chromium-review.googlesource.com/c/chromium/src/+/667405 passed the try jobs (and resolved the issue in manual testing), but I had a full day and didn't have time to write tests for it or include a comment.  Feel free to take it over or I can try to get back to it tomorrow.  Thanks!

Ok, old code is what I suspected (it does have a TODO(clamy) that we should stop doing this which is true).

I didn't have time to look at the CL today - will do so on Monday.

FWIW, the following tests fails (hangs) at ToT (where PlzNavigate is on by default) and passes after applying the fix proposed in #c6 and #c7:

chrome/browser/chrome_navigation_browsertest.cc:

IN_PROC_BROWSER_TEST_F(ChromeNavigationBrowserTest, ChromeSchemeNavFromSadTab) {
  // Kill the renderer.
  content::RenderProcessHost* process = browser()
                                            ->tab_strip_model()
                                            ->GetActiveWebContents()
                                            ->GetMainFrame()
                                            ->GetProcess();
  content::RenderProcessHostWatcher crash_observer(
      process, content::RenderProcessHostWatcher::WATCH_FOR_PROCESS_EXIT);
  process->Shutdown(-1, true);
  crash_observer.Wait();

  // Attempt to navigate to a chrome://... URL.  This used to hang and never
  // commit in PlzNavigate mode -  https://crbug.com/764641 .
  ui_test_utils::NavigateToURL(browser(), GURL(chrome::kChromeUIVersionURL));
}

Thanks! I've uploaded a new version of the fix which adds this test.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b4f0224c393a7df73e3e035095075396f03252c8

commit b4f0224c393a7df73e3e035095075396f03252c8
Author: clamy <clamy@chromium.org>
Date: Tue Sep 19 12:58:27 2017

Use RFH's last successful URL for process swaps.

The last committed NavigationEntry is incorrect when we shortcut a
RenderFrameHost swap due to a sad tab.  It's also wrong for subframes.

BUG= 764641 
TEST=Navigate to a WebUI page from a sad tab (in PlzNavigate)

Cq-Include-Trybots: master.tryserver.chromium.linux:linux_site_isolation
Change-Id: I5580699f27389b10ecfd6d3cfe419c244ddf0a5e
Reviewed-on: https://chromium-review.googlesource.com/667405
Commit-Queue: Camille Lamy <clamy@chromium.org>
Reviewed-by: Charlie Reis (OOO until 9/25) <creis@chromium.org>
Reviewed-by: Łukasz Anforowicz <lukasza@chromium.org>
Cr-Commit-Position: refs/heads/master@{#502836}
[modify] https://crrev.com/b4f0224c393a7df73e3e035095075396f03252c8/chrome/browser/chrome_navigation_browsertest.cc
[modify] https://crrev.com/b4f0224c393a7df73e3e035095075396f03252c8/content/browser/frame_host/render_frame_host_manager.cc

This bug requires manual review: M62 has already been promoted to the beta branch, so this requires manual review
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), bhthompson@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Has this been confirmed and verified in Canary?

Ah sorry, prematurely approved it. 

abdulsyed@ - I've just verified in Canary - please see below:

$ git find-releases b4f0224c393a7df73e3e035095075396f03252c8
commit b4f0224c393a7df73e3e035095075396f03252c8 was:
  initially in 63.0.3220.0

I've just tried the repro steps on Canary on my Mac (which happens to be 63.0.3220.0 - for some reason my Mac doesn't yet see 63.0.3221.0).  I can confirm that the problem doesn't repro on this Canary.

Thanks - approving it for merge in M62. branch:3202

I can help with the merge (since clamy@ is OOO).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/50dacb670cb47fd9010a6f2da3ed8bc861482747

commit 50dacb670cb47fd9010a6f2da3ed8bc861482747
Author: Lukasz Anforowicz <lukasza@chromium.org>
Date: Fri Sep 22 20:49:36 2017

Use RFH's last successful URL for process swaps.

The last committed NavigationEntry is incorrect when we shortcut a
RenderFrameHost swap due to a sad tab.  It's also wrong for subframes.

BUG= 764641 
TEST=Navigate to a WebUI page from a sad tab (in PlzNavigate)
TBR=clamy@chromium.org

(cherry picked from commit b4f0224c393a7df73e3e035095075396f03252c8)

Cq-Include-Trybots: master.tryserver.chromium.linux:linux_site_isolation
Change-Id: I5580699f27389b10ecfd6d3cfe419c244ddf0a5e
Reviewed-on: https://chromium-review.googlesource.com/667405
Commit-Queue: Camille Lamy <clamy@chromium.org>
Reviewed-by: Charlie Reis (OOO until 9/25) <creis@chromium.org>
Reviewed-by: Łukasz Anforowicz <lukasza@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#502836}
Reviewed-on: https://chromium-review.googlesource.com/679269
Cr-Commit-Position: refs/branch-heads/3202@{#406}
Cr-Branched-From: fa6a5d87adff761bc16afc5498c3f5944c1daa68-refs/heads/master@{#499098}
[modify] https://crrev.com/50dacb670cb47fd9010a6f2da3ed8bc861482747/chrome/browser/chrome_navigation_browsertest.cc
[modify] https://crrev.com/50dacb670cb47fd9010a6f2da3ed8bc861482747/content/browser/frame_host/render_frame_host_manager.cc

I think this is now fixed, so marking it as such.