This is not just test failures. The behavior of Clank is also different.

Repro steps:
1. Visit https://output.jsbin.com/fibuhiq/1
2. Click the link (to https://output.jsbin.com/melofuj), and press "back" within 2 seconds
3. Click forward, and wait for 2 seconds

Without PlzNavigate, nothing happens, and you stay at https://output.jsbin.com/melofuj.
With PlzNavigate, the current URL becomes "intent://test/#Intent;scheme=test;package=com.chrome.test;end", and an error page of ERR_UNKNOWN_URL_SCHEME is shown.

adb log in the PlzNavigate case:
03-16 23:27:28.306 I/cr_UrlHandler(19484): shouldOverrideUrlLoading called on https://output.jsbin.com/fibuhiq/1 (get here through bookmark)
03-16 23:27:28.306 I/cr_UrlHandler(19484): NO_OVERRIDE: Navigation in background tab
03-16 23:27:32.630 I/cr_UrlHandler(19484): shouldOverrideUrlLoading called on https://output.jsbin.com/melofuj
03-16 23:27:32.649 I/cr_UrlHandler(19484): NO_OVERRIDE: No specialized handler for URL
03-16 23:27:33.947 I/cr_UrlHandler(19484): shouldOverrideUrlLoading called on https://output.jsbin.com/fibuhiq/1
03-16 23:27:33.947 I/cr_UrlHandler(19484): NO_OVERRIDE: Forward or back navigation
03-16 23:27:35.907 I/cr_UrlHandler(19484): shouldOverrideUrlLoading called on https://output.jsbin.com/melofuj
03-16 23:27:35.907 I/cr_UrlHandler(19484): NO_OVERRIDE: Forward or back navigation
03-16 23:27:38.208 I/cr_UrlHandler(19484): shouldOverrideUrlLoading called on intent://test/#Intent;scheme=test;package=com.chrome.test;end
03-16 23:27:38.211 I/cr_UrlHandler(19484): NO_OVERRIDE: RedirectHandler decision

Found the root cause.

https://cs.chromium.org/chromium/src/content/browser/frame_host/navigation_request.cc?l=359

// TODO(clamy): pass the real value for |is_external_protocol| if needed.

If intent:// is treated as external here, the following tests can pass, and the wrong behavior described in #c3 is also fixed.
org.chromium.chrome.browser.externalnav.UrlOverridingTest#testNavigationFromTimer
org.chromium.chrome.browser.externalnav.UrlOverridingTest#testNavigationFromXHRCallbackAndLongTimeout

As for testNavigationWithFallbackURL, the expected results (without plznavigate) should be:

03-17 03:55:28.419 I/cr_UrlHandler(28688): shouldOverrideUrlLoading called on http://127.0.0.1:54905/chrome/test/data/android/url_overriding/navigation_with_fallback_url.html?replace_text=UEFSQU1fRkFMTEJBQ0tfVVJM:aHR0cDovLzEyNy4wLjAuMTo1NDkwNS9jaHJvbWUvdGVzdC9kYXRhL2FuZHJvaWQvdXJsX292ZXJyaWRpbmcvaGVsbG8uaHRtbA==
03-17 03:55:28.419 I/cr_UrlHandler(28688): NO_OVERRIDE: RedirectHandler decision
03-17 03:55:28.419 E/weiyinchen(28688): result = NO_OVERRIDE
03-17 03:55:28.419 E/weiyinchen(28688): navigationParams.isExternalProtocol = false
03-17 03:55:28.579 E/weiyinchen(28688): onDidFinishNavigation isInMainFrame = true, errorCode = 0
03-17 03:55:28.739 E/weiyinchen(28688): onPageLoadFinished
03-17 03:55:28.985 I/cr_UrlHandler(28688): shouldOverrideUrlLoading called on intent://test/#Intent;scheme=test;package=com.chrome.test;S.browser_fallback_url=http://127.0.0.1:54905/chrome/test/data/android/url_overriding/hello.html;end
03-17 03:55:28.986 I/cr_UrlHandler(28688): OVERRIDE: clobberCurrentTab called
03-17 03:55:28.991 E/weiyinchen(28688): result = OVERRIDE_WITH_CLOBBERING_TAB
03-17 03:55:28.991 E/weiyinchen(28688): navigationParams.isExternalProtocol = true
03-17 03:55:29.016 E/weiyinchen(28688): onDidFinishNavigation isInMainFrame = true, errorCode = -3
03-17 03:55:29.016 E/weiyinchen(28688): onPageLoadFailed errorCode = -3
03-17 03:55:29.044 E/weiyinchen(28688): Expected: http://127.0.0.1:54905/chrome/test/data/android/url_overriding/hello.html actual: http://127.0.0.1:54905/chrome/test/data/android/url_overriding/hello.html
03-17 03:55:30.044 E/weiyinchen(28688): Expected: http://127.0.0.1:54905/chrome/test/data/android/url_overriding/hello.html actual: http://127.0.0.1:54905/chrome/test/data/android/url_overriding/hello.html
03-17 03:55:30.049 I/cr_UrlHandler(28688): shouldOverrideUrlLoading called on http://127.0.0.1:54905/chrome/test/data/android/url_overriding/hello.html
03-17 03:55:30.051 I/cr_UrlHandler(28688): NO_OVERRIDE: RedirectHandler decision
03-17 03:55:30.051 E/weiyinchen(28688): result = NO_OVERRIDE
03-17 03:55:30.051 E/weiyinchen(28688): navigationParams.isExternalProtocol = false
03-17 03:55:30.298 E/weiyinchen(28688): onDidFinishNavigation isInMainFrame = true, errorCode = -102
03-17 03:55:30.299 E/weiyinchen(28688): onPageLoadFailed errorCode = -102

With plznavigate, the sequence becomes:

03-17 04:33:58.334 I/cr_UrlHandler(  776): shouldOverrideUrlLoading called on http://127.0.0.1:40270/chrome/test/data/android/url_overriding/navigation_with_fallback_url.html?replace_text=UEFSQU1fRkFMTEJBQ0tfVVJM:aHR0cDovLzEyNy4wLjAuMTo0MDI3MC9jaHJvbWUvdGVzdC9kYXRhL2FuZHJvaWQvdXJsX292ZXJyaWRpbmcvaGVsbG8uaHRtbA==
03-17 04:33:58.334 I/cr_UrlHandler(  776): NO_OVERRIDE: RedirectHandler decision
03-17 04:33:58.334 E/weiyinchen(  776): result = NO_OVERRIDE
03-17 04:33:58.334 E/weiyinchen(  776): navigationParams.isExternalProtocol = false
03-17 04:33:58.448 E/weiyinchen(  776): onDidFinishNavigation isInMainFrame = true, errorCode = 0
03-17 04:33:58.605 E/weiyinchen(  776): onPageLoadFinished
03-17 04:33:58.797 I/cr_UrlHandler(  776): shouldOverrideUrlLoading called on intent://test/#Intent;scheme=test;package=com.chrome.test;S.browser_fallback_url=http://127.0.0.1:40270/chrome/test/data/android/url_overriding/hello.html;end
03-17 04:33:58.798 I/cr_UrlHandler(  776): OVERRIDE: clobberCurrentTab called
03-17 04:33:58.799 E/weiyinchen(  776): onDidFinishNavigation isInMainFrame = true, errorCode = 0
03-17 04:33:58.804 E/weiyinchen(  776): result = OVERRIDE_WITH_CLOBBERING_TAB
03-17 04:33:58.806 E/weiyinchen(  776): navigationParams.isExternalProtocol = true
03-17 04:33:58.814 I/cr_UrlHandler(  776): shouldOverrideUrlLoading called on http://127.0.0.1:40270/chrome/test/data/android/url_overriding/hello.html
03-17 04:33:58.815 I/cr_UrlHandler(  776): NO_OVERRIDE: RedirectHandler decision
03-17 04:33:58.815 E/weiyinchen(  776): result = NO_OVERRIDE
03-17 04:33:58.815 E/weiyinchen(  776): navigationParams.isExternalProtocol = false
03-17 04:33:58.901 E/weiyinchen(  776): onDidFinishNavigation isInMainFrame = true, errorCode = 0
03-17 04:33:58.929 E/weiyinchen(  776): onPageLoadFinished

Note that the error code became 0, and onPageLoadFailed() and onPageLoadFailed() should have been called.

clamy@, Could you take a look and complete the implementation of is_external_protocol signal? I'll continue looking into other test failures tracked in this bug. Thanks!

Knowing whether a url is handed externally can only be done on the IO thread using net::URLRequestContext::job_factory()->IsHandledURL().

Since navigation requests are started on the UI thread and need this information to give to the navigation throttles also on the same thread, seems like there's no way to avoid a hop to the IO thread and back initially. Camille, am I missing anything?

I just found out that ContentBrowserClient::IsHandledURL provides this, although using it in https://codereview.chromium.org/2760463005/ passes tests except for chrome_public_test_apk which hangs.

mariakhomenko@, should a navigation with fallback URL trigger onPageLoadFailed() like in #c5? I'm thinking whether we should retain this behavior with PlzNavigate or not. Without PlzNavigate, WebLocalFrameImpl::didFail leads to error code -3. With PlzNavigate, didFail is not called.

I would assume that PageLoadFailed() occurs for the original intent:// URL that was sent and then succeeds for the fallback URL we load.

mariakhomenko@, Makes sense. This CL https://codereview.chromium.org/2763843002/ makes sure the returned error status is expected. It didn't check the success status of the fallback URL.

jam@, net::URLRequestContext::job_factory()->IsHandledURL() is exactly how is_external_protocol is set without PlaNavigate. I'm looking into why https://codereview.chromium.org/2760463005/ causes assertion failures in other tests.

@wychen: comment 8 says ContentBrowserClient::IsHandledURL(), which is slihgtly different from net::URLRequestContext::job_factory()->IsHandledURL().. I suspect they're supposed to behave the same, other than per-storage partition differences which shouldn't make a difference here.

FYI in today's PlzNavigate meeting Camille said she'll look into this, so please coordinate with her.

ChromeContentBrowserClient::IsHandledURL() contains a large list in ProfileIOData::IsHandledProtocol(). I haven't checked how URLRequestJobFactoryImpl::IsHandledURL() behaves differently.

As for the assertion failure after the fix https://codereview.chromium.org/2760463005/,  I can locally reproduce it.

03-20 16:36:12.053 W/System.err(  820): java.lang.AssertionError
03-20 16:36:12.054 W/System.err(  820):         at org.chromium.chrome.browser.metrics.UmaUtils.getForegroundStartTime(UmaUtils.java:96)
03-20 16:36:12.054 W/System.err(  820):         at org.chromium.chrome.browser.ChromeActivity.postDeferredStartupIfNeeded(ChromeActivity.java:1984)
03-20 16:36:12.054 W/System.err(  820):         at org.chromium.chrome.browser.ChromeActivity$6.onLoadStopped(ChromeActivity.java:582)
03-20 16:36:12.054 W/System.err(  820):         at org.chromium.chrome.browser.tab.Tab.onLoadStopped(Tab.java:1505)

Looks like the key difference is that we need to treat chrome-native as non-external scheme as well.

For the following two tests, treating "intent" as external scheme is enough. No other side effects on other tests.
UrlOverridingTest#testNavigationFromTimer
UrlOverridingTest#testNavigationFromXHRCallbackAndLongTimeout

For test TabsOpenedFromExternalAppTest#testHttpsReferrerFromFirstPartyNoDowngrade, it's a DCHECK due to having https referer in http request.

Thanks for tracking that!

TabsOpenedFromExternalAppTest#testHttpsReferrerFromFirstPartyNoDowngrade should also be fixed now by Camille's https://codereview.chromium.org/2764363002/ which just landed 2 hours ago (haven't had a chance to verify though).

https://codereview.chromium.org/2764363002/ indeed fixed TabsOpenedFromExternalAppTest#testHttpsReferrerFromFirstPartyNoDowngrade.

https://codereview.chromium.org/2760463005/ fixed the following tests:
UrlOverridingTest#testNavigationFromXHRCallbackAndLongTimeout
externalnav.UrlOverridingTest#testNavigationFromTimer

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ffb0ffaccfcded76b0165c24c6ba0c8c4712089c

commit ffb0ffaccfcded76b0165c24c6ba0c8c4712089c
Author: wychen <wychen@chromium.org>
Date: Fri Mar 24 01:21:37 2017

Use stricter checking in UrlOverridingTest

Check the callback counts in all the test cases, and wait for the
fallback navigation to avoid early abortion.

The failures with PlzNavigate are not fixed yet.

BUG= 699388 

Review-Url: https://codereview.chromium.org/2763843002
Cr-Commit-Position: refs/heads/master@{#459325}

[modify] https://crrev.com/ffb0ffaccfcded76b0165c24c6ba0c8c4712089c/chrome/android/javatests/src/org/chromium/chrome/browser/externalnav/UrlOverridingTest.java

The last failure can be fixed by this CL:
https://codereview.chromium.org/2769373002/

I'm pretty sure it's not the best solution, so feel free to rewrite it in a proper way.

Sorry wrong trace, reposting the comment.
------------

I suspect the CL from #c21 to be causing test failures on the tablet testers:

C 1717.593s Main  [FAIL] org.chromium.chrome.browser.externalnav.UrlOverridingTest#testOpenWindowFromUserGesture:
C 1717.594s Main  junit.framework.AssertionFailedError
C 1717.594s Main  	at org.chromium.chrome.browser.externalnav.UrlOverridingTest.loadUrlAndWaitForIntentUrl(UrlOverridingTest.java:174)
C 1717.594s Main  	at org.chromium.chrome.browser.externalnav.UrlOverridingTest.testOpenWindowFromUserGesture(UrlOverridingTest.java:332)
C 1717.594s Main  	at java.lang.reflect.Method.invokeNative(Native Method)
C 1717.594s Main  	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
C 1717.594s Main  	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
C 1717.594s Main  	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
C 1717.594s Main  	at org.chromium.chrome.test.ChromeActivityTestCaseBase.runTest(ChromeActivityTestCaseBase.java:752)
C 1717.594s Main  	at org.chromium.base.test.BaseTestResult.runParameterized(BaseTestResult.java:161)
C 1717.594s Main  	at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:124)
C 1717.594s Main  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
C 1717.594s Main  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
C 1717.594s Main  	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
C 1717.594s Main  	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)

https://uberchromegw.corp.google.com/i/chromium.android/builders/KitKat%20Tablet%20Tester/builds/7073
https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Tablet%20Tester/builds/7271
https://uberchromegw.corp.google.com/i/chromium.android/builders/Marshmallow%20Tablet%20Tester/builds/7871

wychen@ Can you please look into it?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0f7b1afa9784fa186cc18dae4bb43c096dc6616b

commit 0f7b1afa9784fa186cc18dae4bb43c096dc6616b
Author: dgn <dgn@chromium.org>
Date: Fri Mar 24 14:22:56 2017

Revert of Use stricter checking in UrlOverridingTest (patchset #3 id:40001 of https://codereview.chromium.org/2763843002/ )

Reason for revert:
Breaking tablet testers:

[FAIL] org.chromium.chrome.browser.externalnav.UrlOverridingTest#testOpenWindowFromUserGesture:
junit.framework.AssertionFailedError
	at org.chromium.chrome.browser.externalnav.UrlOverridingTest.loadUrlAndWaitForIntentUrl(UrlOverridingTest.java:174)
	at org.chromium.chrome.browser.externalnav.UrlOverridingTest.testOpenWindowFromUserGesture(UrlOverridingTest.java:332)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
	at org.chromium.chrome.test.ChromeActivityTestCaseBase.runTest(ChromeActivityTestCaseBase.java:752)
	at org.chromium.base.test.BaseTestResult.runParameterized(BaseTestResult.java:161)
	at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:124)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)

See bug for builder links.

Original issue's description:
> Use stricter checking in UrlOverridingTest
>
> Check the callback counts in all the test cases, and wait for the
> fallback navigation to avoid early abortion.
>
> The failures with PlzNavigate are not fixed yet.
>
> BUG= 699388 
>
> Review-Url: https://codereview.chromium.org/2763843002
> Cr-Commit-Position: refs/heads/master@{#459325}
> Committed: https://chromium.googlesource.com/chromium/src/+/ffb0ffaccfcded76b0165c24c6ba0c8c4712089c

TBR=mariakhomenko@chromium.org,wychen@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 699388 

Review-Url: https://codereview.chromium.org/2769413002
Cr-Commit-Position: refs/heads/master@{#459416}

[modify] https://crrev.com/0f7b1afa9784fa186cc18dae4bb43c096dc6616b/chrome/android/javatests/src/org/chromium/chrome/browser/externalnav/UrlOverridingTest.java

Some more traces on debugging why testNavigationWithFallbackURL doesn't return ERR_ABORTED

testNavigationFromUserGesture

03-24 12:32:41.841 E/chromium( 4200): [ERROR:navigation_request.cc(362)] weiyinchen NavigationRequest::BeginNavigation
03-24 12:32:41.841 E/chromium( 4200): [ERROR:intercept_navigation_throttle.cc(70)] weiyinchen CheckIfShouldIgnoreNavigation ext = 1 is_redirect = 0 run_callback_synchronously_ = 0
03-24 12:32:41.843 E/chromium( 4200): [ERROR:intercept_navigation_delegate.cc(44)] weiyinchen CheckIfShouldIgnoreNavigationOnUIThread ext = 1
03-24 12:32:41.844 I/cr_UrlHandler( 4200): shouldOverrideUrlLoading called on intent://test/#Intent;scheme=test;package=com.chrome.test;end
03-24 12:32:41.846 I/cr_UrlHandler( 4200): OVERRIDE_WITH_EXTERNAL_INTENT: Intent to Play Store
03-24 12:32:41.846 E/weiyinchen( 4200): result = OVERRIDE_WITH_EXTERNAL_INTENT
03-24 12:32:41.846 E/weiyinchen( 4200): navigationParams.isExternalProtocol = true
03-24 12:32:41.846 E/chromium( 4200): [ERROR:intercept_navigation_throttle.cc(31)] weiyinchen RunCallback should_ignore_navigation = 1
03-24 12:32:41.846 E/chromium( 4200): [ERROR:intercept_navigation_throttle.cc(130)] weiyinchen InterceptNavigationThrottle::OnAsynchronousChecksPerformed should_ignore_navigation = 1
03-24 12:32:41.846 E/chromium( 4200): [ERROR:navigation_request.cc(594)] weiyinchen NavigationRequest::OnRequestFailed
03-24 12:32:41.849 E/chromium( 4200): [ERROR:web_contents_impl.cc(3379)] weiyinchen WebContentsImpl::DidFinishNavigation intent://test/#Intent;scheme=test;package=com.chrome.test;end err = -3

testNavigationWithFallbackURL

03-24 12:34:25.922 E/chromium( 4805): [ERROR:navigation_request.cc(362)] weiyinchen NavigationRequest::BeginNavigation
03-24 12:34:25.922 E/chromium( 4805): [ERROR:intercept_navigation_throttle.cc(70)] weiyinchen CheckIfShouldIgnoreNavigation ext = 1 is_redirect = 0 run_callback_synchronously_ = 0
03-24 12:34:25.929 E/chromium( 4805): [ERROR:intercept_navigation_delegate.cc(44)] weiyinchen CheckIfShouldIgnoreNavigationOnUIThread ext = 1
03-24 12:34:25.929 I/cr_UrlHandler( 4805): shouldOverrideUrlLoading called on intent://test/#Intent;scheme=test;package=com.chrome.test;S.browser_fallback_url=http://127.0.0.1:58779/chrome/test/data/android/url_overriding/hello.html;end
03-24 12:34:25.931 I/cr_UrlHandler( 4805): OVERRIDE: clobberCurrentTab called
03-24 12:34:25.932 E/chromium( 4805): [ERROR:web_contents_impl.cc(3379)] weiyinchen WebContentsImpl::DidFinishNavigation intent://test/#Intent;scheme=test;package=com.chrome.test;S.browser_fallback_url=http://127.0.0.1:58779/chrome/test/data/android/url_overriding/hello.html;end err = 0
03-24 12:34:25.936 E/weiyinchen( 4805): result = OVERRIDE_WITH_CLOBBERING_TAB
03-24 12:34:25.936 E/weiyinchen( 4805): navigationParams.isExternalProtocol = true
03-24 12:34:25.936 E/chromium( 4805): [ERROR:intercept_navigation_throttle.cc(31)] weiyinchen RunCallback should_ignore_navigation = 1

I think this might be a racing condition. See the working case (testNavigationFromUserGesture), "RunCallback should_ignore_navigation = 1" happens before "WebContentsImpl::DidFinishNavigation", so the error code is -3. For the failed case (testNavigationWithFallbackURL), "RunCallback should_ignore_navigation = 1" happens after "WebContentsImpl::DidFinishNavigation", so the error code is 0.

Thanks for the help in tracking this down. Ted had a theory that it's because we call LoadURL before the previous load is cancelled, and it turned out to be true. The trivial fix is in https://codereview.chromium.org/2768873006/

Abandoned CLs just for reference:
https://codereview.chromium.org/2770663002/
https://codereview.chromium.org/2765213003/
https://codereview.chromium.org/2769373002/
https://codereview.chromium.org/2772583006/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1e31d646391e0b4a897025d83a4d7ba5fdbbc175

commit 1e31d646391e0b4a897025d83a4d7ba5fdbbc175
Author: wychen <wychen@chromium.org>
Date: Thu Apr 13 06:18:05 2017

(reland) Use stricter checking in UrlOverridingTest

Check the callback counts in all the test cases, and wait for the
fallback navigation to avoid early abortion.

The failures with PlzNavigate are not fixed yet.

BUG= 699388 

Review-Url: https://codereview.chromium.org/2770193004
Cr-Commit-Position: refs/heads/master@{#464316}

[modify] https://crrev.com/1e31d646391e0b4a897025d83a4d7ba5fdbbc175/chrome/android/javatests/src/org/chromium/chrome/browser/externalnav/UrlOverridingTest.java

Apologies, applied the wrong component in bulk.