Proposed fix: https://codereview.chromium.org/2039773006/

Thank you for bringing this up, filed another issue to fix more tests: Issue 618282

Thanks pasko! :)

I hadn't noticed that other tests were having the same issue (this was the only one we had high flakyness on because of it), but it definitely makes sense to replace it everywhere.
There's no guarantees for onPageLoadFinished() ever happening after Tab.isLoading()

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/60a4046fb976a0dba8a2aeafcfeacdc8fe6e6f8f

commit 60a4046fb976a0dba8a2aeafcfeacdc8fe6e6f8f
Author: kraush <kraush@amazon.com>
Date: Wed Jun 08 14:56:18 2016

Wait for correct callback in CustomTabActivityTestBase

CustomTabActivityTestBase.startCustomTabActivityWithIntent should
check for onLoadStopped callbacks after verifying that
Tab.isLoading, since it is the only guaranteed callback.

BUG= 618064 

Review-Url: https://codereview.chromium.org/2039773006
Cr-Commit-Position: refs/heads/master@{#398553}

[modify] https://crrev.com/60a4046fb976a0dba8a2aeafcfeacdc8fe6e6f8f/chrome/android/javatests/src/org/chromium/chrome/browser/customtabs/CustomTabActivityTestBase.java

> There's no guarantees for onPageLoadFinished() ever happening after Tab.isLoading()

Is it not even guaranteed on simple webpages we have in tests? Is this fact documented somewhere?

It's not documented anywhere, I'm just looking at the code.
In general browsing in ChromePublic I can't reproduce this problem (I always get onPageLoadFinished when navigating to a website), but this test for some reason doesn't. :/
Purely looking at the code, onLoadStopped is definitely the safer bet though (It is set directly where mIsLoading gets set to false)

From what I can tell, something in the onPageLoadFinished behavior might have changed recently (we've seen this test failure for less than a month)
Since this is also only in CustomTabs: Do you know if CustomTabsActivity has a different handling of WebContentsObserverProxy (which is the one firing those events) ?
Maybe custom tabs (expectedly?) get less callbacks than tabs in a regular ChromeTabbedActivity.

In general it also looks to me that this could happen if the main frame already finished loading (which will trigger onPageLoadFinished, but at which point isLoading will still return true), but a subframe hasn't. (Final subframe finishing will trigger onLoadStopped)
Doesn't seem to be the root cause for this test, as from what I can tell it only has one frame, but it again makes me think onLoadStopped might be the safer bet.

I am not familiar with recent changes in onPageLoadFinished(). They are probably all unintentional.

I know that yusufo@ may have looked at areas close by recently for "tab reparenting" (i.e. customtabs "open in chrome" without reload).