Tentatively suspecting https://codereview.chromium.org/2891343003 since it affects NTP and landed just before the first instance of flake.

There's a new CL about to land that re-works Chrome Home NTP creation https://codereview.chromium.org/2899053004/

If testCloseNTP_Incognito is still flaky after that patch lands, then we can disable it for now.

cc'ing mdjones@ since I'm OOO Thursday and Friday and this may require a quick CL to disable tests tomorrow (depending on when the other in-flight patch lands).

It looks like #testCloseNTP flaky on the Lollipop Phone tester too. I ran these locally and they are still flaking with my in-flight patch. I'm going to disable for now and work on re-enabling them on Tuesday when I return.

C  161.802s Main  [FAIL] org.chromium.chrome.browser.widget.bottomsheet.BottomSheetNewTabControllerTest#testCloseNTP_Incognito:
C  161.802s Main  java.lang.RuntimeException: Exception occured while waiting for runnable
C  161.802s Main  	at org.chromium.base.ThreadUtils.runOnUiThreadBlocking(ThreadUtils.java:81)
C  161.802s Main  	at org.chromium.chrome.browser.widget.bottomsheet.BottomSheetNewTabControllerTest.closeNewTab(BottomSheetNewTabControllerTest.java:201)
C  161.802s Main  	at org.chromium.chrome.browser.widget.bottomsheet.BottomSheetNewTabControllerTest.testCloseNTP_Incognito(BottomSheetNewTabControllerTest.java:167)
C  161.802s Main  	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
C  161.803s Main  	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
C  161.803s Main  	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
C  161.803s Main  	at org.chromium.chrome.test.ChromeActivityTestCaseBase.runTest(ChromeActivityTestCaseBase.java:382)
C  161.803s Main  	at org.chromium.base.test.BaseTestResult.runParameterized(BaseTestResult.java:161)
C  161.803s Main  	at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:124)
C  161.803s Main  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
C  161.803s Main  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
C  161.803s Main  	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555)
C  161.803s Main  	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1853)
C  161.803s Main  Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException: Attempt to invoke virtual method 'android.view.View org.chromium.chrome.browser.ntp.ChromeHomeNewTabPageBase.getCloseButtonForTests()' on a null object reference
C  161.803s Main  	at java.util.concurrent.FutureTask.report(FutureTask.java:93)
C  161.803s Main  	at java.util.concurrent.FutureTask.get(FutureTask.java:163)
C  161.803s Main  	at org.chromium.base.ThreadUtils.runOnUiThreadBlocking(ThreadUtils.java:79)
C  161.803s Main  	... 19 more
C  161.803s Main  Caused by: java.lang.NullPointerException: Attempt to invoke virtual method 'android.view.View org.chromium.chrome.browser.ntp.ChromeHomeNewTabPageBase.getCloseButtonForTests()' on a null object reference
C  161.803s Main  	at org.chromium.chrome.browser.widget.bottomsheet.BottomSheetNewTabControllerTest$5.run(BottomSheetNewTabControllerTest.java:204)
C  161.803s Main  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:422)
C  161.803s Main  	at java.util.concurrent.FutureTask.run(FutureTask.java:237)
C  161.803s Main  	at android.os.Handler.handleCallback(Handler.java:739)
C  161.803s Main  	at android.os.Handler.dispatchMessage(Handler.java:95)
C  161.803s Main  	at android.os.Looper.loop(Looper.java:135)
C  161.803s Main  	at android.app.ActivityThread.main(ActivityThread.java:5254)
C  161.803s Main  	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:903)
C  161.803s Main  	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:698)

It appears that when the tests fail the tab is still displaying about:blank when trying to close the ChromeHomeNewTabPage, which indicates a timing issue in the tests rather than a real failure.

The issue is that #loadChromeHomeNewTab() is using the wrong version ofChromeTabUtils#waitForTabPageLoaded(). The version being used passes "null" for the expected URL, so the tab finishes loading about:blank and we stop waiting; we should really be waiting for the tab to load the NTP.

I think it's as simple as replacing loadChromeHomeNewTab's current use with:

   ChromeTabUtils.loadUrlOnUiThread(tab, UrlConstants.NTP_URL);
   ChromeTabUtils.waitForTabPageLoaded(tab, UrlConstants.NTP_URL);


I'll send this patch out for review Tuesday: https://codereview.chromium.org/2900263003
It passed 51 local runs on a Lollipop phone.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8fb4b9b1913bdda4ad9a3fba78d2f73fbf52ff0f

commit 8fb4b9b1913bdda4ad9a3fba78d2f73fbf52ff0f
Author: twellington <twellington@chromium.org>
Date: Wed May 24 22:54:27 2017

[Home] Disable BottomSheetNewTabController#testCloseNtp*

These tests are for the old NTP UI that's being deprecated. Once the
new NTP design was enabled they started flaking on the Lollipop Phone
Tester. Disabling for now.

BUG= 726032 
TBR=mdjones@chromium.org

Review-Url: https://codereview.chromium.org/2907433002
Cr-Commit-Position: refs/heads/master@{#474462}

[modify] https://crrev.com/8fb4b9b1913bdda4ad9a3fba78d2f73fbf52ff0f/chrome/android/javatests/src/org/chromium/chrome/browser/widget/bottomsheet/BottomSheetNewTabControllerTest.java

Lowering re-enabling tests to P2

Thanks for the quick turn-around looking at this! Removing sheriff label. 

 Issue 726224  has been merged into this issue.

Findit identified the culprit r474525 with confidence 4.7% in the config "tryserver.chromium.android / linux_android_rel_ng"
based on the flakiness trend:

https://findit-for-me.appspot.com/waterfall/flake?key=ag9zfmZpbmRpdC1mb3ItbWVy-AELEhdNYXN0ZXJGbGFrZUFuYWx5c2lzUm9vdCLBAWNocm9taXVtLmxpbnV4L0FuZHJvaWQgVGVzdHMvNDIwNjMvY2hyb21lX3B1YmxpY190ZXN0X2FwayBvbiBBbmRyb2lkL2IzSm5MbU5vY205dGFYVnRMbU5vY205dFpTNWljbTkzYzJWeUxuZHBaR2RsZEM1aWIzUjBiMjF6YUdWbGRDNUNiM1IwYjIxVGFHVmxkRTVsZDFSaFlrTnZiblJ5YjJ4c1pYSlVaWE4wSTNSbGMzUkRiRzl6WlU1VVVBPT0MCxITTWFzdGVyRmxha2VBbmFseXNpcxgBDA

Automatically posted by the findit-for-me app (https://goo.gl/Ot9f7N).
Flake Analyzer is in alpha version.
Feedback is welcome using component Tools>Test>FindIt>Flakiness !

Please ignore the Findit result as it is a false positive due to a timeout of Swarming task to rerun the test.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b80ef087b4b54fedaa98da6ad34112280b5117a4

commit b80ef087b4b54fedaa98da6ad34112280b5117a4
Author: twellington <twellington@chromium.org>
Date: Tue May 30 17:51:46 2017

[Home] Fix BottomSheetNewTabController#testCloseNtp*

Previously the tests were not waiting for the correct URL to load
so sometimes the NTP wasn't loaded when trying to get the close
button.

BUG= 726032 

Review-Url: https://codereview.chromium.org/2900263003
Cr-Commit-Position: refs/heads/master@{#475589}

[modify] https://crrev.com/b80ef087b4b54fedaa98da6ad34112280b5117a4/chrome/android/javatests/src/org/chromium/chrome/browser/widget/bottomsheet/BottomSheetNewTabControllerTest.java