Looked at the last 3. They are not actually issues with the test, but something much lower level that a whole bunch of other content java tests calls as well. So there is no point in disabling this particular test, and I'm guessing the same root cause affects a whole bunch of other tests as well.

cc jbudorick, yolandyan fyi

java.lang.RuntimeException: Could not launch intent Intent { act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] dat=file:///storage/emulated/0/chromium_tests_root/about:blank flg=0x14000000 cmp=org.chromium.content_shell_apk/.ContentShellActivity } within 45 seconds. Perhaps the main thread has not gone idle within a reasonable amount of time? There could be an animation or something constantly repainting the screen. Or the activity is doing network calls on creation? See the threaddump logs. For your reference the last time the event queue was idle before your activity launch request was 1512396762119 and now the last time the queue went idle was: 1512396762119. If these numbers are the same your activity might be hogging the event queue.
	at android.support.test.runner.MonitoringInstrumentation.startActivitySync(MonitoringInstrumentation.java:360)
	at android.support.test.rule.ActivityTestRule.launchActivity(ActivityTestRule.java:219)
	at org.chromium.content_shell_apk.ContentShellActivityTestRule.launchActivityWithIntentForTestCommon(ContentShellActivityTestRule.java:194)
	at org.chromium.content_shell_apk.ContentShellActivityTestRule.launchActivityWithIntentForTestCommon(ContentShellActivityTestRule.java:40)
	at org.chromium.content_shell_apk.ContentShellTestCommon.launchContentShellWithUrl(ContentShellTestCommon.java:82)
	at org.chromium.content_shell_apk.ContentShellTestCommon.launchContentShellWithUrlSync(ContentShellTestCommon.java:88)
	at org.chromium.content_shell_apk.ContentShellActivityTestRule.launchContentShellWithUrlSync(ContentShellActivityTestRule.java:78)
	at org.chromium.content.browser.ChildProcessLauncherIntegrationTest.testCrossDomainNavigationDoNotLoseImportance(ChildProcessLauncherIntegrationTest.java:109)

cc jbudorick, yolandyan for realz

fyi..

Disabling a test that has failed once with no retries in the last week seems ... ambitious. Perhaps @RetryOnFailure, though.

Will look a bit more later today.

From logcat, I *think* it's waiting on this thread:
12-04 14:13:27.160  6451  6478 E THREAD_STATE:   Thread[MonitoringInstrumentation,5,main]
12-04 14:13:27.160  6451  6478 E THREAD_STATE:     java.lang.Object.wait(Native Method)
12-04 14:13:27.160  6451  6478 E THREAD_STATE:     java.lang.Object.wait(Object.java:364)
12-04 14:13:27.160  6451  6478 E THREAD_STATE:     android.app.Instrumentation.startActivitySync(Instrumentation.java:403)
12-04 14:13:27.160  6451  6478 E THREAD_STATE:     android.support.test.runner.MonitoringInstrumentation.access$201(MonitoringInstrumentation.java:90)
12-04 14:13:27.160  6451  6478 E THREAD_STATE:     android.support.test.runner.MonitoringInstrumentation$5.call(MonitoringInstrumentation.java:351)
12-04 14:13:27.160  6451  6478 E THREAD_STATE:     android.support.test.runner.MonitoringInstrumentation$5.call(MonitoringInstrumentation.java:348)
12-04 14:13:27.160  6451  6478 E THREAD_STATE:     java.util.concurrent.FutureTask.run(FutureTask.java:237)
12-04 14:13:27.160  6451  6478 E THREAD_STATE:     java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
12-04 14:13:27.160  6451  6478 E THREAD_STATE:     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
12-04 14:13:27.160  6451  6478 E THREAD_STATE:     java.lang.Thread.run(Thread.java:841)

Looking at the code for Instrumentation.startActivitySync, it does wait until the UI queue is idle. And guess what, the thread dumps didn't dump the UI thread. Yay...

Here's from another run:
11-22 01:23:17.369 20143 20159 E THREAD_STATE:   Thread[main,5,main]
11-22 01:23:17.369 20143 20159 E THREAD_STATE:     android.os.MessageQueue.nativePollOnce(Native Method)
11-22 01:23:17.369 20143 20159 E THREAD_STATE:     android.os.MessageQueue.next(MessageQueue.java:138)
11-22 01:23:17.369 20143 20159 E THREAD_STATE:     android.os.Looper.loop(Looper.java:123)
11-22 01:23:17.369 20143 20159 E THREAD_STATE:     android.app.ActivityThread.main(ActivityThread.java:5001)
11-22 01:23:17.369 20143 20159 E THREAD_STATE:     java.lang.reflect.Method.invokeNative(Native Method)
11-22 01:23:17.369 20143 20159 E THREAD_STATE:     java.lang.reflect.Method.invoke(Method.java:515)
11-22 01:23:17.369 20143 20159 E THREAD_STATE:     com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:785)
11-22 01:23:17.369 20143 20159 E THREAD_STATE:     com.android.internal.os.ZygoteInit.main(ZygoteInit.java:601)
11-22 01:23:17.369 20143 20159 E THREAD_STATE:     dalvik.system.NativeStart.main(Native Method)

So UI thread is sitting there idle. Great job android.

I'm gonna add retries here unless someone objects?

no objections to @RetryOnFailure.

I don't mean add RetryOnFailure, I mean catch the RunTimeException and retry the launch in ContentShellActivityTestRule.launchActivityWithIntentForTestCommon

Add RetryOnFailure to a single test is meaning less. Pretty much all content instrumentation tests (and all instrumentation tests in general?) will actually launch an activity. And android has given us flaky code to write tests with...

argh, I see what you mean. :(

That sounds reasonable.

The @SmallTest timeout is 1min, and this timeout is 45s, so there is no point in retrying the launch, there is no time.

-> yoland to investigate?

This bug is caused by ContentViewCoreSelectionTest#testSelectActionBarSearchAndShareLaunchesNewTask before this test.

I am not able to repro this bug locally, but I compared the success run's log with 3 failure run, and in the failures, there is always an intent triggering AGSA (com.google.android.googlequicksearchbox/com.google.android.launcher.GEL).

Comparing the host script output, you would see that the error happens when testSelectActionBarSearchAndShareLaunchesNewTask runs before testCrossDomainNavigationDoNotLoseImportance. my guess would be search intent didn't trigger agsa until the next test already launched chrome. Which means, this bug might not just be limited to ChildProcessLauncherIntegrationTest#testCrossDomainNavigationDoNotLoseImportance

The fix should be blocking the search intent actually launching in testSelectActionBarSearchAndShareLaunchesNewTask

The VR tests seem to run into this (or a very similar issue with startActivitySync not being able to launch the activity) pretty regularly, maybe once a day or so. Not sure if there's anything they're doing that causes it to happen more frequently.

The three cases I've gone back and looked at all seem to be on the Pixel XLs with N and the Daydream View headset paired, although that configuration runs by far the most number of tests, so it's entirely possible that it's just more likely to show up there.

Sample failed build https://ci.chromium.org/buildbot/chromium.fyi/Android%20VR%20Tests/15458 and swarming task https://chromium-swarm.appspot.com/task?id=3b5e51e7d5d12410&refresh=10&show_raw=1