Issue 664031  has been merged into this issue.

This is likely not about CoreTest.MessagePipe itself, it always fails with a cluster of other tests. Looking at the logs (one sample attached), there is a StartInstrumentation timeout:

Still working on StartInstrumentation(06ad3c9d003b6fe0, org.chromium.native_test/org.chromium.native_test.NativeTestInstrumentationTestRunner, retries=0, extras={'org.chromium.native_test.NativeTestInstrumentationTestRunner.ShardNanoTimeout': 120000000000, 'org.chromium.native_test.NativeTestInstrumentationTestRunner.NativeTestActivity': 'org.chromium.native_test.NativeUnitTestActivity', 'org.chromium.native_test.NativeTestInstrumentationTestRunner.TestList': '/data/local/tmp/temp_file-6ccbc7728de25'}, timeout=120.0, raw=False)
ERROR:root:Forwarding signal 15 to test process

@jbudorick -- git blame says you could know about NativeTestInstrumentationTestRunner. Would you mind triaging this bug further?

Thanks!
Today's sheriff

Deleted: Log File contents.html 201 KB

Log File contents.html 201 KB View Download

 Issue 664069  has been merged into this issue.

 Issue 664083  has been merged into this issue.

 Issue 664082  has been merged into this issue.

 Issue 664073  has been merged into this issue.

 Issue 664155  has been merged into this issue.

The timeout is the tests in a given shard taking too long (ie, the 30 tests that are marked TIMEOUT). Attempting to identify which one.

 Issue 664385  has been merged into this issue.

 Issue 664270  has been merged into this issue.

 Issue 664327  has been merged into this issue.

 Issue 664338  has been merged into this issue.

 Issue 664412  has been merged into this issue.

Detected 3 new flakes for test/step "MessagePipeTest.SharedBufferHandlePingPong". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyNQsSBUZsYWtlIipNZXNzYWdlUGlwZVRlc3QuU2hhcmVkQnVmZmVySGFuZGxlUGluZ1BvbmcM. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Detected 3 new flakes for test/step "AwakableListTest.KeepAwakablesReturningTrue". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyNgsSBUZsYWtlIitBd2FrYWJsZUxpc3RUZXN0LktlZXBBd2FrYWJsZXNSZXR1cm5pbmdUcnVlDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "OptionsValidationTest.Valid". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJgsSBUZsYWtlIhtPcHRpb25zVmFsaWRhdGlvblRlc3QuVmFsaWQM. This message was posted automatically by the chromium-try-flakes app.

 Issue 664784  has been merged into this issue.

 Issue 664785  has been merged into this issue.

 Issue 664786  has been merged into this issue.

Trying to determine the cause of timeout.
I investigated 18 failing builds, and there is no single test that failed in all of them, so there might be two or more tests with flaky timeout.

Either of the following two tests failed in all of the failing builds, so I'll speculatively disable them to see whether flakiness is resolved.
EmbedderTest.PipeSetup_LaunchFailure
MultiprocessMessagePipeTestWithPeerSupport.ChannelPipesWithMultipleChildren/0

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/aae520affa1d2c4397bf4d4758a9ff48d5a0cc88

commit aae520affa1d2c4397bf4d4758a9ff48d5a0cc88
Author: hiroshige <hiroshige@chromium.org>
Date: Mon Nov 14 09:26:47 2016

Disable two tests in mojo_system_unittests that are suspected to be timing out

BUG= 663998 
TBR=pdr@chromium.org, timloh@chromium.org

Review-Url: https://codereview.chromium.org/2497203002
Cr-Commit-Position: refs/heads/master@{#431847}

[modify] https://crrev.com/aae520affa1d2c4397bf4d4758a9ff48d5a0cc88/mojo/edk/embedder/embedder_unittest.cc
[modify] https://crrev.com/aae520affa1d2c4397bf4d4758a9ff48d5a0cc88/mojo/edk/system/multiprocess_message_pipe_unittest.cc

AFter #21, still many tests are timing out flakily:
https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_swarming_rel/builds/67021
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/180305

Thus reverting #21.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ba86cdb48c680aa78a142dee329d1c57ba22b00a

commit ba86cdb48c680aa78a142dee329d1c57ba22b00a
Author: hiroshige <hiroshige@chromium.org>
Date: Tue Nov 15 05:38:44 2016

Revert of Disable two tests in mojo_system_unittests that are suspected to be timing out (patchset #2 id:20001 of https://codereview.chromium.org/2497203002/ )

Reason for revert:
 Issue 663998  still persists after this CL.

Original issue's description:
> Disable two tests in mojo_system_unittests that are suspected to be timing out
>
> BUG= 663998 
> TBR=pdr@chromium.org, timloh@chromium.org
>
> Committed: https://crrev.com/aae520affa1d2c4397bf4d4758a9ff48d5a0cc88
> Cr-Commit-Position: refs/heads/master@{#431847}

TBR=
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 663998 

Review-Url: https://codereview.chromium.org/2504533002
Cr-Commit-Position: refs/heads/master@{#432117}

[modify] https://crrev.com/ba86cdb48c680aa78a142dee329d1c57ba22b00a/mojo/edk/embedder/embedder_unittest.cc
[modify] https://crrev.com/ba86cdb48c680aa78a142dee329d1c57ba22b00a/mojo/edk/system/multiprocess_message_pipe_unittest.cc

I've reproduced this flakily several times. The hangs appear to be happening inconsistently in various MessagePipeTests and MultiprocessMessagePipeTests. I'm still struggling to figure out why.

+rockot +yzshen in case they have ideas, as they landed changes to mojo/ on Thursday.

Possible trace:

(gdb) bt
#0  0x4005f904 in __futex_syscall3 () from /tmp/jbudorick-adb-gdb-libs/system/lib/libc.so
#1  0x4004cec8 in __pthread_cond_timedwait_relative () from /tmp/jbudorick-adb-gdb-libs/system/lib/libc.so
#2  0x4004cf28 in __pthread_cond_timedwait () from /tmp/jbudorick-adb-gdb-libs/system/lib/libc.so
#3  0x756d491a in base::ConditionVariable::Wait () at ../../base/synchronization/condition_variable_posix.cc:69
#4  0x75891646 in mojo::edk::Waiter::Wait () at ../../mojo/edk/system/waiter.cc:64
#5  0x758805d0 in mojo::edk::Core::WaitManyInternal () at ../../mojo/edk/system/core.cc:1174
#6  0x758806f4 in mojo::edk::Core::Wait () at ../../mojo/edk/system/core.cc:422
#7  0x7589249c in MojoWaitImpl () at ../../mojo/edk/embedder/entrypoints.cc:35
#8  0x758f80c0 in operator() () at ../../mojo/edk/system/multiprocess_message_pipe_unittest.cc:324
#9  0x758f86ac in StartClientWithHandler<mojo::edk::(anonymous namespace)::MultiprocessMessagePipeTest_SharedBufferPassing_Test::TestBody()::<lambda(MojoHandle, int*)> > () at ../../mojo/edk/test/mojo_test_base.h:76
#10 TestBody () at ../../mojo/edk/system/multiprocess_message_pipe_unittest.cc:366
#11 0x75925bd8 in testing::Test::Run () at ../../testing/gtest/src/gtest.cc:2475
#12 0x75925d0a in testing::TestInfo::Run () at ../../testing/gtest/src/gtest.cc:2656
#13 0x75925d92 in testing::TestCase::Run () at ../../testing/gtest/src/gtest.cc:2774
#14 0x75927698 in testing::internal::UnitTestImpl::RunAllTests () at ../../testing/gtest/src/gtest.cc:4647
#15 0x75922dc0 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> () at ../../testing/gtest/src/gtest.cc:2458
#16 testing::UnitTest::Run () at ../../testing/gtest/src/gtest.cc:4258
#17 0x75915efe in RUN_ALL_TESTS () at ../../testing/gtest/include/gtest/gtest.h:2237
#18 base::TestSuite::Run () at ../../base/test/test_suite.cc:246
#19 0x75917bd0 in base::internal::RunMixin<base::Callback<int (), (base::internal::CopyMode)1, (base::internal::RepeatMode)1> >::Run() const () at ../../base/callback.h:64
#20 0x75917d34 in LaunchUnitTestsInternal () at ../../base/test/launcher/unit_test_launcher.cc:187
#21 base::LaunchUnitTests(int, char**, base::Callback<int (), (base::internal::CopyMode)1, (base::internal::RepeatMode)1> const&) () at ../../base/test/launcher/unit_test_launcher.cc:458
#22 0x758be3d4 in main () at ../../mojo/edk/test/run_all_unittests.cc:53
#23 0x75914dfa in RunTests () at ../../testing/android/native_test/native_test_launcher.cc:136
#24 Java_org_chromium_native_1test_NativeTest_nativeRunTests () at gen/testing/android/native_test/native_test_jni_headers/testing/jni/NativeTest_jni.h:56

Well, that's interesting. I wonder if issue 664491 is somehow related.

Getting this on my radar, will look into it ASAP