New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 666356 link

Starred by 5 users

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Jan 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 1
Type: Bug

Blocked on:
issue 678146



Sign in to add a comment

Many mojo system tests are flaky

Project Member Reported by roc...@chromium.org, Nov 17 2016

Issue description

This issue appeared within the past few weeks. Tracking bug for all of them since they're likely the same underlying issue.
 

Comment 1 by roc...@chromium.org, Nov 17 2016

Cc: jbudorick@chromium.org roc...@chromium.org
 Issue 663998  has been merged into this issue.

Comment 2 by roc...@chromium.org, Nov 17 2016

 Issue 664662  has been merged into this issue.

Comment 3 by roc...@chromium.org, Nov 17 2016

 Issue 665925  has been merged into this issue.
Project Member

Comment 4 by chromium...@appspot.gserviceaccount.com, Nov 17 2016

Labels: Sheriff-Chromium
Detected 5 new flakes for test/step "AwakableListTest.KeepAwakablesReturningTrue". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyNgsSBUZsYWtlIitBd2FrYWJsZUxpc3RUZXN0LktlZXBBd2FrYWJsZXNSZXR1cm5pbmdUcnVlDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Comment 5 by roc...@chromium.org, Nov 17 2016

Cc: bcwh...@chromium.org
 Issue 665961  has been merged into this issue.

Comment 6 by roc...@chromium.org, Nov 17 2016

Labels: -Pri-3 OS-Android Pri-1
Issue appears to be Android only, which is a bit surprising.

Comment 7 by roc...@chromium.org, Nov 17 2016

There are no changes to mojo/edk which are even potentially relevant within the period of time the tests started flaking.

Furthermore the fact that the flake only appears on Android suggests that perhaps something changed in the Android testing framework, bot configuration, or related Android-only //base or //testing code which has caused this flake to appear.

jbudorick@ are you aware of any such changes in the last ~3 weeks that might somehow be relevant?

My initial hunch is that it could have something to do with how multiprocess tests  are run on Android, e.g. maybe we're flakily failing to launch a child process or something. Many of the flaking tests are not multiprocess tests, but it's possible that their failures are only a symptom of multiprocess tests hanging the test process.

Comment 8 by roc...@chromium.org, Nov 17 2016

Scratch that. I now suspect that some tests added by r429710 may be the source of flake, possibly because of bad interaction with named pipes on the filesystem. I am speculatively disabling them on Android only.
A little late now, but no, I'm not aware of android-specific test changes that would be relevant here. Glad to hear you have a suspect, though.
Project Member

Comment 10 by chromium...@appspot.gserviceaccount.com, Nov 17 2016

Detected 4 new flakes for test/step "OptionsValidationTest.Valid". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyJgsSBUZsYWtlIhtPcHRpb25zVmFsaWRhdGlvblRlc3QuVmFsaWQM. This message was posted automatically by the chromium-try-flakes app.
 Issue 666390  has been merged into this issue.
 Issue 666391  has been merged into this issue.
 Issue 666389  has been merged into this issue.
 Issue 666388  has been merged into this issue.
Project Member

Comment 15 by bugdroid1@chromium.org, Nov 17 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3bd8a572a0ab8ec0dd9df7195d4212ed86d1f6ae

commit 3bd8a572a0ab8ec0dd9df7195d4212ed86d1f6ae
Author: rockot <rockot@chromium.org>
Date: Thu Nov 17 18:40:45 2016

Mojo EDK: Speculatively disable peer connection tests on Android

Lots of flake appeared in the mojo_system_unittests suite on
Android bots after these tests were added.

This speculatively disables those tests on Android, where the
feature isn't used in production anyway.

BUG= 666356 
TBR=sammc@chromium.org

Review-Url: https://codereview.chromium.org/2505393003
Cr-Commit-Position: refs/heads/master@{#432925}

[modify] https://crrev.com/3bd8a572a0ab8ec0dd9df7195d4212ed86d1f6ae/mojo/edk/embedder/embedder_unittest.cc

Cc: -sa...@chromium.org
Labels: -Pri-1 Pri-3
Owner: sa...@chromium.org
It's quite likely that disabling the peer connection tests did resolve the Android flake, though I have no idea how or why they were the cause. I suspect it must have something to do with how named pipes are created for the tests, but that is wild speculation.

De-prioritizing and handing over to sammc@ - could you please look into this when you have some time?
Labels: -Sheriff-Chromium
Cc: -roc...@chromium.org sa...@chromium.org
Labels: -Pri-3 Pri-1
Owner: roc...@chromium.org
Aaaaand there's still flake. I'm stumped, as that was the only change to the EDK or EDK tests in the regression range.

Still unable to repro flake locally.
Project Member

Comment 19 by bugdroid1@chromium.org, Nov 18 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b202f5d598b5cb4e03ca662fe439d2f4363d82eb

commit b202f5d598b5cb4e03ca662fe439d2f4363d82eb
Author: rockot <rockot@chromium.org>
Date: Fri Nov 18 15:35:26 2016

Revert of Mojo EDK: Speculatively disable peer connection tests on Android (patchset #1 id:1 of https://codereview.chromium.org/2505393003/ )

Reason for revert:
Didn't fix the problem.

Original issue's description:
> Mojo EDK: Speculatively disable peer connection tests on Android
>
> Lots of flake appeared in the mojo_system_unittests suite on
> Android bots after these tests were added.
>
> This speculatively disables those tests on Android, where the
> feature isn't used in production anyway.
>
> BUG= 666356 
> TBR=sammc@chromium.org
>
> Committed: https://crrev.com/3bd8a572a0ab8ec0dd9df7195d4212ed86d1f6ae
> Cr-Commit-Position: refs/heads/master@{#432925}

TBR=sammc@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 666356 

Review-Url: https://codereview.chromium.org/2504193005
Cr-Commit-Position: refs/heads/master@{#433202}

[modify] https://crrev.com/b202f5d598b5cb4e03ca662fe439d2f4363d82eb/mojo/edk/embedder/embedder_unittest.cc

It seems that when we see these failures, it's always an entire batch of tests that fails, and it looks like the tests aren't even being run. I'm not sure how to properly interpret the output, but ultimately what I notice is something like:

I   71.150s run_tests_on_device(03d0bc5e006ade65)  Still working on StartInstrumentation(03d0bc5e006ade65, ... (etc)
ERROR:root:Forwarding signal 15 to test process
C  119.852s Main  Received SIGTERM. Stopping test execution.
C  119.854s Main  ********************************************************************************
C  119.855s Main  Detailed Logs
C  119.855s Main  ********************************************************************************
C  119.856s Main  [TIMEOUT] AwakableListTest.BasicAwakeSatisfied:
C  119.856s Main    Suite execution terminated, probably due to swarming timeout.
C  119.856s Main    Your test may not have run.
C  119.856s Main  [TIMEOUT] CoreTest.GetTimeTicksNow:

etc

And unlike all other devices, we never even see the start of some output from this device, e.g. the "Google Test filter = " line.

Any idea what sort of failures can lead to this behavior?
On android, we execute gtests in batches and, as you've noticed, currently lose the results for chunks in which a test hangs (as appears to be happening here), even for tests that ran before it. I'm working on a fix for that s.t. we'll get the output up until a test hangs.
Issue 666782 has been merged into this issue.
So I've managed to repro some flake on local Linux runs of some tests which use named pipes. There appears to be a race between named pipe creation and child process startup, because a child process may report something like:

[32256:32256:1118/125154:169662254048:ERROR:named_platform_handle_utils_posix.cc(95)] connect /tmp/55B9F2D4A06041AE6888EC521574131E: No such file or directory
[32256:32256:1118/125154:169662254156:FATAL:broker_posix.cc(70)] Check failed: sync_channel_.is_valid(). 
#0 0x7ff1a291ed4e base::debug::StackTrace::StackTrace()
#1 0x7ff1a29437ab logging::LogMessage::~LogMessage()
#2 0x7ff1a2e366aa mojo::edk::Broker::Broker()
#3 0x7ff1a2e5179f mojo::edk::NodeController::ConnectToParent()
#4 0x7ff1a2e3b43a mojo::edk::Core::InitChild()
#5 0x7ff1a2e65997 mojo::edk::SetParentPipeHandle()
#6 0x000000531b6b mojo::edk::test::MultiprocessTestHelper::ChildSetup()
#7 0x000000550637 multi_process_function_list::InvokeChildProcessTest()
#8 0x0000005199a0 base::TestSuite::Run()
#9 0x00000051ae5d base::(anonymous namespace)::LaunchUnitTestsInternal()
#10 0x00000051acd4 base::LaunchUnitTests()
#11 0x00000051055b main
#12 0x7ff1a1d47f45 __libc_start_main
#13 0x000000420e51 <unknown>

and then the test hangs, presumably because the named pipe is in working order and the parent is waiting for someone to connect to it.
Project Member

Comment 24 by bugdroid1@chromium.org, Nov 21 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f39ddac7b3677fc425fc89beba67000b59cbb9ad

commit f39ddac7b3677fc425fc89beba67000b59cbb9ad
Author: rockot <rockot@chromium.org>
Date: Mon Nov 21 00:07:10 2016

Mojo EDK: Fix race in named pipe creation for tests

This ensures named pipes are created before child process
launch, addressing a potential race where a child can attempt
to connect before a socket/handle is bound to the pipe path by
the server process.

BUG= 666356 
R=sammc@chromium.org

Review-Url: https://codereview.chromium.org/2515043002
Cr-Commit-Position: refs/heads/master@{#433464}

[modify] https://crrev.com/f39ddac7b3677fc425fc89beba67000b59cbb9ad/mojo/edk/test/multiprocess_test_helper.cc

Comment 25 by kbr@chromium.org, Nov 21 2016

Cc: kbr@chromium.org
Does the above patch address the kind of errors seen in this run of mojo_system_unittests?

https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/184487

The logs all say:

C  119.856s Main  [TIMEOUT] AwakableListTest.BasicAwakeSatisfied:
C  119.856s Main    Suite execution terminated, probably due to swarming timeout.
C  119.856s Main    Your test may not have run.
C  119.856s Main  [TIMEOUT] CoreTest.DataPipe:
C  119.856s Main    Suite execution terminated, probably due to swarming timeout.
C  119.856s Main    Your test may not have run.


which seems like a poor and confusing error message, since a Swarming timeout would definitely not allow the test harness to cooperatively report such errors -- it would be killed hard.

#25: swarming sends tasks a SIGTERM, waits a short length of time (30s?), then sends a SIGKILL (iirc). When we print that, it's because we received the SIGTERM from swarming and are trying to quickly shut down before we get killed.
I need help. I have a hammerhead running KTU84P. I can take the same batch of failed tests I see on a bot and throw it at the device, and in 1000 runs, they'll never hang.

If I run all 212 tests in the suite (which takes considerably longer), I can usually get a hang repro within ~50 runs, but it happens so infrequently that I have so far missed any opportunity to collect more information, such as which specific test is failing.

Is there some way I can ensure I'm locally mimicking the exact test environment as the bot (sans multiple devices)? Currently I'm just running run_mojo_system_unittests, optionally with some set of filters.
Cc: bpastene@chromium.org stip@chromium.org
+stip,bpastene for advice on reproducing a swarming bot locally.

The fix I mentioned in #21 is https://codereview.chromium.org/2514943002/. Once that lands, you should have an easier time at least seeing which specific test is failing.
 Issue 667554  has been merged into this issue.
 Issue 667029  has been merged into this issue.
re #27: How are you rerunning it? Swarming has instructions on rerunning the exact binaries under the "Reproducing the task locally" section on the task page. For instance, to try out https://chromium-swarm.appspot.com/task?id=329c978f64945010 (a failing mojo system tests), you can run "python swarming.py reproduce -S chromium-swarm.appspot.com 329c978f64945010"

That'll leave the test binaries and sources under work/ which you can then poke and prod
 Issue 666819  has been merged into this issue.
Maybe there's some side effect of the test process here (e.g. opening too many FDs or something?). It looks like the hang happens in any random multiprocess test, of which we have quite a few in this test suite.

There are two child processes below the main test process when hung:

The first one looks like some helper for launching a multiprocess test child. It appears to be waiting for the child pid:

#0  0x4004e3d8 in recvmsg () from /tmp/rockot-adb-gdb-libs/system/lib/libc.so
#1  0x7576406c in base::UnixDomainSocket::RecvMsgWithFlags (fd=fd@entry=46, buf=buf@entry=0x759d5008, length=length@entry=4096, flags=297, flags@entry=0, fds=fds@entry=0xbebc80f0, out_pid=out_pid@entry=0x0)
    at ../../base/posix/unix_domain_socket_linux.cc:127
#2  0x75764338 in base::UnixDomainSocket::RecvMsgWithPid (fd=fd@entry=46, buf=buf@entry=0x759d5008, length=length@entry=4096, fds=fds@entry=0xbebc80f0, pid=pid@entry=0x0)
    at ../../base/posix/unix_domain_socket_linux.cc:99
#3  0x75764348 in base::UnixDomainSocket::RecvMsg (fd=fd@entry=46, buf=buf@entry=0x759d5008, length=length@entry=4096, fds=fds@entry=0xbebc80f0) at ../../base/posix/unix_domain_socket_linux.cc:90
#4  0x759674ae in base::(anonymous namespace)::LaunchHelper::Recv (fd=46, buf=buf@entry=0x759d5008, fds=fds@entry=0xbebc80f0, this=<optimized out>) at ../../base/test/multiprocess_test_android.cc:179
#5  0x75967a32 in base::(anonymous namespace)::LaunchHelper::DoHelper (fd=<optimized out>, this=0x7599a828 <base::(anonymous namespace)::g_launch_helper+4>) at ../../base/test/multiprocess_test_android.cc:214
#6  base::(anonymous namespace)::LaunchHelper::Init (this=0x7599a828 <base::(anonymous namespace)::g_launch_helper+4>, main=main@entry=0x7590d5b5 <main(int, char**)>)
    at ../../base/test/multiprocess_test_android.cc:156
#7  0x75968094 in base::InitAndroidMultiProcessTestHelper (main=0x7590d5b5 <main(int, char**)>) at ../../base/test/multiprocess_test_android.cc:378
#8  0x7590d5d2 in main (argc=2, argv=0x759b5f38) at ../../mojo/edk/test/run_all_unittests.cc:31
#9  0x75964364 in testing::android::RunTests (obj=..., app_context=..., jtest_data_dir=..., jstdout_fifo=0 '\000', jstdout_file_path=..., jcommand_line_file_path=..., jcommand_line_flags=..., env=<optimized out>)
    at ../../testing/android/native_test/native_test_launcher.cc:136
#10 testing::android::Java_org_chromium_native_1test_NativeTest_nativeRunTests (env=<optimized out>, jcaller=<optimized out>, commandLineFlags=<optimized out>, commandLineFilePath=<optimized out>, 
    stdoutFilePath=0x79900025, stdoutFifo=0 '\000', appContext=0x2000029, testDataDir=0xc0f0002d) at gen/testing/android/native_test/native_test_jni_headers/testing/jni/NativeTest_jni.h:56
#11 0x4149cbd0 in dvmPlatformInvoke () from /tmp/rockot-adb-gdb-libs/system/lib/libdvm.so
#12 0x414cd126 in dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*) () from /tmp/rockot-adb-gdb-libs/system/lib/libdvm.so
#13 0x414a5fe4 in dvmJitToInterpNoChain () from /tmp/rockot-adb-gdb-libs/system/lib/libdvm.so


And the second one is the actual test client process, which is in turn blocking on... something?

#0  0x4004e904 in __futex_syscall3 () from /tmp/rockot-adb-gdb-libs/system/lib/libc.so
#1  0x4003bec8 in __pthread_cond_timedwait_relative () from /tmp/rockot-adb-gdb-libs/system/lib/libc.so
#2  0x4003bf28 in __pthread_cond_timedwait () from /tmp/rockot-adb-gdb-libs/system/lib/libc.so
#3  0x414d26b6 in ?? () from /tmp/rockot-adb-gdb-libs/system/lib/libdvm.so
#4  0x414d2c78 in dvmChangeStatus(Thread*, ThreadStatus) () from /tmp/rockot-adb-gdb-libs/system/lib/libdvm.so
#5  0x414c850a in ?? () from /tmp/rockot-adb-gdb-libs/system/lib/libdvm.so
#6  0x414c8c6a in ?? () from /tmp/rockot-adb-gdb-libs/system/lib/libdvm.so
#7  0x756c8612 in _JNIEnv::NewString (len=<optimized out>, unicodeChars=<optimized out>, this=0x4147cf70) at ../../third_party/android_tools/ndk/platforms/android-16/arch-arm/usr/include/jni.h:861
#8  (anonymous namespace)::ConvertUTF16ToJavaStringImpl (str=..., env=0x4147cf70) at ../../base/android/jni_string.cc:16
#9  base::android::ConvertUTF8ToJavaString (env=env@entry=0x4147cf70, str=...) at ../../base/android/jni_string.cc:74
#10 0x756c15f4 in base::android::OpenApkAsset (file_path=..., region=0x757e9050 <base::i18n::(anonymous namespace)::g_icudtl_region>) at ../../base/android/apk_assets.cc:25
#11 0x757ac288 in base::i18n::(anonymous namespace)::LazyInitIcuDataFile () at ../../base/i18n/icu_util.cc:91
#12 base::i18n::InitializeICU () at ../../base/i18n/icu_util.cc:277
#13 0x75964cae in base::test::InitializeICUForTesting () at ../../base/test/icu_test_util.cc:27
#14 0x75965768 in base::TestSuite::Initialize (this=0xbebc7f58) at ../../base/test/test_suite.cc:364
#15 0x759653ee in base::TestSuite::Run (this=0xbebc7f58) at ../../base/test/test_suite.cc:258



I notice ICU was rolled right around the time this flake appeared. Any ideas?
Project Member

Comment 34 by bugdroid1@chromium.org, Nov 22 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cf1d7d0b6ef21635ebaaa217389f108a3d6db6e8

commit cf1d7d0b6ef21635ebaaa217389f108a3d6db6e8
Author: rockot <rockot@chromium.org>
Date: Tue Nov 22 05:27:27 2016

Mojo EDK: Clean shutdown for ScopedIPCSupport in tests

This properly supports clean shutdown in test processes which
initialize the EDK, blocking ScopedIPCSupport destruction to wait
on EDK shutdown. Fixes some rare races in child process exit which
can cause premature message pipe breakage.

BUG= 666356 
R=yzshen@chromium.org
TBR=vabr@chromium.org (ICWYU)
TBR=ben@chromium.org (toplevel ICWYU)

Review-Url: https://codereview.chromium.org/2514093002
Cr-Commit-Position: refs/heads/master@{#433795}

[modify] https://crrev.com/cf1d7d0b6ef21635ebaaa217389f108a3d6db6e8/chrome/renderer/autofill/form_autocomplete_browsertest.cc
[modify] https://crrev.com/cf1d7d0b6ef21635ebaaa217389f108a3d6db6e8/chrome/renderer/chrome_render_frame_observer_browsertest.cc
[modify] https://crrev.com/cf1d7d0b6ef21635ebaaa217389f108a3d6db6e8/components/password_manager/content/renderer/credential_manager_client_browsertest.cc
[modify] https://crrev.com/cf1d7d0b6ef21635ebaaa217389f108a3d6db6e8/ipc/ipc_mojo_bootstrap_unittest.cc
[modify] https://crrev.com/cf1d7d0b6ef21635ebaaa217389f108a3d6db6e8/mojo/edk/test/run_all_unittests.cc
[modify] https://crrev.com/cf1d7d0b6ef21635ebaaa217389f108a3d6db6e8/mojo/edk/test/scoped_ipc_support.cc
[modify] https://crrev.com/cf1d7d0b6ef21635ebaaa217389f108a3d6db6e8/mojo/edk/test/scoped_ipc_support.h

 Issue 667630  has been merged into this issue.
 Issue 667839  has been merged into this issue.
All the hangs seem to be deadlocks inside dalvik, always in a test child process.
Cc: amistry@chromium.org roc...@chromium.org
 Issue 667848  has been merged into this issue.
At this point I'm fairly sure this is a side effect of the multiprocess test launcher using fork() on Android. This is unsafe to do since we're always in a threaded context.

There are several dalvik threads active at the time of fork() and if any of them have the heap lock acquired at that moment, the forked child will deadlock in allocation. Note that newer versions of dalvik don't have this heap lock, so that would explain why they don't repro.

Unfortunately I think the best thing to do will be to disable -all- our multiprocess tests on Android until process launch can be done properly (e.g. similar to how content does it) in base::MultiprocessTest.

Comment 40 by torne@chromium.org, Nov 24 2016

Cc: torne@chromium.org
Yes, you're correct. In a "normal" Android process (i.e. one containing the VM) it's not safe to fork() without exec(). It might happen to work or not in any particular case, but it's never safe on any OS version.

I don't think any other android tests depend on being able to do this.
Well I see two potential options.

1. Use a native binary for mojo_system_unittests and ipc_unittests so we can fork/exec like we do elsewhere on POSIX. I can't tell if use_raw_android_executable is actually useful here, and in any case it would still require some hacking to get working for unit tests.

2. Use isolated service processes to run test clients. This would be great but we block the main-thread Looper in unit tests, leaving us unable to bind services. I wonder if we could change the native test launcher so the main thread just loops Looper we run the TestSuite on a background thread.

Owner: jcivelli@chromium.org
Over to jcivelli who's going to continue the work proposed in c#41
Blockedon: 678146
Project Member

Comment 44 by bugdroid1@chromium.org, Jan 6 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5513e4622febbc838dfebddb51347ab797747836

commit 5513e4622febbc838dfebddb51347ab797747836
Author: jcivelli <jcivelli@chromium.org>
Date: Fri Jan 06 01:41:35 2017

Multiprocess test client: Android child process launcher rework.

Changing the way test start child processese on Android.
Forking natively should not be used on Andoid, instead child processes
should be started through the Android framework, which requires going
to Java and using Android services.

For that we introduce the MultiprocessTestClientLauncher java class
which starts a service in a new process. We provide several
MultiprocessTestClientService service classes to ensure we can have
multiple child processes.
File descriptor are passed from the parent to the child process as
Parcelable, and we use the GlobalDescriptors class to register them.

TEST=Run the mojo_system_unittests
BUG= 666356 

Review-Url: https://codereview.chromium.org/2549363004
Cr-Commit-Position: refs/heads/master@{#441799}

[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/android_webview/test/BUILD.gn
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/BUILD.gn
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/debug/proc_maps_linux_unittest.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/files/file_locking_unittest.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/posix/global_descriptors.h
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/process/process.h
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/process/process_posix.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/process/process_util_unittest.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/process/process_win.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/BUILD.gn
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/java/src/org/chromium/base/FileDescriptorInfo.aidl
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/java/src/org/chromium/base/FileDescriptorInfo.java
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/java/src/org/chromium/base/ITestClient.aidl
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/java/src/org/chromium/base/MainReturnCodeResult.aidl
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/java/src/org/chromium/base/MainReturnCodeResult.java
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/java/src/org/chromium/base/MultiprocessTestClientLauncher.java
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/java/src/org/chromium/base/MultiprocessTestClientService.java
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/java/src/org/chromium/base/MultiprocessTestClientService0.java
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/java/src/org/chromium/base/MultiprocessTestClientService1.java
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/java/src/org/chromium/base/MultiprocessTestClientService2.java
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/java/src/org/chromium/base/MultiprocessTestClientService3.java
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/java/src/org/chromium/base/MultiprocessTestClientService4.java
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/multiprocess_test_client_service.cc
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/multiprocess_test_client_service.h
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/test_support_jni_registrar.cc
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/android/test_support_jni_registrar.h
[add] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/main_stub.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/multiprocess_test.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/multiprocess_test.h
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/multiprocess_test_android.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/base/test/test_support_android.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/build/android/pylib/gtest/gtest_test_instance.py
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/components/test_runner/BUILD.gn
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/content/browser/memory/memory_coordinator_impl_unittest.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/content/shell/android/BUILD.gn
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/ipc/run_all_unittests.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/mojo/android/BUILD.gn
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/mojo/edk/embedder/platform_channel_pair.h
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/mojo/edk/embedder/platform_channel_pair_posix.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/mojo/edk/test/multiprocess_test_helper.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/mojo/edk/test/run_all_perftests.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/mojo/edk/test/run_all_unittests.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/net/android/BUILD.gn
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/testing/android/native_test/java/AndroidManifest.xml.jinja2
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/testing/android/native_test/native_test_launcher.cc
[modify] https://crrev.com/5513e4622febbc838dfebddb51347ab797747836/testing/test.gni

Project Member

Comment 45 by bugdroid1@chromium.org, Jan 10 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f4462a35264c65c6e554d22b00bb76c423c2e703

commit f4462a35264c65c6e554d22b00bb76c423c2e703
Author: jcivelli <jcivelli@chromium.org>
Date: Tue Jan 10 04:45:59 2017

Reland of new multiprocess test client process launcher.
Changed the way the native main function is invoked so that the main stub
is not required anymore. Launching in the sub-process is now done from Java
by calling into the new MainRunner class whose JNI code is in the same
module as the one used to start the test natively and is already linked
with main().

BUG= 666356 

Original issue's description:
> Multiprocess test client: Android child process launcher rework.
>
> Changing the way test start child processese on Android.
> Forking natively should not be used on Andoid, instead child processes
> should be started through the Android framework, which requires going
> to Java and using Android services.
>
> For that we introduce the MultiprocessTestClientLauncher java class
> which starts a service in a new process. We provide several
> MultiprocessTestClientService service classes to ensure we can have
> multiple child processes.
> File descriptor are passed from the parent to the child process as
> Parcelable, and we use the GlobalDescriptors class to register them.
>
> TEST=Run the mojo_system_unittests
> BUG= 666356 
> Review-Url: https://codereview.chromium.org/2549363004
> Cr-Commit-Position: refs/heads/master@{#441799}
> Committed: https://chromium.googlesource.com/chromium/src/+/5513e4622febbc838dfebddb51347ab797747836

Review-Url: https://codereview.chromium.org/2611323002
Cr-Commit-Position: refs/heads/master@{#442477}

[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/android_webview/test/BUILD.gn
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/BUILD.gn
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/debug/proc_maps_linux_unittest.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/files/file_locking_unittest.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/posix/global_descriptors.h
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/process/process.h
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/process/process_posix.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/process/process_util_unittest.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/process/process_win.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/BUILD.gn
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/android/java/src/org/chromium/base/FileDescriptorInfo.aidl
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/android/java/src/org/chromium/base/FileDescriptorInfo.java
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/android/java/src/org/chromium/base/ITestClient.aidl
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/android/java/src/org/chromium/base/MainReturnCodeResult.aidl
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/android/java/src/org/chromium/base/MainReturnCodeResult.java
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/android/java/src/org/chromium/base/MultiprocessTestClientLauncher.java
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/android/java/src/org/chromium/base/MultiprocessTestClientService.java
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/android/java/src/org/chromium/base/MultiprocessTestClientService0.java
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/android/java/src/org/chromium/base/MultiprocessTestClientService1.java
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/android/java/src/org/chromium/base/MultiprocessTestClientService2.java
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/android/java/src/org/chromium/base/MultiprocessTestClientService3.java
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/android/java/src/org/chromium/base/MultiprocessTestClientService4.java
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/multiprocess_test.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/multiprocess_test.h
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/multiprocess_test_android.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/base/test/test_support_android.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/build/android/pylib/gtest/gtest_test_instance.py
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/content/browser/memory/memory_coordinator_impl_unittest.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/ipc/run_all_unittests.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/mojo/android/BUILD.gn
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/mojo/edk/embedder/platform_channel_pair.h
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/mojo/edk/embedder/platform_channel_pair_posix.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/mojo/edk/test/multiprocess_test_helper.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/mojo/edk/test/run_all_perftests.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/mojo/edk/test/run_all_unittests.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/testing/android/native_test/BUILD.gn
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/testing/android/native_test/java/AndroidManifest.xml.jinja2
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/testing/android/native_test/java/src/org/chromium/native_test/MainRunner.java
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/testing/android/native_test/main_runner.cc
[add] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/testing/android/native_test/main_runner.h
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/testing/android/native_test/native_test_launcher.cc
[modify] https://crrev.com/f4462a35264c65c6e554d22b00bb76c423c2e703/testing/test.gni

Status: Fixed (was: Assigned)
Project Member

Comment 47 by bugdroid1@chromium.org, Aug 17

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c0eebdcaed9e23b05a88082e142f1ee3e6cd0f39

commit c0eebdcaed9e23b05a88082e142f1ee3e6cd0f39
Author: Ken Rockot <rockot@chromium.org>
Date: Fri Aug 17 01:39:46 2018

[mojo-core] Fix flaky shared buffer tests

This (probably) fixes rare flake in
SharedBufferTest.CreateAndPassReadOnlyBuffer and
SharedBufferBufferTest.CreateAndPassFromChildReadOnlyBuffer.

Right now the child process expects to receive a "quit" message
at the end, and then it fires an "ok" to acknowledge that. The parent
process sends "quit" and waits to hear "ok" before terminating
the test.

The problem is that the child process may terminate itself before
its IPC thread has had a chance to forward the outgoing "ok" message.
This is normal behavior for the system, and the test should not
rely on any contrary delivery guarantees.

This basically just swaps the order of operations: now the child sends
its "ok" and waits to read a "quit" message before terminating. The
parent waits to receive an "ok" and then fires a "quit" message before
waiting on the child process's death. Everything works out just fine.

This re-enables the latter test on Fuchsia, and both tests on Android
where they were disabled a long long time ago for unrelated and
no-longer-relevant reasons (see  https://crbug.com/666356 ).

Bug:  874719 , 666356 
Change-Id: I2977b5a00b5dd0f0653289fb550bac17cfbeb21a
Reviewed-on: https://chromium-review.googlesource.com/1179201
Reviewed-by: Reilly Grant <reillyg@chromium.org>
Commit-Queue: Ken Rockot <rockot@chromium.org>
Cr-Commit-Position: refs/heads/master@{#583929}
[modify] https://crrev.com/c0eebdcaed9e23b05a88082e142f1ee3e6cd0f39/mojo/core/shared_buffer_unittest.cc

Sign in to add a comment