Hi Xing, David suggested assigning this to you. Thanks

Sure, thanks.

Upload a crash log for NetworkServiceBrowserTest.WebUIBindingsNoHttp, it seems the process is killed for some reason, still investigating.

Deleted: tomb.txt 123 KB

tomb.txt 123 KB View Download

Another log file, it seems the embedded test server can't retrieve the test cert on Android.

Deleted: adb_logcat.txt 81.1 KB

adb_logcat.txt 81.1 KB View Download

Still investigating,  the embedded test server seems to try to read root_ca_cert.pem for https connection and crash, this happens on some early mojo service manager calls. Probably the file isn't there.

It seems the early mojo call triggered the embedded test server to load from an incorrect path, 

Correct path of cert used by other content browsertests:
/storage/emulated/0/chromium_tests_root/net/data/ssl/certificates/root_ca_cert.pem

The mojo triggered cert path: net/data/ssl/certificates/root_ca_cert.pem.

If adb shell to the device, the cert file is in */chromium_tests_root/*, I guess there is some early Android setup code that should run after some mojo call, not sure still looking.

The issue is that getaddrinfo() returns EAI_NODATA, is the network process has enough permission to perform such syscalls?

There's a method IsUnsandboxedSandboxType() that should only define sandbox_network as a sandboxed type for linux,  not the other platforms.  It shouldn't be turned on for android.  Maybe I botched something with the ifdefs at https://cs.chromium.org/chromium/src/services/service_manager/sandbox/sandbox_type.cc?rcl=6fb985fcca5d982de74bd9faad387956562613cd&l=18

Is there any known workaround for this? I tried using --no-sandbox and --service-sandbox-type=none but still name resolution fails.

I have a workaround: https://chromium-review.googlesource.com/c/chromium/src/+/806800 that'll run the service in process. It's just waiting for a review, and should land today.

@tsepez: i believe this didn't work before any sandboxing changes, so it might not be related.

For the content test, android use base::InitAndroidTestPaths to hook up the test directory, it's called in the test launcher which is same as browser main process in test, and in ctor of EmbeddedTestServerImpl.

However the network service starts another process which probably don't have the test launcher code to hook up the directory, and try to read the cert file before any EmbeddedTestServer is created.

In out-of-process case, the problem is that on Android, the network service is using the same ChildProcessService that is used by all the renderer process, which is isolated and doesn't have any permissions to access network and disk storage.

I have a working patch for out-of-process case: 
https://chromium-review.googlesource.com/c/chromium/src/+/807152

I think qinmin@'s patch will fix a lot of issue.

For content test, the cert path issue still needs to be fixed, EmbeddedTestServer::RegisterTestCerts is introduced to fix some ssl error in mojo.fyi.network_content_browsertests.filter in bd131f2f5352b20469f0866e5377077967fa04a4.

However, it's a bit weird that EmbeddedTestServer needs to start in a network service process, which shouldn't depend on test contexts?

EmbeddedTestServer runs in the browser process always. EmbeddedTestServer::RegisterTestCerts doesn't start a server, it just registers the server's root certificate into net/ globals.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/adbbdb4cf60848abc9f5aa10d6e8e4b1dbf283b8

commit adbbdb4cf60848abc9f5aa10d6e8e4b1dbf283b8
Author: Xing Liu <xingliu@chromium.org>
Date: Fri Jan 05 02:11:08 2018

Add Mojo Android builder in FYI waterfall.

This CL adds a Android builder under FYI master.

This CL is associated with src side CL:
https://chromium-review.googlesource.com/c/chromium/src/+/847308

Bug:783374,796724
Change-Id: I5d14a45d2fe8f5c13fc66c926b218ec502a63d8b
Reviewed-on: https://chromium-review.googlesource.com/842476
Commit-Queue: Xing Liu <xingliu@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/adbbdb4cf60848abc9f5aa10d6e8e4b1dbf283b8/masters/master.tryserver.chromium.android/builders.pyl
[modify] https://crrev.com/adbbdb4cf60848abc9f5aa10d6e8e4b1dbf283b8/scripts/slave/recipe_modules/chromium_tests/trybots.py
[modify] https://crrev.com/adbbdb4cf60848abc9f5aa10d6e8e4b1dbf283b8/masters/master.chromium.fyi/master.cfg
[modify] https://crrev.com/adbbdb4cf60848abc9f5aa10d6e8e4b1dbf283b8/scripts/slave/recipe_modules/chromium_tests/chromium_fyi.py
[modify] https://crrev.com/adbbdb4cf60848abc9f5aa10d6e8e4b1dbf283b8/masters/master.chromium.fyi/slaves.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/49b8c0dbb7a4e20dfbcf6e442e3c9cbb237b6358

commit 49b8c0dbb7a4e20dfbcf6e442e3c9cbb237b6358
Author: Xing Liu <xingliu@chromium.org>
Date: Fri Jan 05 04:29:09 2018

Add Mojo Android builder in FYI master to run network service tests.

This is the src side CL to add Mojo Android builder under FYI master.
The builder will run network service related tests.

This CL is associated with CL:
https://chromium-review.googlesource.com/c/chromium/tools/build/+/842476/

Bug: 783374,796724
Change-Id: I41b6b047b9a1cd381fad22773572e38daa7a8ed6
Reviewed-on: https://chromium-review.googlesource.com/847308
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Xing Liu <xingliu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#527202}
[modify] https://crrev.com/49b8c0dbb7a4e20dfbcf6e442e3c9cbb237b6358/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/49b8c0dbb7a4e20dfbcf6e442e3c9cbb237b6358/testing/buildbot/test_suites.pyl
[modify] https://crrev.com/49b8c0dbb7a4e20dfbcf6e442e3c9cbb237b6358/testing/buildbot/waterfalls.pyl
[modify] https://crrev.com/49b8c0dbb7a4e20dfbcf6e442e3c9cbb237b6358/tools/mb/mb_config.pyl

+bpastene@ for trybot suggestions.

Hi Ben, thank you for helping me setup the trybot. Currently we can run 'Mojo Android' trybot, but it has build issue that I suspect some configuration is still missing or wrong.

Here is a link of failed builder:
https://luci-milo.appspot.com/buildbot/tryserver.chromium.android/android_mojo/6

Hrmm, that's a strange-looking error. My best guess is that you'd have to add the line "'gclient_apply_config': ['android']," to this dict:
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/chromium_fyi.py?rcl=8d30b2285801d896b09e01342a41557d6fde32ce&l=765

+jbud,hzl if they have any ideas

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/2d8e6bb694fafb432831a5f604a6e125ec6d5170

commit 2d8e6bb694fafb432831a5f604a6e125ec6d5170
Author: Xing Liu <xingliu@chromium.org>
Date: Wed Jan 17 01:56:23 2018

Fix an issue in Mojo Android builder.

Currently it won't build. This CL adds a gclient config to this builder.

Bug:783374
Change-Id: I89dcc6638b547d95fb9c1d8896ef0149c35b6fb2
Reviewed-on: https://chromium-review.googlesource.com/868840
Reviewed-by: Benjamin Pastene <bpastene@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/2d8e6bb694fafb432831a5f604a6e125ec6d5170/scripts/slave/recipe_modules/chromium_tests/chromium_fyi.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/58fb6e298af57bdf501df1904e31316797b13ca3

commit 58fb6e298af57bdf501df1904e31316797b13ca3
Author: Xing Liu <xingliu@chromium.org>
Date: Fri Jan 19 02:07:39 2018

Mojo android bot: Adds dimension set.

Currently Mojo Android bot can successfully build but will yields
NoDevicesError.

The cause might be the builder targets arm64 where default device list
is arm32. This CL adds the correct device dimension.


Bug: 783374
Change-Id: Id5edc08a62d6c1b8a08dde179ae849f4c6511fd7
Reviewed-on: https://chromium-review.googlesource.com/875250
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Xing Liu <xingliu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#530391}
[modify] https://crrev.com/58fb6e298af57bdf501df1904e31316797b13ca3/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/58fb6e298af57bdf501df1904e31316797b13ca3/testing/buildbot/waterfalls.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/880ae61e68a8806986cd8678641c8d6e0a0c1d3c

commit 880ae61e68a8806986cd8678641c8d6e0a0c1d3c
Author: Xing Liu <xingliu@chromium.org>
Date: Wed Jan 31 02:42:38 2018

Mojo Android bot: Increase hard timeout for swarm configuration.

Currently Mojo Android bot can compile but will timeout during
StartInstrumentation. This CL increases hard timeout for the swarm
to wait longer.

Failed bot log is here:
https://logs.chromium.org/v/?s=chromium%2Fbb%2Ftryserver.chromium.android%2Fandroid_mojo%2F19%2F%2B%2Frecipes%2Fsteps%2Fnetwork_service_content_browsertests__with_patch__on_Android%2F0%2Fstdout


Bug: 783374
Change-Id: I153fa452e579aa5d4b3247e779756c89b929a692
Reviewed-on: https://chromium-review.googlesource.com/893902
Reviewed-by: Benjamin Pastene <bpastene@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Xing Liu <xingliu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#533174}
[modify] https://crrev.com/880ae61e68a8806986cd8678641c8d6e0a0c1d3c/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/880ae61e68a8806986cd8678641c8d6e0a0c1d3c/testing/buildbot/waterfalls.pyl

Mojo Android bot will still time out even if we use 2 hours hard_timeout. Not sure what to do next here.


The following factors may contribute to this:

1. Lots of tests in content_browsertests on Android with network service flags enabled will hang. 

2. Currently there is only one device running content_browsertests in one shard for this bot.

BrowserTestBase::InitializeNetworkProcess seems hang on Android in some circumstances. 
In this function, 
network::mojom::NetworkServiceTestPtr::AddRules seems hang forever.

content::BrowserTestBase runs BrowserMain instead of ContentMain on Android, where service_manager::Main is called from ContentMain, I think that might cause the hang where service manager totally didn't spin up for browser tests.

In NativeBrowserTestActivity.java, we always add the "--single_process" command line, does android browser tests actually support multiple processes?

If so, there might be more work than expected to make android support multi processes in tests.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b1583b0f2dcaa19d357d4823d9ed237b12b49fb9

commit b1583b0f2dcaa19d357d4823d9ed237b12b49fb9
Author: Xing Liu <xingliu@chromium.org>
Date: Tue Feb 13 23:11:36 2018

Mojo Android bot: Only run in-process network service.

This CL tweaks the configuration and make the try bot only run in-process
network service for now, so it can yield useful result and not always
times out.

The android_mojo bot uses the same filter file as linux mojo bot.

Still working on the out of process case.


Bug: 783374
Change-Id: I0584a1e2c913929c1095fb2c45d81a6a83dd5dc8
Reviewed-on: https://chromium-review.googlesource.com/915222
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Xing Liu <xingliu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#536537}
[modify] https://crrev.com/b1583b0f2dcaa19d357d4823d9ed237b12b49fb9/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/b1583b0f2dcaa19d357d4823d9ed237b12b49fb9/testing/buildbot/test_suites.pyl
[modify] https://crrev.com/b1583b0f2dcaa19d357d4823d9ed237b12b49fb9/testing/buildbot/waterfalls.pyl

Found an issue that sync mojo call NetworkServiceTest::AddRules will block the dispatch of a Android aidl call to get the command line in utility process, so utility process's main thread in Java won't get command line and never enter c++'s UtilityMain where eventually create NetworkServiceTestHelper. Kind of dead lock between browser process and utility process.

Still trying to figure out why sync message or something else block this idl call.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d81ace664dd98b801bc8ad0c6b268db2074941ec

commit d81ace664dd98b801bc8ad0c6b268db2074941ec
Author: Xing Liu <xingliu@chromium.org>
Date: Mon Feb 26 21:02:38 2018

Network service test: push DNS rules after child process is connected.

Currently we use a synchronize mojo call to push the DNS rules to
network service process in browser test.

However, this causes tests to hang on Android, that browser test in
browser process waits on utility process to create
NetworkServiceTestHelper, but the utility process also waits on browser
process to send the command lines in Java to utility through Android's
aidl call, which is blocked by sync mojo call on launcher process.

Utility process Java main thread is wait on command lines in here:
https://chromium.googlesource.com/chromium/src/+/66.0.3348.3/base/android/java/src/org/chromium/base/process_launcher/ChildProcessService.java#158

We send command line from launcher process to utility process here, this
call is dispatched through message loop and blocked by the sync call.
https://chromium.googlesource.com/chromium/src/+/66.0.3348.3/base/android/java/src/org/chromium/base/process_launcher/ChildProcessConnection.java#480

In this CL, we send the DNS rules after browser-child process
connection is established, to avoid the deadlock on Android.

Bug: 783374
Change-Id: I3f7db4eba80fd263b1d37bcaa0ddcb1d83414485
Reviewed-on: https://chromium-review.googlesource.com/923283
Commit-Queue: Xing Liu <xingliu@chromium.org>
Reviewed-by: John Abd-El-Malek <jam@chromium.org>
Cr-Commit-Position: refs/heads/master@{#539266}
[modify] https://crrev.com/d81ace664dd98b801bc8ad0c6b268db2074941ec/content/public/test/browser_test_base.cc
[modify] https://crrev.com/d81ace664dd98b801bc8ad0c6b268db2074941ec/mojo/public/cpp/bindings/sync_call_restrictions.h
[modify] https://crrev.com/d81ace664dd98b801bc8ad0c6b268db2074941ec/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/d81ace664dd98b801bc8ad0c6b268db2074941ec/testing/buildbot/test_suites.pyl

Pada tanggal 27 Feb 2018 4.03 AM, "bugdro… via monorail" <
monorail+v2.3275348242@chromium.org> menulis:

Service working on Android and setup FYI bot

https://chromium.googlesource.com/chromium/src.git/+/d81ace664dd98b801bc8ad0c6b268db2074941ec

https://chromium.googlesource.com/chromium/src/+/66.0.3348.3/base/android/java/src/org/chromium/base/process_launcher/ChildProcessService.java#158

https://chromium.googlesource.com/chromium/src/+/66.0.3348.3/base/android/java/src/org/chromium/base/process_launcher/ChildProcessConnection.java#480

https://crrev.com/d81ace664dd98b801bc8ad0c6b268db2074941ec/content/public/test/browser_test_base.cc

https://crrev.com/d81ace664dd98b801bc8ad0c6b268db2074941ec/mojo/public/cpp/bindings/sync_call_restrictions.h

https://crrev.com/d81ace664dd98b801bc8ad0c6b268db2074941ec/testing/buildbot/chromium.fyi.json

https://crrev.com/d81ace664dd98b801bc8ad0c6b268db2074941ec/testing/buildbot/test_suites.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a8a0539cc70f606275ac046b93e85ee7bf3af66d

commit a8a0539cc70f606275ac046b93e85ee7bf3af66d
Author: Xing Liu <xingliu@chromium.org>
Date: Mon Mar 05 19:40:54 2018

Enable NetworkServiceRestartBrowserTest on Android.

This CL includes NetworkServiceRestartBrowserTest to run on Android
platform.

TBR=jam@chromium.org,dtrainor@chromium.org

Bug: 783374
Change-Id: Ic4210f63a2586f7bcf20424cdd1dc759cea7b14d
Reviewed-on: https://chromium-review.googlesource.com/947471
Reviewed-by: Xing Liu <xingliu@chromium.org>
Commit-Queue: Xing Liu <xingliu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#540899}
[modify] https://crrev.com/a8a0539cc70f606275ac046b93e85ee7bf3af66d/content/test/BUILD.gn

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c005394adc58733a96ea413fc2f420b449c1bd0e

commit c005394adc58733a96ea413fc2f420b449c1bd0e
Author: Xing Liu <xingliu@chromium.org>
Date: Tue Mar 06 18:40:44 2018

Revert "Enable NetworkServiceRestartBrowserTest on Android."

This reverts commit a8a0539cc70f606275ac046b93e85ee7bf3af66d.

Reason for revert: failing on internal.client.clank/asan-clang-phone
Bug:819230

Original change's description:
> Enable NetworkServiceRestartBrowserTest on Android.
> 
> This CL includes NetworkServiceRestartBrowserTest to run on Android
> platform.
> 
> TBR=jam@chromium.org,dtrainor@chromium.org
> 
> Bug: 783374
> Change-Id: Ic4210f63a2586f7bcf20424cdd1dc759cea7b14d
> Reviewed-on: https://chromium-review.googlesource.com/947471
> Reviewed-by: Xing Liu <xingliu@chromium.org>
> Commit-Queue: Xing Liu <xingliu@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#540899}

TBR=jam@chromium.org,dtrainor@chromium.org,xingliu@chromium.org

Change-Id: I64391e1bfe82b8978c7cd0d8afef653a5ba06fe2
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 783374
Reviewed-on: https://chromium-review.googlesource.com/951664
Reviewed-by: Xing Liu <xingliu@chromium.org>
Commit-Queue: Xing Liu <xingliu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#541159}
[modify] https://crrev.com/c005394adc58733a96ea413fc2f420b449c1bd0e/content/test/BUILD.gn