thanks for filing, so yes some discussion going on in https://codereview.chromium.org/2528433004/...

Let's continue the discussion here?

I had a hard time running the tests locally with the up-to-date idl's and even
> C++ code - ninja says there's no work to do and the test fails with
> disableRemotePlayback exposed on WebView despite the clear
> [RuntimeEnabled=RemotePlayback].
> 
> I'm very frustrated and hope it will pass on the bots.

torne: I think you're not building/installing the right things? The shell doesn't contain any blink code (or any native code at all) and so it does indeed not need to be recompiled. The shell is testing the actual WebView installed on the current device - you need to build the webview that you want to test against and install it in addition to the test shell.

timvolodine: yes, that's what the bots do as well: explicitly build and install the webview to test. I guess this can be confusing, not sure if we should make system_webview_shell_layout_test_apk depend on webview though..

torne: I think that'd just be even more confusing because you'd wonder why nothing changes even though it rebuilt.

First, I think it'd be great if there was some documentation for the test. Now that torne pointed out I'm not installing the right thing, I looked up for Android WebView and installing it on a device with AOSP build (which I don't have atm) - things make more sense.

However, when I googled system_webview_shell_layout_test_apk I didn't find any instructions so assumed it should work like content_shell_test_apk which does depend on content_shell_apk and installs it from the run_content_shell_test_apk AFAIK.

I think if the device build is AOSP with root, that should work for webview too, right? Could the script try that and if it fails print a warning message like "Couldn't install WebView from the build, running the test against the WebView installed in the system".

Also, since installing Chrome Canary allows me to use it as WebView, can I do the same with a Chromium build on a rooted N device or something?

Hit this while sheriffing (https://bugs.chromium.org/p/chromium/issues/detail?id=683153). This suite is difficult to run correctly.

Tim, can you take a look at writing some better docs for this and maybe getting the run script to do it automatically if possible?

sure will take a look once I get through the release blockers.. the script may need checking / updating

Actually spinning out the script issue / documentation into  crbug.com/686114 . This is separate from the build behavior described originally.