The easiest thing to do at the moment is to add these tests to configs that aren't part of the CQ, e.g. "Mac10.11 Tests" and "Win10 x64 Tests". We should have a test config for Android that we can put these on as well ...

So, hopefully we can do this for now w/o any code changes.

I think the idea of supporting "waterfall-only" tests in the //testing/buildbot/*.json is a good one, but (a) we should discourage people from using it and (b) we should probably fold this idea into a larger plan because if we just do the simplest possible thing it'll probably confuse people who won't understand why the test wasn't run on the CQ.

I've filed bug 699295 to track the idea but it sounds like we don't need it implemented for this.

Ok sure. Which Android tester would be a good choice?

#2: either https://luci-milo.appspot.com/buildbot/chromium.android/Marshmallow%2064%20bit%20Tester/ (n5x on m, similar configuration to android_n5x_swarming_rel % debug) or https://luci-milo.appspot.com/buildbot/chromium.linux/Android%20Tests%20%28dbg%29/ (n5 on k, similar configuration to linux_android_rel_ng % debug)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b86c2dc8b581ec650806e80b7c112682078ffb0b

commit b86c2dc8b581ec650806e80b7c112682078ffb0b
Author: jam <jam@chromium.org>
Date: Wed Mar 08 05:54:55 2017

Run big test suites with PlzNavigate on Mac/Win main-waterfall-only bots.

These are the test suites:
browser_tests
components_browsertests
components_unittests
content_browsertests
content_unittests
interactive_ui_tests
extensions_browsertests
unit_tests

Win10 x64 and Mac 10.11 are picked since we don't have capacity to run these tests on CQ yet.

BUG= 699242 

Review-Url: https://codereview.chromium.org/2736943004
Cr-Commit-Position: refs/heads/master@{#455387}

[modify] https://crrev.com/b86c2dc8b581ec650806e80b7c112682078ffb0b/testing/buildbot/chromium.mac.json
[modify] https://crrev.com/b86c2dc8b581ec650806e80b7c112682078ffb0b/testing/buildbot/chromium.win.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1187604e8e7c592bcdd7d82c1b741ed33f6f78ff

commit 1187604e8e7c592bcdd7d82c1b741ed33f6f78ff
Author: jam <jam@chromium.org>
Date: Wed Mar 08 08:33:12 2017

Disable interactive_ui_tests on Win 10 x64 bot again.

BUG= 699442 ,  699242 
TBR=dpranke

Review-Url: https://codereview.chromium.org/2733383002
Cr-Commit-Position: refs/heads/master@{#455401}

[modify] https://crrev.com/1187604e8e7c592bcdd7d82c1b741ed33f6f78ff/testing/buildbot/chromium.win.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5bcb8c23c0ac564edeeeef974050f8b58372b3af

commit 5bcb8c23c0ac564edeeeef974050f8b58372b3af
Author: jam <jam@chromium.org>
Date: Wed Mar 08 09:11:25 2017

Don't run extensions_browsertests on Mac 10.11 as there are a few failures.

BUG=699432,  699242 
TBR=dpranke

Review-Url: https://codereview.chromium.org/2738743004
Cr-Commit-Position: refs/heads/master@{#455411}

[modify] https://crrev.com/5bcb8c23c0ac564edeeeef974050f8b58372b3af/testing/buildbot/chromium.mac.json

er, following up: use https://luci-milo.appspot.com/buildbot/chromium.android/Marshmallow%2064%20bit%20Tester/.

Thanks.

I couldn't find a way to pass flags to instrumentation tests; am I missing something? In the meantime I can at least run it on content_browsertests and other C++ tests.

also it seems the format for Android tests is different? i.e. I don't see a "name" value. I do see "override_isolate_target", but I'm not familiar with that one?

There's not a great way to do so at the moment -- we support doing so with a file passed via --device-flags-file currently but don't support something like --device-flags="--enable-browser-side-navigation" -- but that should be pretty easily fixable. I'll get back to you.

re "name": yeah, I don't think we ever adapted to https://chromium.googlesource.com/chromium/tools/build/+/365dbfe3cbe2c1aef44c52534d5feab5ce6426a9

re "override_isolate_target": we just haven't gotten around to cleaning it up yet :/ (https://chromium.googlesource.com/chromium/tools/build/+/0b8c9fb349a7e818219110a3045dbb9bd630f294/scripts/slave/recipe_modules/chromium_tests/steps.py#1079) but it should be the same as the test name.

following up from offline questions:

 - as suspected, M64's matching trybot is https://luci-milo.appspot.com/buildbot/tryserver.chromium.android/android_n5x_swarming_dbg/

 - arguments should be able to be passed to android gtest suites by adding

    "args": [
      "--test-arguments='--enable-browser-side-navigation'"
    ],

   to the spec. I don't think there are any current examples of this, though.

Also the marshmallow bot is red for a day; does anyone keep it green? If not, then adding PlzNavigate tests to it might not be useful.

It should be monitored by the android sheriff, though it's been hidden on SoM from back when we had blink_platform_unittests running on it.

Ok, is there a bug or task assigned to someone to show it on SoM? I'm not familiar with how bots are hidden there.

just landed https://chromium-review.googlesource.com/c/452685/ to do so.

Great, thanks

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9d7a165ce1c347e2b9a2ff253421d30c5d03ba5c

commit 9d7a165ce1c347e2b9a2ff253421d30c5d03ba5c
Author: jam <jam@chromium.org>
Date: Fri Mar 10 18:01:00 2017

Run gtest based test suites that are affected by PlzNavigate on the Android waterfall.

These are:
components_browsertests
components_unittests
content_browsertests
content_unittests
unit_tests

Marshmallow 64 bit bot is used since it's not on the CQ, as we don't have capacity yet to enable this on the main waterfall. Also OS specific breakages should be rare.

In a followup we'll add some instrumentation test suites when support for passing command lines is available.

BUG= 699242 

Review-Url: https://codereview.chromium.org/2745683002
Cr-Commit-Position: refs/heads/master@{#456103}

[modify] https://crrev.com/9d7a165ce1c347e2b9a2ff253421d30c5d03ba5c/testing/buildbot/chromium.android.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3b377cf1db6b26aff0789df18a59f532f4f82f6b

commit 3b377cf1db6b26aff0789df18a59f532f4f82f6b
Author: jam <jam@chromium.org>
Date: Sat Mar 18 04:36:19 2017

Run instrumentation tests with PlzNavigate on the Android waterfall.

chrome_public_test_apk has 8 tests still failing, so it'll be added later.

Marshmallow 64 bit bot is used since it's not on the CQ, as we don't have capacity yet to enable this on the main waterfall. Also OS specific breakages should be rare.

BUG= 699242 

Review-Url: https://codereview.chromium.org/2750003005
Cr-Commit-Position: refs/heads/master@{#457957}

[modify] https://crrev.com/3b377cf1db6b26aff0789df18a59f532f4f82f6b/testing/buildbot/chromium.android.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/105dc7df0557f8dc718a6f2914e66a1af13bfcba

commit 105dc7df0557f8dc718a6f2914e66a1af13bfcba
Author: jam <jam@chromium.org>
Date: Fri Mar 24 22:19:33 2017

Run chrome_public_test_apk with PlzNavigate.

BUG= 699242 

Review-Url: https://codereview.chromium.org/2771903003
Cr-Commit-Position: refs/heads/master@{#459574}

[modify] https://crrev.com/105dc7df0557f8dc718a6f2914e66a1af13bfcba/testing/buildbot/chromium.android.json