I think we need to swarm CTS tests before we look at this.

Swarming sounds nice, but IMO it's p0 that right now the only way for most devs to know if their relands because of broken CTS tests work is by committing again.

jbudorick@, are you ok if I setup an unswarmed tester on the tryserver to run CTS tests per comment 2 for now while the swarming task is pending?

Begrudgingly yes, I suppose. We really do not want to be in the business of setting up additional unswarmed bots, though.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/7354473a918f9b10528a3bed65cf45c35ed7247f

commit 7354473a918f9b10528a3bed65cf45c35ed7247f
Author: Andrew Luo <aluo@google.com>
Date: Thu Jun 07 21:55:23 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/538ed88dd77c8190c29fa95e17ae474a06e84042

commit 538ed88dd77c8190c29fa95e17ae474a06e84042
Author: Andrew Luo <aluo@chromium.org>
Date: Fri Jun 08 17:43:10 2018

Add trybot for webview cts

Bug:838650
Change-Id: I1745729443727a8250215977596ae321b5517d75
Reviewed-on: https://chromium-review.googlesource.com/1083939
Commit-Queue: Andrew Luo <aluo@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/538ed88dd77c8190c29fa95e17ae474a06e84042/tests/masters_recipes_test.py
[modify] https://crrev.com/538ed88dd77c8190c29fa95e17ae474a06e84042/scripts/slave/recipe_modules/chromium_tests/trybots.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c344f85d81d73445cb7a438425f8da3de3709803

commit c344f85d81d73445cb7a438425f8da3de3709803
Author: Andrew Luo <aluo@chromium.org>
Date: Fri Jun 08 18:07:10 2018

Add android_unswarmed_pixel_aosp

Bug:  838650 
Change-Id: Ibda5c05eafda372ecd56e35f3814efeedac86f56
Reviewed-on: https://chromium-review.googlesource.com/1083940
Commit-Queue: Andrew Luo <aluo@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#565683}
[modify] https://crrev.com/c344f85d81d73445cb7a438425f8da3de3709803/infra/config/global/cr-buildbucket.cfg
[modify] https://crrev.com/c344f85d81d73445cb7a438425f8da3de3709803/infra/config/global/luci-milo.cfg

Tasks are not getting started due to no bots satisfying the dimensions.  I don't see the bot listed here: https://luci-migration.appspot.com/masters/tryserver.chromium.android/

Will debug this next week.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d95ee34714282fdf1d0c2d471df18178dae884e4

commit d95ee34714282fdf1d0c2d471df18178dae884e4
Author: Andrew Luo <aluo@chromium.org>
Date: Tue Jun 12 01:58:29 2018

Revert "Add android_unswarmed_pixel_aosp"

This reverts commit c344f85d81d73445cb7a438425f8da3de3709803.

Reason for revert: I used the wrong bot type, will re-land this when the swarming bot is setup.

Original change's description:
> Add android_unswarmed_pixel_aosp
> 
> Bug:  838650 
> Change-Id: Ibda5c05eafda372ecd56e35f3814efeedac86f56
> Reviewed-on: https://chromium-review.googlesource.com/1083940
> Commit-Queue: Andrew Luo <aluo@chromium.org>
> Reviewed-by: John Budorick <jbudorick@chromium.org>
> Reviewed-by: Ben Pastene <bpastene@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#565683}

TBR=bpastene@chromium.org,jbudorick@chromium.org,aluo@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  838650 
Change-Id: I281c9d90759bcf3bffef3bb0d21c105eabdffb3a
Reviewed-on: https://chromium-review.googlesource.com/1096375
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#566265}
[modify] https://crrev.com/d95ee34714282fdf1d0c2d471df18178dae884e4/infra/config/global/cr-buildbucket.cfg
[modify] https://crrev.com/d95ee34714282fdf1d0c2d471df18178dae884e4/infra/config/global/luci-milo.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6fe7110aae2ed9f0ea79fef0ab5a40ef5596d43f

commit 6fe7110aae2ed9f0ea79fef0ab5a40ef5596d43f
Author: Andrew Luo <aluo@chromium.org>
Date: Tue Jun 12 21:26:47 2018

Reland Add android_unswarmed_pixel_aosp

Bug:  838650 
Change-Id: I40fd338c2c9586be3b780c469620dd6a741dafbc
Reviewed-on: https://chromium-review.googlesource.com/1097737
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Andrew Luo <aluo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#566576}
[modify] https://crrev.com/6fe7110aae2ed9f0ea79fef0ab5a40ef5596d43f/infra/config/global/cr-buildbucket.cfg
[modify] https://crrev.com/6fe7110aae2ed9f0ea79fef0ab5a40ef5596d43f/infra/config/global/luci-milo.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6c23b698ee4c48b713dcc855f4f938a1a0c4c368

commit 6c23b698ee4c48b713dcc855f4f938a1a0c4c368
Author: Andrew Luo <aluo@chromium.org>
Date: Tue Jun 19 02:51:41 2018

Add android_unswarmed_pixel_aosp to mb_config

Bug:  838650 
Change-Id: I9850ab16c61592bf46068c2948535ab9b95d843c
Reviewed-on: https://chromium-review.googlesource.com/1099566
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Andrew Luo <aluo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#568317}
[modify] https://crrev.com/6c23b698ee4c48b713dcc855f4f938a1a0c4c368/tools/mb/mb_config.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/3626fbd0a361633bc10879549bd2f6b56e4d2e06

commit 3626fbd0a361633bc10879549bd2f6b56e4d2e06
Author: Andrew Luo <aluo@chromium.org>
Date: Fri Jun 29 17:26:07 2018

Make WebViewCTSTest an AndroidTest

Bug:838650
Change-Id: I07130b2f63018f1288e4ac58ac76c46607a6c0f8

Recipe-Nontrivial-Roll: build_limited_scripts_slave
Change-Id: I07130b2f63018f1288e4ac58ac76c46607a6c0f8
Reviewed-on: https://chromium-review.googlesource.com/1112837
Commit-Queue: Andrew Luo <aluo@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>

[add] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_android/tests/run_webview_cts_test.expected/with details pass.json
[modify] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_tests/tests/steps/web_view_cts_test.py
[add] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_tests/tests/steps/web_view_cts_test.expected/with suffix.json
[add] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_tests/tests/steps/web_view_cts_test.expected/basic fail.json
[rename] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_android/tests/run_webview_cts_test.expected/with details fail.json
[add] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_android/tests/run_webview_cts_test.py
[add] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_android/tests/run_webview_cts_test.expected/basic pass.json
[modify] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_cts_test.expected/basic.json
[add] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_tests/tests/steps/web_view_cts_test.expected/basic pass.json
[modify] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/README.recipes.md
[add] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_android/tests/run_webview_cts_test.expected/with results file pass.json
[modify] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_tests/steps.py
[modify] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_cts_test.py
[add] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_android/tests/run_webview_cts_test.expected/with suffix.json
[add] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_android/tests/run_webview_cts_test.expected/basic fail.json
[add] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_android/tests/run_webview_cts_test.expected/with commandline args.json
[add] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_android/tests/run_webview_cts_test.expected/with results file fail.json
[modify] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_android/api.py
[modify] https://crrev.com/3626fbd0a361633bc10879549bd2f6b56e4d2e06/scripts/slave/recipe_modules/chromium_android/examples/full.expected/webview_cts_basic.json

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/f54baa4ce25fd999e1cf4b7e42637121c402969f

commit f54baa4ce25fd999e1cf4b7e42637121c402969f
Author: Andrew Luo <aluo@google.com>
Date: Fri Jun 29 21:20:10 2018

This is working now, please select android_unswarmed_pixel_aosp bot if you want to run CTS tests.  It is running the N version of the tests.  I'm working on getting this swarmed, then we can run on more SDK levels.

Nate reports on https://chromium-review.googlesource.com/1114348 that this bot always returns a purple result even though it did actually run the tests - is that expected?

It "was" expected.  But it has since been fixed with the commit in c15 so it shouldn't be purple after successful run now.

> But it has since been fixed with the commit in c15 so it shouldn't be purple after successful run now.

Nice! Thanks for adding this.