New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 633739 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Last visit > 30 days ago
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

Add support to run Android CTS test on multiple android platforms

Project Member Reported by mikec...@chromium.org, Aug 2 2016

Issue description

With our current setup, we only support running the CTS tests built for Lollipop on our infrastructure. These CTS tests do not have CTS tests for newer android apis and also, the older version of trade-fed packaged with these tests does not work on N phones.

We will have to update this file with CTS zips for each platform...
https://cs.chromium.org/chromium/src/android_webview/tools/cts_config/webview_cts_gcs_path.txt

We will also have to update the recipes to download the correct CTS zip based on the devices it is using.

Finally, we will have to bug infra to put java 1.8 on some machines since the newer versions of trade-fed (needed to run the N CTS tests) require it.
 
Project Member

Comment 1 by bugdroid1@chromium.org, Aug 5 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6fc6c1e68eb8bda9e79bfbc472499074e64c000f

commit 6fc6c1e68eb8bda9e79bfbc472499074e64c000f
Author: mikecase <mikecase@chromium.org>
Date: Fri Aug 05 18:22:58 2016

Adding platform specific CTS tests.

We should be running CTS tests on the platform they were built for.
Currently, we just run the L CTS tests everywhere. Changing the
webview_cts_gcs_path file to json, and will add CTS zips for each
platform we are interested in running tests on.

BUG=633739

Review-Url: https://codereview.chromium.org/2202853003
Cr-Commit-Position: refs/heads/master@{#410115}

[modify] https://crrev.com/6fc6c1e68eb8bda9e79bfbc472499074e64c000f/android_webview/tools/cts_config/expected_failure_on_bot.json
[add] https://crrev.com/6fc6c1e68eb8bda9e79bfbc472499074e64c000f/android_webview/tools/cts_config/webview_cts_gcs_path.json
[delete] https://crrev.com/55da79583bf83778309007c14f53092295cc0a7d/android_webview/tools/cts_config/webview_cts_gcs_path.txt

Project Member

Comment 2 by bugdroid1@chromium.org, Aug 5 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/9698e1fca3cdca11d25bdef82986b13ab37c24ca

commit 9698e1fca3cdca11d25bdef82986b13ab37c24ca
Author: mikecase <mikecase@chromium.org>
Date: Fri Aug 05 18:34:20 2016

[WebView] Update CTS logic to load test name from json.

We will want to be able to specify multiple CTS test zips for
different android platforms. Adding logic to grab the CTS test
name from a formatted json file isntead of just a text file.

BUG=633739

Review-Url: https://codereview.chromium.org/2206113002

[modify] https://crrev.com/9698e1fca3cdca11d25bdef82986b13ab37c24ca/scripts/slave/recipe_modules/chromium_android/api.py
[modify] https://crrev.com/9698e1fca3cdca11d25bdef82986b13ab37c24ca/scripts/slave/recipe_modules/chromium_android/example.expected/webview_cts_basic.json
[modify] https://crrev.com/9698e1fca3cdca11d25bdef82986b13ab37c24ca/scripts/slave/recipe_modules/chromium_android/example.expected/webview_cts_expected_failure.json
[modify] https://crrev.com/9698e1fca3cdca11d25bdef82986b13ab37c24ca/scripts/slave/recipe_modules/chromium_android/example.expected/webview_cts_invalid_output.json
[add] https://crrev.com/9698e1fca3cdca11d25bdef82986b13ab37c24ca/scripts/slave/recipe_modules/chromium_android/example.expected/webview_cts_invalid_platform.json
[modify] https://crrev.com/9698e1fca3cdca11d25bdef82986b13ab37c24ca/scripts/slave/recipe_modules/chromium_android/example.expected/webview_cts_unexecuted_tests.json
[modify] https://crrev.com/9698e1fca3cdca11d25bdef82986b13ab37c24ca/scripts/slave/recipe_modules/chromium_android/example.expected/webview_cts_unexpected_class_failure.json
[modify] https://crrev.com/9698e1fca3cdca11d25bdef82986b13ab37c24ca/scripts/slave/recipe_modules/chromium_android/example.expected/webview_cts_unexpected_method_failure.json
[modify] https://crrev.com/9698e1fca3cdca11d25bdef82986b13ab37c24ca/scripts/slave/recipe_modules/chromium_android/example.py
[modify] https://crrev.com/9698e1fca3cdca11d25bdef82986b13ab37c24ca/scripts/slave/recipes/chromium.expected/full_chromium_android_Android_WebView_CTS_L_MR1__dbg_.json

Project Member

Comment 3 by bugdroid1@chromium.org, Aug 5 2016

Project Member

Comment 4 by bugdroid1@chromium.org, Aug 5 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/613384915b046fb43c05c7bbcfcf0d8e93dfa515

commit 613384915b046fb43c05c7bbcfcf0d8e93dfa515
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Aug 05 21:48:45 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/448c0c7029e4f4d6cd8381032800c65fc839bca0 [Android] Run result_details logic in a finally block. (jbudorick@chromium.org)
  https://crrev.com/47e92bc8f5032a745175810c0f7096e4b3d7d3b2 [Android] Pass known devices file path to host info to detect missing devices. (rnephew@chromium.org)
  https://crrev.com/e93a9b8a319982bbab04851673467a375ddd3730 [cronet] Update PERF_IDs for new builders. (jbudorick@chromium.org)
  https://crrev.com/ef300a91ffa556e0b7eeed363b926aeea9eb3a40 Revert of Removing old recipe code (patchset #1 id:1 of https://codereview.chromium.org/2219653003/ ) (martiniss@chromium.org)
  https://crrev.com/a6c0bfd3638afcdab3870fc06e38da7026f64a02 Add phajdan.jr@ to recipe roller gatekeeper (martiniss@chromium.org)
  https://crrev.com/9698e1fca3cdca11d25bdef82986b13ab37c24ca [WebView] Update CTS logic to load test name from json. (mikecase@chromium.org)

R=jbudorick@chromium.org,rnephew@chromium.org,iannucci@chromium.org,estaab@chromium.org,mikecase@chromium.org,martiniss@chromium.org
BUG= 634894 , 592140 ,633739

TBR=martiniss@chromium.org,phajdan.jr@chromium.org

Review-Url: https://codereview.chromium.org/2217313002
Cr-Commit-Position: refs/heads/master@{#410174}

[modify] https://crrev.com/613384915b046fb43c05c7bbcfcf0d8e93dfa515/infra/config/recipes.cfg

Sign in to add a comment