New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 629357 link

Starred by 2 users
Status: Verified
Owner: ----
Closed: Aug 2016
Cc:
█ scunning...@chromium.org
dchan@chromium.org
█ krishna...@chromium.org
kathrelk...@chromium.org
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Feature

Blocking:
issue 637432



Sign in to add a comment

Enterprise Autotest Refactor: Split testcases into multiple control files

Project Member Reported by kathrelk...@chromium.org, Jul 19 2016 
Instead of using mode='all' when running in the autotest lab, split each test into multiple control files
(e.g. control.notset, control.true, control.false).

For now, leave the /control file for use by the CPanel team, but remove the suite variable from it (so it doesn't run in our lab).

Whenever possible, pass in test values through the control file and make the tests value-agnostic.

Comment 1 by kathrelk...@chromium.org, Jul 19 2016

Cc: scunning...@chromium.org
Owner: ----
Status: Available (was: Assigned)

Comment 2 by scunning...@chromium.org, Jul 25 2016

Labels: -Type-Bug -Pri-3 Pri-2 Type-Feature
Owner: scunning...@chromium.org
Status: Assigned (was: Available)

Comment 3 by scunning...@chromium.org, Jul 29 2016

Status: Started (was: Assigned)
Project Member

Comment 4 by bugdroid1@chromium.org, Aug 12 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/8b20510d28820d2b788a4e7446d9ba282e4e9d13

commit 8b20510d28820d2b788a4e7446d9ba282e4e9d13
Author: Scott Cunningham <scunningham@chromium.org>
Date: Fri Jul 29 01:31:57 2016

Split test cases into separate control.case files.

Instead of running all the test cases defined in a test class for a
policy, create separate control files for each test case. This will
provide more precise test result reporting.

For now, the default control files will continue work as is, so the
CPanel E2E test framework can continue to pass args into the test.
However, remove the ATTRIBUTES property from them, so the default
test is not scheduled to run in the hardware lab. Also, move
args_to_dict from base class into the default control files.

BUG= chromium:629357 
TEST=Run test_that <IP> policy_<test>.<case> for each test case of
each policy test. Also, policy_<test> with --args for each test case,
to ensure that passing args thru the default control file still works
with existing framework.

Change-Id: I432d3c112a0de1a7f8fdf4cb4819128506406fc8
Reviewed-on: https://chromium-review.googlesource.com/364340
Commit-Ready: Scott Cunningham <scunningham@chromium.org>
Tested-by: Scott Cunningham <scunningham@chromium.org>
Reviewed-by: Scott Cunningham <scunningham@chromium.org>
Reviewed-by: Katherine Threlkeld <kathrelkeld@chromium.org>

[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ImagesAllowedForUrls/control.notset_block
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_JavaScriptBlockedForUrls/control.multipleurls_block
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_RestoreOnStartupURLs/control
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ForceGoogleSafeSearch/control.true_safe
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_JavaScriptAllowedForUrls/control.multipleurls_block
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_RestoreOnStartupURLs/control.notset_notabs
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ManagedBookmarks/control.multiplebookmarks_shown
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ProxySettings/control
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ManagedBookmarks/policy_ManagedBookmarks.py
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_CookiesBlockedForUrls/control.multipleurls_block
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_RestoreOnStartupURLs/control.singleurl_1tab
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_JavaScriptBlockedForUrls/control.multipleurls_allow
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/cros/enterprise_policy_base.py
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_EditBookmarksEnabled/control
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_URLBlacklist/control.multiplepages_blocked
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_URLWhitelist/control.multiplepages_allowed
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ImagesAllowedForUrls/control
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ProxySettings/policy_ProxySettings.py
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ImagesBlockedForUrls/control.1url_block
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_CookiesAllowedForUrls/control.notset_block
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ForceGoogleSafeSearch/control.false_notsafe
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_PopupsAllowedForUrls/control.3urls_allow
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_JavaScriptAllowedForUrls/control.notset_block
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ImagesAllowedForUrls/control.1url_allow
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_CookiesBlockedForUrls/control
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ForceGoogleSafeSearch/control.notset_notsafe
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_JavaScriptBlockedForUrls/control
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_URLWhitelist/control
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ProxySettings/control.directproxy_usenoproxy
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_CookiesBlockedForUrls/control.singleurl_block
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ImagesAllowedForUrls/control.2urls_block
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_JavaScriptBlockedForUrls/policy_JavaScriptBlockedForUrls.py
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ProxySettings/control.notset_usenoproxy
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_CookiesBlockedForUrls/control.multipleurls_allow
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ForceYouTubeSafetyMode/policy_ForceYouTubeSafetyMode.py
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_EditBookmarksEnabled/README
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ImagesBlockedForUrls/control.notset_allow
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_URLBlacklist/control
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ForceYouTubeSafetyMode/control.true_safetyactive
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ManagedBookmarks/control.notset_notshown
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_URLBlacklist/control.singlepage_blocked
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_PopupsBlockedForUrls/control.3urls_block
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ManagedBookmarks/control.singlebookmark_shown
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_CookiesAllowedForUrls/control.singleurl_allow
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ManagedBookmarks/control
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_PopupsBlockedForUrls/policy_PopupsBlockedForUrls.py
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_EditBookmarksEnabled/control.true_enable
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_JavaScriptAllowedForUrls/control.multipleurls_allow
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_EditBookmarksEnabled/control.false_disable
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ForceGoogleSafeSearch/control
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ImagesAllowedForUrls/control.3urls_allow
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_PopupsAllowedForUrls/control.1url_allow
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_CookiesAllowedForUrls/control.multipleurls_block
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ForceYouTubeSafetyMode/control
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_JavaScriptBlockedForUrls/control.singleurl_block
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ImagesAllowedForUrls/policy_ImagesAllowedForUrls.py
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_JavaScriptBlockedForUrls/control.notset_allow
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ImagesBlockedForUrls/control
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_PopupsAllowedForUrls/policy_PopupsAllowedForUrls.py
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_PopupsBlockedForUrls/control
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_PopupsAllowedForUrls/control.2urls_block
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_CookiesBlockedForUrls/control.notset_allow
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_CookiesAllowedForUrls/control.multipleurls_allow
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_CookiesBlockedForUrls/policy_CookiesBlockedForUrls.py
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_CookiesAllowedForUrls/policy_CookiesAllowedForUrls.py
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_URLBlacklist/policy_URLBlacklist.py
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_RestoreOnStartupURLs/control.multipleurls_3tabs
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_PopupsAllowedForUrls/control
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_EditBookmarksEnabled/control.notset_enable
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_RestoreOnStartupURLs/policy_RestoreOnStartupURLs.py
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ProxySettings/control.fixedproxy_usefixedproxy
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ForceYouTubeSafetyMode/control.notset_safetyinactive
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ImagesBlockedForUrls/control.3urls_block
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ForceYouTubeSafetyMode/control.false_safetyinactive
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_PopupsBlockedForUrls/control.notset_allow
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ImagesBlockedForUrls/policy_ImagesBlockedForUrls.py
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_PopupsBlockedForUrls/control.1url_block
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_PopupsBlockedForUrls/control.2urls_allow
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_JavaScriptAllowedForUrls/policy_JavaScriptAllowedForUrls.py
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_CookiesAllowedForUrls/control
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ForceGoogleSafeSearch/policy_ForceGoogleSafeSearch.py
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_JavaScriptAllowedForUrls/control.singleurl_allow
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_URLWhitelist/policy_URLWhitelist.py
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_URLWhitelist/control.singlepage_allowed
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ImagesBlockedForUrls/control.2urls_allow
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_URLBlacklist/control.notset_allowed
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_JavaScriptAllowedForUrls/control
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_PopupsAllowedForUrls/control.notset_block
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_ProxySettings/control.pacproxy_usepacfile
[modify] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_EditBookmarksEnabled/policy_EditBookmarksEnabled.py
[add] https://crrev.com/8b20510d28820d2b788a4e7446d9ba282e4e9d13/client/site_tests/policy_URLWhitelist/control.notset_blocked

Comment 5 by scunning...@chromium.org, Aug 12 2016

Status: Fixed (was: Started)

Comment 6 by scunning...@chromium.org, Aug 13 2016

Status: Verified (was: Fixed)
Tests running in lab, and being reported correctly in wmatrix.

Comment 7 by scunning...@chromium.org, Aug 16 2016

Blocking: 637432

Comment 8 by scunning...@chromium.org, Mar 3 2017

Owner: ----
►

Sign in to add a comment