New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 852416 link

Starred by 1 user

Issue metadata

Status: Untriaged
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

Consider making testing/buildbot/generate_buildbot_json.py apply exceptions for main test suite to derived suites too

Project Member Reported by thakis@chromium.org, Jun 13 2018

Issue description

We have a bunch of tests that just run another test with a certain flag. For example, network_service_browser_tests just runs browser_tests with --enable-features=NetworkService, viz_browser_tests runs browser_tests with --enable-features=VizDisplayCompositor, etc.

browser_tests has a bunch of entries in test_suite_exceptions.pyl. viz_browser_tests and network_service_browser_tests need to duplicate them, which confuses folks (e..g https://chromium-review.googlesource.com/1096490 https://chromium-review.googlesource.com/1097498 https://chromium-review.googlesource.com/c/chromium/src/+/1098630 etc).


We should probably make it so that generate_buildbot_json.py uses the exceptions of the base test for the derivative. In rare cases where this is not desired, the derived test can then have its own exception to override the base exception.
 
It's not clear to me that you when you'd want the same exceptions to apply and when you wouldn't, or how you'd indicate that, so I'm not sure what the right way to do this would be. 

Comment 2 by thakis@chromium.org, Jun 13 2018

In practice, just from looking at the history of the _exceptions file, it seems like you'd always want the same exceptions.
That could be true, but I feel like we don't really have enough data points to be sure or to give me any guidance in what the design might look like. 

You've been spending more time than I looking at this, though, so if you think you've got a sense of what might make sense, feel free to propose something / code it up / whatever to give us something concrete to talk about.

Comment 4 by kbr@chromium.org, Jun 25 2018

How about adding a new flag to the exceptions which indicate whether they should apply not only to the test suite of that name, but also to those where that's the 'test'? The default could be either true or false, but I think the option should be there to toggle it.

For the particular examples of browser_tests, viz_browser_tests, network_service_browser_tests, and site_per_process_browser_tests, we do *not* want to run all four test suites everywhere exactly the same way. 

The three most obvious reasons for this are that (a) the command lines might vary, and the exceptions are what we use to capture that variance, (b) often these suites are either still being brought up, or only make sense on some platforms, and (c) we may not have the capacity to actually run all four of them on particular configs, and we get enough coverage in the CQ by only running them on some of the configs.

So, I am still kinda skeptical. It could be that a flag is the right thing to do, but like I said, I'd want to see concrete proposals to evaluate.

Sign in to add a comment