Consider making testing/buildbot/generate_buildbot_json.py apply exceptions for main test suite to derived suites too |
|
Issue descriptionWe have a bunch of tests that just run another test with a certain flag. For example, network_service_browser_tests just runs browser_tests with --enable-features=NetworkService, viz_browser_tests runs browser_tests with --enable-features=VizDisplayCompositor, etc. browser_tests has a bunch of entries in test_suite_exceptions.pyl. viz_browser_tests and network_service_browser_tests need to duplicate them, which confuses folks (e..g https://chromium-review.googlesource.com/1096490 https://chromium-review.googlesource.com/1097498 https://chromium-review.googlesource.com/c/chromium/src/+/1098630 etc). We should probably make it so that generate_buildbot_json.py uses the exceptions of the base test for the derivative. In rare cases where this is not desired, the derived test can then have its own exception to override the base exception.
,
Jun 13 2018
In practice, just from looking at the history of the _exceptions file, it seems like you'd always want the same exceptions.
,
Jun 13 2018
That could be true, but I feel like we don't really have enough data points to be sure or to give me any guidance in what the design might look like. You've been spending more time than I looking at this, though, so if you think you've got a sense of what might make sense, feel free to propose something / code it up / whatever to give us something concrete to talk about.
,
Jun 25 2018
How about adding a new flag to the exceptions which indicate whether they should apply not only to the test suite of that name, but also to those where that's the 'test'? The default could be either true or false, but I think the option should be there to toggle it.
,
Jun 26 2018
For the particular examples of browser_tests, viz_browser_tests, network_service_browser_tests, and site_per_process_browser_tests, we do *not* want to run all four test suites everywhere exactly the same way. The three most obvious reasons for this are that (a) the command lines might vary, and the exceptions are what we use to capture that variance, (b) often these suites are either still being brought up, or only make sense on some platforms, and (c) we may not have the capacity to actually run all four of them on particular configs, and we get enough coverage in the CQ by only running them on some of the configs. So, I am still kinda skeptical. It could be that a flag is the right thing to do, but like I said, I'd want to see concrete proposals to evaluate. |
|
►
Sign in to add a comment |
|
Comment 1 by dpranke@chromium.org
, Jun 13 2018