To reword the question a bit: machenbach, you run tests on swarming on the v8 bot, right? How is that implemented?

I think the chromium recipe is a pretty decent model for how a generic "build then test" approach might want to run, so getting it to work for other projects wouldn't be a bad thing.

I don't think it would be a lot of work to refactor the chromium and chromium_tests modules to be useful to other projects, but people on the Infra>Client>Chrome team are not likely to have time for this until we complete the LUCI migration. If ANGLE/v8 folks wanted to help, I'd have no objection to this work.

There's a lot of cruft in the recipe modules, so any refactoring would probably be a bit hobbled by wanting to clean up some of the stuff we don't need any more as well.

I don't know if it would be particularly easy to extract just the swarming testing logic.

We're for sure be OK with waiting until after the LUCI migration. Something we might need going forward, but not a priority immediately. We can discuss our team contributing depending on the complexity level - breaking other project's waterfalls is a real risk that is probably better managed by those familiar with the code.

Most parts of V8's test stack are homegrown. Including v8-side json configs, reading those configs, triggering tests on swarming and collecting results.

I'd be happy to reuse more parts of Chrome if that becomes feasible.

It would be nice if the V8 team's work could be integrated more generically so that the other client recipes could pick it up and reuse it.

Agree with Ken. machenbach@, where does the code live for your test stack? Just curious.

Various places in V8 recipe module, e.g.:
https://chromium.googlesource.com/chromium/tools/build/+/4ac2c58b6/scripts/slave/recipe_modules/v8/resources/collect_v8_task.py
https://chromium.googlesource.com/chromium/tools/build/+/4ac2c58b6/scripts/slave/recipe_modules/v8/api.py#386

Maybe we could distill the essence of this and Chromium's way of running tests and somehow migrate it into a new, Chromium-independent recipe module.

Alternatively, are there really that many Chromium dependencies in recipe_modules/chromium_tests? (The answer may well be "yes".)

There aren't really that many chromium dependencies, I think it would be doable to make it independent.

The harder part, of course, would be adding the equivalent of the //testing/buildbot/* files for configuration and the wrapper scripts to implement the right API contracts, but even that is not that hard, and it would enable you to leverage more pieces of the system.

We could require that client projects use a copy of the testing/buildbot generator scripts for the JSON files. If we supported only gtests and isolated script tests then the contracts between the recipe and test targets would be pretty minimal.

Does the testing/buildbot machinery require projects to use src/base's test runner? (For some json output flag maybe?) If so, that'd need to be fixed somehow too.

It doesn't require the use of //base/testing specifically, but it does require you to implement a particular command line API contract and set of output, so that we can get the list of tests that fail to know what to retry.

I think the Chromium recipe or chromium_tests recipe module hardcodes knowledge of loading the JSON file from a testing/buildbot directory at the root of the solution. Of course that's not hard to change, but it could be documented as a necessary contract for client recipes.

#14: we've been working on making that configurable for use internally: https://chromium.googlesource.com/chromium/tools/build/+/2b836d474b772427eec11b6cb41bd60747b552f4/scripts/slave/recipe_modules/chromium_tests/config.py#13

#15: cool!

We're also interested in this for Dawn

Dirk, re: comment 13, do you know what particular command line APIs and what output is needed by the testing machinery? I'd like to follow up on this. In particular, is there anything special needed on Android?

I looked at WebRTC. They seem to be running fully sharded tests on their standalone infra, including on MSVC targets. It looks as though they have their own test harness. They also use "gtest-parallel" to split up their tests. Maybe can use some lessons from their solutions in designing the appropriate shareable bits for ANGLE and other clients.

The API you need to implement is described in the docs for //testing/scripts/run_isolated_script_test.py (basically the --isolated-script-test-output and --isolated-script-test-filter arguments). The JSON file you need to produce is documented at bit.ly/chromium-json-test-results-format.

There is nothing specific required for Android.

It's straightforward to run tests in parallel under swarming, but to get things like FindIt support and "retry without patch" requires a bit more work.

Dirk, what about the swarming related arguments? Namely:

--test-launcher-total-shards
--test-launcher-shard-index

as well there's some other flags used by the launcher, like

--test-launcher-retry-limit
--test-launcher-batch-limit
--test-launcher-filter-file
--test-launcher-jobs
--single-process-tests

Some of these might not be necessary but some of them are used right now by the recipes.

Also it look as though gtest-parallel handles a few of these arguments under different names. --dump_json_test_results instead of --isolated-script-test-output and --shard_count/--shard_index instead of --test-launcher-total-shards/--test-launcher-shard-index. Link:

https://github.com/google/gtest-parallel/blob/master/gtest_parallel.py#L683

Maybe non-Chromium clients interested in running gtest based tests can use a wrapper around gtest-parallel like WebRTC:

https://chromium.googlesource.com/external/webrtc/+/HEAD/tools_webrtc/gtest-parallel-wrapper.py