+cc GPU folks, any test you want to add other than gpu_unittests in https://cs.chromium.org/chromium/src/tools/code_coverage/test_suite.txt. This will then automatically show up on https://chromium-coverage.appspot.com/

Could you please explain the environment in which these tests run on the bots?

It's my understanding that they run in a VM, which is a different environment than running on a local Linux workstation (no real graphics stack, for example).

How can we tell that adding test suites to this list won't break things?

Assigning back for more details.

Environment is GCE vm (ubuntu 16.04) and yes it wont have actual gpus. But can the tests use software swiftshader ?

To ensure it does not break things, you verify locally first. see help on coverage.py
e.g.
python tools/code_coverage/coverage.py crypto_unittests -b out/coverage -o out/report -c 'out/coverage/crypto_unittests'

Would be happy to help with this, but think it's not as high priority. Lowering. Feel free to assign to me if you agree it's not pri-1.

Sounds good. Not high priority, but good to do it some time. feels more pri-2 to me.

OK, thanks.

Running on SwiftShader is a reasonable long term plan but won't help ANGLE until we have SwiftShader integration. We could conceivably run angle_end2end_tests on D3D11/WARP without too much effort.

jmadill@: the code coverage setup only works on Linux right now, so it wouldn't be possible to test the D3D backend.

mmoroz@: do you have a list of test targets you have in mind to run? Can per-test-suite command line arguments be added? That would be needed in order to specify --use-gl=swiftshader for some or many of the harnesses.

Well, I was hoping we could add angle_end2end_tests, but it looks like this might be more complicated than the Chromium GPU tests. Maybe it would make sense to file a separate issue for that.

kbr@, I don't have any targets in mind, this issue has started from jmadill@'s email to chrome-code-coverage@. I've just moved the discussion to the bug tracker :)

It's up to you to decide which tests you'd like to see on the aggregated code coverage dashboard (https://chromium-coverage.appspot.com/). I'm not an expert in that area.

As for the interface for running the tests, currently we have two options:

1) If GN target has the same name as the binary executable produced, and it doesn't require special args, just add it to https://cs.chromium.org/chromium/src/tools/code_coverage/test_suite.txt

2) If test target requires some special handling, follow the instructions in https://bugs.chromium.org/p/chromium/issues/detail?id=843185#c7


If you only need to add some arguments to specific targets, option 2) would be an overkill. We can think of something more universal, e.g. add an option to specify arguments in the test suite file. Let us know if that would work for you.

Thanks mmoroz@ for the detailed info.

Jamie: I guess without SwiftShader integrated into ANGLE it'll be difficult to get angle_end2end_tests running.

Looking through the test suites in https://chromium.googlesource.com/chromium/src/+/master/content/test/gpu/generate_buildbot_json.py , the main test suites that could run with code coverage enabled (which aren't already in https://cs.chromium.org/chromium/src/tools/code_coverage/test_suite.txt ) would be the Telemetry-based GPU tests, run in the browser with --use-gl=swiftshader .

Jamie, is this something that you would be interested in hooking up, or should we find someone else to do it? The best way would be to add a script which runs them per https://bugs.chromium.org/p/chromium/issues/detail?id=843185#c7 .

mmoroz@ note that some of these test suites take a very long time to run (hours) so probably we would not incorporate those.

Ken: I was thinking more for ANGLE's testing specifically. It sounds like this might be a ways off due to multiple blockers so assigning this back to you.

Nice! Though are you planning to enable code coverage on all the configs or just a few special configs? 

I worry that enable the code coverage may make the browser integration test a bit less realistic.

We'll need a volunteer from the GPU team to help pick this up; I can't take this on personally.