> This should minimally support things that conform to 
> bit.ly/chromium-test-runner-api, though supporting other binaries as well
> would be ideal.

I'm not sure that anything actually supports this today (I have to go back and re-read the doc). I think probably what I would propose is that we tweak that doc to find the right subset of flags that does (or almost does) work today, and then make all of the binaries support that API.

Supporting multiple APIs pretty much defeats the point of much of this.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6c448b73b6f2ee03435eb8d2d9e6d1125b22a2cb

commit 6c448b73b6f2ee03435eb8d2d9e6d1125b22a2cb
Author: Dirk Pranke <dpranke@chromium.org>
Date: Wed Jun 13 00:23:20 2018

Use wrapper scripts on desktop to run tests.

In order to simplify how we configure tests, it would be good if
every test target was invoked the same way on every platform.

Today, on Android, CrOS, and Fuchsia, we generate wrapper scripts
for every test target and put them in $root_build_dir/bin; these
scripts are responsible for setting up the environment (devices,
VMs, etc.) and invoking the test directly.

This CL continues this pattern and generates wrappers that will
work on Linux, Mac, and Win following the same pattern (on Windows,
the wrappers will be bin/run_base_unittests.bat instead of
bin/run_base_unittests).

As a proof of concept, this CL switches url_unittests to run via
the wrapper. Assuming this change lands and sticks, in follow-up
changes we'll convert the others as well.

This design should also help us eventually get rid of
gn_isolate_map.pyl; in order to do that, we will also need to
not translate between ninja target names and GN labels, but
that will require changes to GN.

Bug: 816629
Change-Id: I66f8ca78472564ee46cc8d3d012591a275566d1f
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/736162
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#566644}
[modify] https://crrev.com/6c448b73b6f2ee03435eb8d2d9e6d1125b22a2cb/testing/buildbot/gn_isolate_map.pyl
[add] https://crrev.com/6c448b73b6f2ee03435eb8d2d9e6d1125b22a2cb/testing/gen_desktop_runner_script.py
[modify] https://crrev.com/6c448b73b6f2ee03435eb8d2d9e6d1125b22a2cb/testing/test.gni
[modify] https://crrev.com/6c448b73b6f2ee03435eb8d2d9e6d1125b22a2cb/tools/mb/mb.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6c448b73b6f2ee03435eb8d2d9e6d1125b22a2cb

commit 6c448b73b6f2ee03435eb8d2d9e6d1125b22a2cb
Author: Dirk Pranke <dpranke@chromium.org>
Date: Wed Jun 13 00:23:20 2018

Use wrapper scripts on desktop to run tests.

In order to simplify how we configure tests, it would be good if
every test target was invoked the same way on every platform.

Today, on Android, CrOS, and Fuchsia, we generate wrapper scripts
for every test target and put them in $root_build_dir/bin; these
scripts are responsible for setting up the environment (devices,
VMs, etc.) and invoking the test directly.

This CL continues this pattern and generates wrappers that will
work on Linux, Mac, and Win following the same pattern (on Windows,
the wrappers will be bin/run_base_unittests.bat instead of
bin/run_base_unittests).

As a proof of concept, this CL switches url_unittests to run via
the wrapper. Assuming this change lands and sticks, in follow-up
changes we'll convert the others as well.

This design should also help us eventually get rid of
gn_isolate_map.pyl; in order to do that, we will also need to
not translate between ninja target names and GN labels, but
that will require changes to GN.

Bug: 816629
Change-Id: I66f8ca78472564ee46cc8d3d012591a275566d1f
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/736162
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#566644}
[modify] https://crrev.com/6c448b73b6f2ee03435eb8d2d9e6d1125b22a2cb/testing/buildbot/gn_isolate_map.pyl
[add] https://crrev.com/6c448b73b6f2ee03435eb8d2d9e6d1125b22a2cb/testing/gen_desktop_runner_script.py
[modify] https://crrev.com/6c448b73b6f2ee03435eb8d2d9e6d1125b22a2cb/testing/test.gni
[modify] https://crrev.com/6c448b73b6f2ee03435eb8d2d9e6d1125b22a2cb/tools/mb/mb.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2f5f734017ff28abdd5d2389de3cfeef42b6ffa9

commit 2f5f734017ff28abdd5d2389de3cfeef42b6ffa9
Author: Abhishek Arya <inferno@chromium.org>
Date: Wed Jun 13 16:59:44 2018

Revert "Use wrapper scripts on desktop to run tests."

This reverts commit 6c448b73b6f2ee03435eb8d2d9e6d1125b22a2cb.

Reason for revert: Breaks ClusterFuzz libFuzzer bots

We shouldn't add bin/ directory with run_<fuzzer_name>_fuzzer.

Original change's description:
> Use wrapper scripts on desktop to run tests.
> 
> In order to simplify how we configure tests, it would be good if
> every test target was invoked the same way on every platform.
> 
> Today, on Android, CrOS, and Fuchsia, we generate wrapper scripts
> for every test target and put them in $root_build_dir/bin; these
> scripts are responsible for setting up the environment (devices,
> VMs, etc.) and invoking the test directly.
> 
> This CL continues this pattern and generates wrappers that will
> work on Linux, Mac, and Win following the same pattern (on Windows,
> the wrappers will be bin/run_base_unittests.bat instead of
> bin/run_base_unittests).
> 
> As a proof of concept, this CL switches url_unittests to run via
> the wrapper. Assuming this change lands and sticks, in follow-up
> changes we'll convert the others as well.
> 
> This design should also help us eventually get rid of
> gn_isolate_map.pyl; in order to do that, we will also need to
> not translate between ninja target names and GN labels, but
> that will require changes to GN.
> 
> Bug: 816629
> Change-Id: I66f8ca78472564ee46cc8d3d012591a275566d1f
> No-Try: true
> Reviewed-on: https://chromium-review.googlesource.com/736162
> Commit-Queue: Dirk Pranke <dpranke@chromium.org>
> Reviewed-by: Ben Pastene <bpastene@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#566644}

TBR=dpranke@chromium.org,bpastene@chromium.org,jbudorick@chromium.org,shenghuazhang@chromium.org

Change-Id: I48d2be908c2f2ec529195e27b44c1032bd2d62a7
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 816629
Reviewed-on: https://chromium-review.googlesource.com/1099295
Reviewed-by: Abhishek Arya <inferno@chromium.org>
Commit-Queue: Abhishek Arya <inferno@chromium.org>
Cr-Commit-Position: refs/heads/master@{#566879}
[modify] https://crrev.com/2f5f734017ff28abdd5d2389de3cfeef42b6ffa9/testing/buildbot/gn_isolate_map.pyl
[delete] https://crrev.com/11a4d06d37bd1e94e6b69a21cd40271a7a5d36ac/testing/gen_desktop_runner_script.py
[modify] https://crrev.com/2f5f734017ff28abdd5d2389de3cfeef42b6ffa9/testing/test.gni
[modify] https://crrev.com/2f5f734017ff28abdd5d2389de3cfeef42b6ffa9/tools/mb/mb.py

Note that when this is re-checked, it is not enabled on any of clusterfuzz builders. these include

all lkgr bots
https://ci.chromium.org/p/chromium/g/chromium.lkgr/console

and all libfuzzer, afl bots
on https://ci.chromium.org/p/chromium/g/chromium.fyi/console

This confuses the logic in clusterfuzz when looking for fuzzer binaries.

Can we change the logic in clusterfuzz? Is it doing something like globbing on *fuzzer to find binaries? Maybe it should be ignoring things in the bin/ directory?

This has been discussed offline with Dirk. fuzzer_test will be excluded from this.

I may work on this for Telemetry tests & layout tests.

+Juan: this is a cool new direction that will allow removing all the apk/browser inferring logic in Telemetry. The idea is the runnable benchmark script will be generated during BUILD time & have all the needed runtime constant filled.