IMO, we can simply replace 'gtest' to 'test' so cover more generically:
1. test_filter=test1::test2::test3
2. test_repeat=30
3. --test-launcher-retry-limit=0 
4. --test_also_run_disabled_tests

for the 4th flag, I propose to just use --also_run_disabled_tests

Another few questions:
1) What happen when test-filter doesn't match any test? Should the test suite return code 0 or 1 or something else?
2) For --test_repeat=2, assuming the test to run are [A, B, C]. Would the test suite run [A, A, B, B, C, C] or [A, B, C, A, B, C]? (Telemetry benchmark prefers the latter)

Reply for Ned's questions:
1) For Findit, we expect the test suite to return 0 in this case if test specified by test-filter doesn't exist. Findit is actually using this case to check when a test is newly added.

2) For Findit, we also prefer tests to run in [A, B, C, A, B, C] order.

As mentioned in the meeting I think we should reuse --isolated-script-test-filter and rename the other flags to support that contract. It is a big problem when two test harnesses have collisions between these command line options and there is no chance that command line flags starting with "--isolated-script" will collide with any flags currently supported by gtest, typ, the layout tests, etc.

[apologies for bike-shedding style nit]

Whatever names we choose, can we consistently separate words with "-"? I find it jarring e.g. to have:

> --test-launcher-retry-limit=0 
> --also_run_disabled_tests

The later should be --also-run-disabled-tests

Reply for kbr@'s question: sorry I missed this discussion in my note. But in my opinion, the flags will be used by many test harnesses not just isolated script tests, so I'm not sure if we should have isolated-script-test (nor gtest like what we are using now) in the names.

Let's move to --isolated-script-WHATEVER for now, to be consistent, and change later.

Sorry for the late response.

IIUC, the decision is that we change the flags(which will work for both gtests and isolated-script-tests) to:
1. --isolated-script-test-filter=test1::test2::test3
2. --isolated-script-test-repeat=30
3. --isolated-script-test-launcher-retry-limit=0 
4. --isolated-script-test-also-run-disabled-tests

Could you confirm these modified flags look good to you?

lgtm.

Great. LGTM too.

+jdarpinian as FYI.

> 2) For Findit, we also prefer tests to run in [A, B, C, A, B, C] order.

Why? Mind explaining this in detail? It's a bit of work to support this for issue 894254, so I want to make sure it is important.

#16: note that with --story-set-repeat flag, Telemetry already run tests in [A, B, C, A, B, C] order

Note, the Telemetry flag is --pageset-repeat (but certainly could be renamed, --story-set-repeat would be more accurate today).

#17: Are you sure? I don't think that is true. 

Specifically if we just used the --pageset-repeat flag in Telemetry, 

--isolated-script-test-filter=A::B --isolated-script-test-repeat=2

would be equivalent to

--isolated-script-test-filter=B::A --isolated-script-test-repeat=2

See this paste for proof: https://paste.googleplex.com/5713693899751424

(Note that we map "--isolated-script-test-filter=A::B::C" to "--story-filter=(A|B|C)" 

using https://chromium.googlesource.com/chromium/src/+/e8f5446356355e6c6d5309d931d020945436639c/testing/scripts/run_telemetry_benchmark_as_googletest.py#125 )

In order to support this we would need no longer use regex for our story-filter. https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/story/story_filter.py?type=cs&q=story-filter&sq=package:chromium&g=0&l=100


My question in #16 stands. I don't want to do this work unless you can justify that it will be useful. If you can't, then I will nominally support your API but I will I may potentially do something like:

when given
--isolated-script-test-filter=B::A --isolated-script-test-repeat=2 
I will run
[A, B, A, B]

As an aside, I wouldn't mind dropping regex support for --story-filter. It's pretty confusing and hard to run *only* a story named foo if there is also a story named foo_bar. If we want to preserve some sort of pattern selection mechanism for stories I would go for glob-like '*' and '?' instead (which, unlike the regex '.', do not show up as part of existing story names!).

I'm not sure if I understand the problem here.

Are you asking the order between each test? Like if we want to run the tests in the exact same order as we list them in the --isolated-script-test-filter? I don't think we care that order.

I mean for Findit, we just prefer repeat tests in order like [A, B, A, B] rather than [A, A, B, B]. But it shouldn't matter if the order is [A, B, A, B] or [ B, A, B, A].

Ah, thanks! Please add that to your doc. It isn't clear right now.

#21: Yes, we can look into this. I agree that it is not user friendly currently.

#23, I've updated the doc. Let me know if there is anything else that's unclear.