kbr@, to double check,  --isolated-script-test-filter=A::B::C has been supported for all telemetry-based test runners?

If so, supporting --isolated-script-test-filter=A::B::C will be no-op.

Context: https://docs.google.com/document/d/1XZrAecLT33sJng9JTHsWCNpcBVU_BT7weLE76nwsTXA/edit

Yes, these three wrappers for different kinds of Telemetry-based tests:

https://cs.chromium.org/chromium/src/testing/scripts/run_gpu_integration_test_as_googletest.py
https://cs.chromium.org/chromium/src/testing/scripts/run_telemetry_as_googletest.py
https://cs.chromium.org/chromium/src/testing/scripts/run_telemetry_benchmark_as_googletest.py

already support the --isolated-script-test-filter flag.

I think these cover telemetry_perf_unittests, Telemetry's smoke tests, and the Chrome GPU team's tests.

The other flags will have to be plumbed down and some of them (in particular --isolated-script-test-also-run-disabled-tests ) will have different handling in the different Telemetry based tests.

chanli@: how should retry-limit flag be mixed with repeat flag?

e.g: if I run the test with --repeat=4, and --retry-limit=3. Assuming the test suite has two test A & B, and the run with out retry results in:

A_Pass B_Pass A_Fail B_Pass A_Pass B_Pass A_Fail B_Fail

What should the retry run strategy be? Should it be:
[rerun A until it pass (up to 3 times)] [rerun B until it pass (up to 3 times)] [rerun A until it pass (up to 3 times)]

Or should it just be:
[rerun A until it pass (up to 3 times)] [rerun B until it pass (up to 3 times)] ?

Dirk & Ken: thoughts?

re #9, for Findit's use case, we will always set --repeat to an integer greater than 0 and --retry-limit=0.

And IIUC, In your example, it looks the strategy will look like:
[rerun A until it pass (up to 3 times)] [rerun B until it pass (up to 3 times)] [rerun A until it pass (up to 3 times)] ...

Re #9:
To match the behavior of gtest, the logic is:

tests = [test1, test2, ...]
for (int i = 0; i < repeat; ++ i) {
  for test in tests {
    for (int j = 0; j < retry-limit; ++j) {
      passed = test.run()
      if (passed)
        break;
    }
  }
}

#12: I believe most python tests only retry at the end after running through test suite once.

Dirk: should we update either Python test/gtest to have the same retry behavior or we should let them diverge for now?

Actually I believe a major reason why tests should be retried at the end is that's the only sane way to ensure that tests are retried in non-parallel mode. 

For this reason, I think I prefer retrying the failed tests at the end of each run of the suite.

Shutao: aren't gtest run in parallel? 

re: #c13 - we should match the behavior (we should try to have the same behavior in all of our test suites).

I think there are some test suites (layout tests, possibly gtests) that retry the tests in parallel, but I'd have to check the code to confirm.

Re #c14:
gtests runs in parallel for the first-time run, but retry in serial https://cs.chromium.org/chromium/src/base/test/launcher/unit_test_launcher.cc?q=RetryTests

I had bad memory on the detail. I updated the code snippet as below:

tests = [test1, test2, ...]
for (int i = 0; i < repeat; ++ i) {
  failed_tests = ["run tests in parallel, and return failed tests"]
  for test in failed_tests {
    for (int j = 0; j < retry-limit; ++j) {
      passed = test.run()
      if (passed)
        break;
    }
  }
}

#16: so that do behave similar to typ today (except typ doesn't implement --repeat) yet :-)

This super bug is fixed now that all the blockers are resolved.

The arguments are not all working correctly in Findit.