Can you better describe what independent retry means?

Most test suites that have retry logic do so by way of new processes, so that bad state is not carried between runs.

> Can you better describe what independent retry means?

I am observing examples [e.g.  issue 908481 ] where a flaky test [flakes 10% of the time when I run it locally] appears to sometimes deterministically pass or fail on the trybots [e.g. fails 40 times in a row]. 

My current best guess is that this occurs because state is leaking from the parent, test runner process into child processes [maybe from ChromeTestSuite::Initialize (?)].  I have not yet confirmed this. The only other possibility I can think of is that there's machine-specific state that's causing the test to deterministically fail. That seems less likely to me.

unit_tests do run in the same process, which means previous state can certainly impact tests. I believe retries are started in a new process, hence the pass on retry.

This shouldn't impact browser_tests though, as browser_tests always run in a new process.

I finished investigating this particular source of flakiness, see  Issue 908481 . 

The test was incorrectly written and relied on the timing of a background thread being woken up and executing a task. This is entirely at the whims of the OS scheduler and availability of other cores on the device. I imagine that if there are no other available cores, then the test will very likely fail. This seems more likely to happen on the trybots, as those are run on VM images with hardware sharing.

I'm going to leave this bug open, and available for now, as there are still improvements that we could make to the test suite runner itself to make retries more independent.

As a quick test, I ran unit_tests on Linux, macOS and Windows with --test-launcher-batch-limit=1. Note that the default batch size is 10. This causes a new process to be spawned for every test.

======Linux [building for CrOS]======
[this is with target_os = "chromeos", and same GN config as the crOS rel trybot: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux-chromeos-rel/144853]

On my Linux 440 device [12 cores w/ HT], unit_tests takes 
$ time ./out/gn/unit_tests  --test-launcher-bot-mode --cfi-diag=0
real	1m26.581s
user	6m9.316s
sys	1m45.496s

This is with the default batch limit of 10:
https://cs.chromium.org/chromium/src/base/test/launcher/unit_test_launcher.cc?l=47

When I set --test-launcher-batch-limit=1, unit_tests takes:
$ time ./out/gn/unit_tests  --test-launcher-bot-mode --cfi-diag=0 --test-launcher-batch-limit=1
real	5m14.393s
user	31m12.820s
sys	7m45.952s

======macOS======
[24 cores w/ HT]. release, static, symbol_level = 0.

$ time ./out/gn/unit_tests  --test-launcher-bot-mode --cfi-diag=0
real	0m59.826s
user	4m35.727s
sys	2m25.196s

$ time ./out/gn/unit_tests  --test-launcher-bot-mode --cfi-diag=0 --test-launcher-batch-limit=1
real	5m47.229s
user	25m28.403s
sys	11m15.812s

=====Windows====
28 cores w/ HT. release, static, symbol_level=1, 32-bit binary. Same as win7_chromium_rel_ng trybot.

$ Measure-Command {.\out\gn\unit_tests --test-launcher-bot-mode --cfi-diag=0}
...
TotalSeconds      : 194.6314689

$ Measure-Command {.\out\gn\unit_tests --test-launcher-bot-mode --cfi-diag=0 --test-launcher-batch-limit=1}
...
TotalSeconds      : 1914.8142326

=====Conclusion======
Setting a batch size of 1 increases run time by ~4-10X. Given how quickly unit_tests runs, even with --test-launcher-batch-limit=1, this seems like acceptable overhead for independent test results. 

Note: retries are already spawned in separate processes, but the original test runs are run in batches of 10.

IIRC, 10 is the default, but it could be as high as 60.
I saw 60 in a local run of some gtest suite.

Available, but no owner or component? Please find a component, as no one will ever find this without one.