Harden unit_tests and other test targets against flakiness regressions |
|||
Issue descriptionChrome Version: (copy from chrome://version) OS: (e.g. Win7, OSX 10.9.5, etc...) What steps will reproduce the problem? (1) unit_tests with --test-launcher-retry-limit=0 What is the expected result? Expect that all the tests pass, every time. What happens instead? On my workstation[1], ~3 tests fail and ~70-75 actually crash; I'll file separate bugs for burning down those flakes. It's easy for these regressions to creep in because we have a default test-launcher retry-limit of 3 times. For the vast majority of unit-tests, there is no legitimate reason for flake - we should be able to switch those tests to run without retries - but some tests are inherently fragile e.g. due to pesky operating system edge-case conditions. To allow for tests which are valuable most of the time but inevitably flake once in a while, we could add an annotation to the test name (remember our old friend FLAKY_, anyone), which would tell the test-launcher to allow that individual test to be retried, and disallow retries for all other tests. e.g. if we named the annotation EVIL_RETRY_HACK_ then we might have: TEST(ExtensionServiceTest, LovelySimpleTest) TEST(ExtensionServiceTest, EVIL_RETRY_HACK_TestThatReliesOnSystemResourceAndSometimesFailsOnXP)
,
Mar 2 2017
Somewhat related proposal is tracked in issue 694603.
,
Jul 11 2017
kbr: WDYT - should we try this? :)
,
Jul 11 2017
Maybe. I agree there's no excuse for unit tests to be flaking. Talk with Pawel and Sergiy; Pawel wrote the test harness and Sergiy's working on the flakiness pipeline.
,
Jul 17 2017
A few weeks back jbudorick, dpranke, and I were talking about test-level timeouts to prevent regressions, it would be good to think about test-level retries along with that. |
|||
►
Sign in to add a comment |
|||
Comment 1 by efoo@google.com
, Mar 1 2017