New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 697215 link

Starred by 1 user

Issue metadata

Status: Untriaged
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 2
Type: Bug



Sign in to add a comment

Harden unit_tests and other test targets against flakiness regressions

Project Member Reported by w...@chromium.org, Feb 28 2017

Issue description

Chrome Version: (copy from chrome://version)
OS: (e.g. Win7, OSX 10.9.5, etc...)

What steps will reproduce the problem?
(1) unit_tests with --test-launcher-retry-limit=0

What is the expected result?

Expect that all the tests pass, every time.

What happens instead?

On my workstation[1], ~3 tests fail and ~70-75 actually crash; I'll file separate bugs for burning down those flakes.

It's easy for these regressions to creep in because we have a default test-launcher retry-limit of 3 times.

For the vast majority of unit-tests, there is no legitimate reason for flake - we should be able to switch those tests to run without retries - but some tests are inherently fragile e.g. due to pesky operating system edge-case conditions.

To allow for tests which are valuable most of the time but inevitably flake once in a while, we could add an annotation to the test name (remember our old friend FLAKY_, anyone), which would tell the test-launcher to allow that individual test to be retried, and disallow retries for all other tests.

e.g. if we named the annotation EVIL_RETRY_HACK_ then we might have:

TEST(ExtensionServiceTest, LovelySimpleTest)
TEST(ExtensionServiceTest, EVIL_RETRY_HACK_TestThatReliesOnSystemResourceAndSometimesFailsOnXP)



 

Comment 1 by efoo@google.com, Mar 1 2017

Components: -Infra Infra>Flakiness
Somewhat related proposal is tracked in issue 694603.

Comment 3 by w...@chromium.org, Jul 11 2017

kbr: WDYT - should we try this? :)

Comment 4 by kbr@chromium.org, Jul 11 2017

Cc: serg...@chromium.org phajdan.jr@chromium.org
Components: Tests
Maybe. I agree there's no excuse for unit tests to be flaking. Talk with Pawel and Sergiy; Pawel wrote the test harness and Sergiy's working on the flakiness pipeline.

Comment 5 by estaab@chromium.org, Jul 17 2017

Cc: jbudorick@chromium.org dpranke@chromium.org
A few weeks back jbudorick, dpranke, and I were talking about test-level timeouts to prevent regressions, it would be good to think about test-level retries along with that.

Sign in to add a comment