New issue
Advanced search Search tips

Issue 919953 link

Starred by 2 users

Issue metadata

Status: Untriaged
Owner: ----
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

False rejects caused by lack of proper retries for ChromeDriverTest

Project Member Reported by erikc...@chromium.org, Jan 8

Issue description

This CL introduced a flaky test:
https://chromium-review.googlesource.com/c/chromium/src/+/1392281

This caused a spike in false rejects until it was reverted:
https://bugs.chromium.org/p/chromium/issues/detail?id=919075

Looking at a sample build:
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/216491

The test failed in 'with_patch' [no retries].
The test succeeded in 'retry_without_patch' [no retries].
The test failed in 'retry_with_patch' [no retries].

The whole build was retried:
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/216534

The test failed in 'with_patch' [no retries].
The test failed in 'retry_without_patch' [no retries].
The build was marked as a success.

As per https://docs.google.com/document/d/1O9nzVMA6rEe2-rhjsni_8wS9Lw4OnFpK2NE0wWB5VaY/edit#heading=h.xzptrog8pyxf, we need retries so prevent a spike in false rejects when a flaky test is landed, and we should make the retries as cheap as possible. We should add test-suite layer retries to match every other test suite in our code base.
 

Sign in to add a comment