False rejects caused by lack of proper retries for ChromeDriverTest |
|
Issue descriptionThis CL introduced a flaky test: https://chromium-review.googlesource.com/c/chromium/src/+/1392281 This caused a spike in false rejects until it was reverted: https://bugs.chromium.org/p/chromium/issues/detail?id=919075 Looking at a sample build: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/216491 The test failed in 'with_patch' [no retries]. The test succeeded in 'retry_without_patch' [no retries]. The test failed in 'retry_with_patch' [no retries]. The whole build was retried: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/216534 The test failed in 'with_patch' [no retries]. The test failed in 'retry_without_patch' [no retries]. The build was marked as a success. As per https://docs.google.com/document/d/1O9nzVMA6rEe2-rhjsni_8wS9Lw4OnFpK2NE0wWB5VaY/edit#heading=h.xzptrog8pyxf, we need retries so prevent a spike in false rejects when a flaky test is landed, and we should make the retries as cheap as possible. We should add test-suite layer retries to match every other test suite in our code base. |
|
►
Sign in to add a comment |
|