New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 844567 link

Starred by 2 users

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocking:
issue 837855



Sign in to add a comment

Flake detection idea: ignore without path rerun step

Project Member Reported by liaoyuke@chromium.org, May 18 2018

Issue description

Ignore the retry logic, and determine a test as a flaky test if a test fails at test-runner level for 3 unrelated CLs.

This is a very nice change to have as it makes things simpler and would make the detection logic naturally support other projects such as iOS and WebRTC.

There is a case that this change would make the flake detection fail to catch:
b1: t1 (with patch) t1 (without patch)
b2: t2 (with patch) t2 (without patch)

So, if t1 is a consistent failure happened to b1, but then it is reverted, and before b2 is triggered, t2 becomes a consistent failure.

Given that we need 3 different CLs to observe the exact same test failures, my initial judgement is that the chance this will happen is very rare, but more investigations are needed to verify that!
 
Labels: flake-detection

Comment 2 by liaoyuke@chromium.org, Jan 17 (5 days ago)

Description: Show this description

Comment 3 by liaoyuke@chromium.org, Jan 17 (5 days ago)

Owner: ----
Status: Available (was: Assigned)
Hi Chan, here is the idea I mentioned.

Comment 4 by chanli@chromium.org, Jan 17 (5 days ago)

Thanks. Yeah I had a similar idea while working on assigning weights to flake types though your idea is at a higher level! This is definitely a direction we should explore.

Sign in to add a comment