Flake detection idea: ignore without path rerun step |
|||
Issue descriptionIgnore the retry logic, and determine a test as a flaky test if a test fails at test-runner level for 3 unrelated CLs. This is a very nice change to have as it makes things simpler and would make the detection logic naturally support other projects such as iOS and WebRTC. There is a case that this change would make the flake detection fail to catch: b1: t1 (with patch) t1 (without patch) b2: t2 (with patch) t2 (without patch) So, if t1 is a consistent failure happened to b1, but then it is reverted, and before b2 is triggered, t2 becomes a consistent failure. Given that we need 3 different CLs to observe the exact same test failures, my initial judgement is that the chance this will happen is very rare, but more investigations are needed to verify that!
,
Jan 17
(5 days ago)
,
Jan 17
(5 days ago)
Hi Chan, here is the idea I mentioned.
,
Jan 17
(5 days ago)
Thanks. Yeah I had a similar idea while working on assigning weights to flake types though your idea is at a higher level! This is definitely a direction we should explore. |
|||
►
Sign in to add a comment |
|||
Comment 1 by liaoyuke@chromium.org
, Aug 2