New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 889548 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Oct 31
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Surface flaky tests hid by "retry with patch" steps

Project Member Reported by liaoyuke@chromium.org, Sep 26

Issue description

 crbug.com/883321  introduced 'retry with patch', and Flake Detector should be adapted to detect and surface the flaky tests that failed in 'with patch' steps but succeeded later in 'retry with patch' steps.
 
I'm not sure if this change will affect our queries for false rejections. 

But in my opinion, this kind of flaky tests can be classified as 'hidden flakes' that we plan to support in the near future.
It shouldn't change the false rejection queries, since those are based on the results of the build, not individual steps in the build.
Yes, cq false rejection queries doesn't need to change, but with the 'retry with patch' steps, a lot of flakes won't cause false rejections now, which means that they won't be detected now.

So, in order to surface those flaky tests failed in 'with patch' steps but succeeded later in 'retry with patch' steps, a new query is needed.
Yes, definitely other queries will need to change :).
Project Member

Comment 5 by bugdroid1@chromium.org, Oct 5

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/b60982db761a2140aaf13a4856e82c8df124918d

commit b60982db761a2140aaf13a4856e82c8df124918d
Author: Yuke Liao <liaoyuke@chromium.org>
Date: Fri Oct 05 20:50:57 2018

[Findit] Add query to detect retry with try hidden flakes

This CL adds a query to detect flaky tests hidden by retry with patch
steps. The retry with patch can be either a success or failure:
1. t1 and t2 failed in with patch step, and both t1 and t2 passed in
   retry with patch step. The retry with patch step is a success and
   both t1 and t2 are valid flaky tests.
2. t1 and t2 failed in with patch step, and t1 passed in
   retry with patch step, but t2 failed again. The retry with patch
   step is a failure, and t1 is a valid flaky test, but t2 is not.

Bug:  889548 
Change-Id: I0c64cb4e970ecd403f780c121d9a98af55db6934
Reviewed-on: https://chromium-review.googlesource.com/c/1258590
Commit-Queue: Yuke Liao <liaoyuke@chromium.org>
Reviewed-by: Shuotao Gao <stgao@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18119}
[add] https://crrev.com/b60982db761a2140aaf13a4856e82c8df124918d/appengine/findit/services/flake_detection/flaky_tests.retry_with_patch.sql

Project Member

Comment 6 by bugdroid1@chromium.org, Oct 16

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/1808a9be4e336a9f114443d1970b12c645497a56

commit 1808a9be4e336a9f114443d1970b12c645497a56
Author: Yuke Liao <liaoyuke@chromium.org>
Date: Tue Oct 16 06:37:26 2018

[Findit] Adapt the model for retry with patch flakes

This CL adapts the model for retry with patch, and it is a pure
refactoring CL, and once this CL lands, data needs to be migrated.

Bug:  889548 
Change-Id: I9d0d7cb80b3541ca56991a11c4653009acef0ac5
Reviewed-on: https://chromium-review.googlesource.com/c/1266175
Reviewed-by: Chan Li <chanli@chromium.org>
Commit-Queue: Yuke Liao <liaoyuke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18346}
[modify] https://crrev.com/1808a9be4e336a9f114443d1970b12c645497a56/appengine/findit/model/flake/detection/test/flake_occurrence_test.py
[modify] https://crrev.com/1808a9be4e336a9f114443d1970b12c645497a56/appengine/findit/services/test/apis_test.py
[modify] https://crrev.com/1808a9be4e336a9f114443d1970b12c645497a56/appengine/findit/services/flake_reporting/test/component_test.py
[modify] https://crrev.com/1808a9be4e336a9f114443d1970b12c645497a56/appengine/findit/services/flake_detection/test/detect_cq_false_rejection_flakes_test.py
[modify] https://crrev.com/1808a9be4e336a9f114443d1970b12c645497a56/appengine/findit/services/flake_reporting/component.py
[modify] https://crrev.com/1808a9be4e336a9f114443d1970b12c645497a56/appengine/findit/services/flake_detection/update_flake_counts_service.py
[modify] https://crrev.com/1808a9be4e336a9f114443d1970b12c645497a56/appengine/findit/handlers/flake/detection/flake_detection_utils.py
[modify] https://crrev.com/1808a9be4e336a9f114443d1970b12c645497a56/appengine/findit/services/flake_detection/test/flake_report_util_test.py
[modify] https://crrev.com/1808a9be4e336a9f114443d1970b12c645497a56/appengine/findit/services/flake_detection/test/update_flake_counts_service_test.py
[modify] https://crrev.com/1808a9be4e336a9f114443d1970b12c645497a56/appengine/findit/model/flake/detection/flake_occurrence.py
[modify] https://crrev.com/1808a9be4e336a9f114443d1970b12c645497a56/appengine/findit/services/flake_detection/flake_report_util.py
[modify] https://crrev.com/1808a9be4e336a9f114443d1970b12c645497a56/appengine/findit/services/flake_detection/detect_cq_false_rejection_flakes.py
[modify] https://crrev.com/1808a9be4e336a9f114443d1970b12c645497a56/appengine/findit/handlers/flake/detection/test/show_flake_test.py
[modify] https://crrev.com/1808a9be4e336a9f114443d1970b12c645497a56/appengine/findit/handlers/flake/detection/test/flake_detection_utils_test.py

Status: Fixed (was: Assigned)

Sign in to add a comment