New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Dec 20
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment
link

Issue 896006: [Findit] Flake Detection: support hidden flakes by 'retry_with_patch' step

Reported by chanli@chromium.org, Oct 16 Project Member

Issue description

support hidden flakes by 'retry_with_patch' step

1. store occurrences of this type
2. rank flakes properly
3. Display occurrences of this new type
 

Comment 1 by bugdroid1@chromium.org, Oct 17

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/38a003cd576042f35cff02dd197e94e59fdde978

commit 38a003cd576042f35cff02dd197e94e59fdde978
Author: Chan <chanli@chromium.org>
Date: Wed Oct 17 21:51:25 2018

[Findit] Store detected 'retry_with_patch' flakes.

In this CL, run the flaky_tests.retry_with_patch.sql to detect 'retry_with_patch' flake occurrences and save them in datastore.

Flake model change and counting will be in a separate CL.

Bug:  896006 
Change-Id: Ifffae40f39fce2fc9f5ad2fdd377455b814bdf3e
Reviewed-on: https://chromium-review.googlesource.com/c/1285470
Commit-Queue: Chan Li <chanli@chromium.org>
Reviewed-by: Yuke Liao <liaoyuke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18408}
[modify] https://crrev.com/38a003cd576042f35cff02dd197e94e59fdde978/appengine/findit/handlers/flake/detection/test/detect_flakes_test.py
[rename] https://crrev.com/38a003cd576042f35cff02dd197e94e59fdde978/appengine/findit/services/flake_detection/test/detect_flake_occurrences_test.py
[rename] https://crrev.com/38a003cd576042f35cff02dd197e94e59fdde978/appengine/findit/services/flake_detection/detect_flake_occurrences.py
[modify] https://crrev.com/38a003cd576042f35cff02dd197e94e59fdde978/appengine/findit/handlers/flake/detection/detect_flakes.py
[modify] https://crrev.com/38a003cd576042f35cff02dd197e94e59fdde978/appengine/findit/handlers/flake/detection/test/show_flake_test.py
[modify] https://crrev.com/38a003cd576042f35cff02dd197e94e59fdde978/appengine/findit/model/flake/detection/flake_occurrence.py
[modify] https://crrev.com/38a003cd576042f35cff02dd197e94e59fdde978/appengine/findit/services/flake_detection/flake_report_util.py
[modify] https://crrev.com/38a003cd576042f35cff02dd197e94e59fdde978/appengine/findit/model/flake/flake_type.py
[modify] https://crrev.com/38a003cd576042f35cff02dd197e94e59fdde978/appengine/findit/handlers/flake/detection/test/flake_detection_utils_test.py

Comment 2 by bugdroid1@chromium.org, Oct 18

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/8c57e39a5e733ff8b7b647a510d53753d82ef000

commit 8c57e39a5e733ff8b7b647a510d53753d82ef000
Author: Chan <chanli@chromium.org>
Date: Thu Oct 18 17:30:10 2018

[Findit] Update flakes with counts for 'retry_with_patch' flakes as well.

This CL is a quick workaround to count 'retry_with_patch' flakes: it counts flakes of different types separately but it reuses the existing false_rejection_count_last_week as if it doesn't differentiate them from false rejections.

This is to bring back flake data on the dashboard asap.

I'll have a follow-up CL to properly use the counts.

Bug:  896006 
Change-Id: I7859608fd09f63c4a898359cc50c67ec447c06ad
Reviewed-on: https://chromium-review.googlesource.com/c/1285471
Commit-Queue: Chan Li <chanli@chromium.org>
Reviewed-by: Yuke Liao <liaoyuke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18439}
[modify] https://crrev.com/8c57e39a5e733ff8b7b647a510d53753d82ef000/appengine/findit/model/flake/detection/test/flake_occurrence_test.py
[modify] https://crrev.com/8c57e39a5e733ff8b7b647a510d53753d82ef000/appengine/findit/services/flake_detection/update_flake_counts_service.py
[modify] https://crrev.com/8c57e39a5e733ff8b7b647a510d53753d82ef000/appengine/findit/services/flake_detection/test/update_flake_counts_service_test.py
[modify] https://crrev.com/8c57e39a5e733ff8b7b647a510d53753d82ef000/appengine/findit/model/flake/flake.py
[modify] https://crrev.com/8c57e39a5e733ff8b7b647a510d53753d82ef000/appengine/findit/handlers/flake/detection/test/show_flake_test.py
[modify] https://crrev.com/8c57e39a5e733ff8b7b647a510d53753d82ef000/appengine/findit/handlers/flake/detection/test/flake_detection_utils_test.py

Comment 3 by bugdroid1@chromium.org, Oct 18

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/41bc2f4b081c25661ede0ef93e5974e2fd439908

commit 41bc2f4b081c25661ede0ef93e5974e2fd439908
Author: Chan <chanli@chromium.org>
Date: Thu Oct 18 17:50:13 2018

[Findit] Show occurrences of 'retry_with_patch' on occurrences pages.

Although if the number of false rejections has hit the cap, 'retry_with_patch' flakes will not show.

Bug:  896006 
Change-Id: Ie7e20d0960b6152b0bfbe94c00a3843ee290e7a1
Reviewed-on: https://chromium-review.googlesource.com/c/1285475
Commit-Queue: Chan Li <chanli@chromium.org>
Reviewed-by: Yuke Liao <liaoyuke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18441}
[modify] https://crrev.com/41bc2f4b081c25661ede0ef93e5974e2fd439908/appengine/findit/handlers/flake/detection/test/flake_detection_utils_test.py
[modify] https://crrev.com/41bc2f4b081c25661ede0ef93e5974e2fd439908/appengine/findit/handlers/flake/detection/flake_detection_utils.py

Comment 4 by bugdroid1@chromium.org, Oct 22

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/ef01ef34fdf17ed2ea69de064b97f8b579f793db

commit ef01ef34fdf17ed2ea69de064b97f8b579f793db
Author: Chan <chanli@chromium.org>
Date: Mon Oct 22 18:02:31 2018

[Findit] File bugs for detected flakes if there are no less than 3 impacted cls within 24 hours.

The impacted cls will include both falsely rejected CLs and CLs imapcted by 'retry with patch' step.

Bug:  896006 
Change-Id: Ieec5d6a08c8efcc37ce9e5c185cd282a97f8b248
Reviewed-on: https://chromium-review.googlesource.com/c/1294049
Reviewed-by: Yuke Liao <liaoyuke@chromium.org>
Commit-Queue: Chan Li <chanli@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18525}
[modify] https://crrev.com/ef01ef34fdf17ed2ea69de064b97f8b579f793db/appengine/findit/services/flake_detection/test/flake_report_util_test.py
[modify] https://crrev.com/ef01ef34fdf17ed2ea69de064b97f8b579f793db/appengine/findit/services/flake_detection/flake_report_util.py
[modify] https://crrev.com/ef01ef34fdf17ed2ea69de064b97f8b579f793db/appengine/findit/docs/flake_detector.md

Comment 5 by bugdroid1@chromium.org, Oct 31

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/1a3d7c60d35a934c07e6d90ac1de20e1033fe46a

commit 1a3d7c60d35a934c07e6d90ac1de20e1033fe46a
Author: Chan Li <chanli@chromium.org>
Date: Wed Oct 31 18:23:40 2018

Revert "[Findit] File bugs for detected flakes if there are no less than 3 impacted cls within 24 hours."

This reverts commit ef01ef34fdf17ed2ea69de064b97f8b579f793db.

Reason for revert: Findit is filing many bugs for each individual flaky test when they have the same cause and always failed at the same time. 

Original change's description:
> [Findit] File bugs for detected flakes if there are no less than 3 impacted cls within 24 hours.
> 
> The impacted cls will include both falsely rejected CLs and CLs imapcted by 'retry with patch' step.
> 
> Bug:  896006 
> Change-Id: Ieec5d6a08c8efcc37ce9e5c185cd282a97f8b248
> Reviewed-on: https://chromium-review.googlesource.com/c/1294049
> Reviewed-by: Yuke Liao <liaoyuke@chromium.org>
> Commit-Queue: Chan Li <chanli@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#18525}

TBR=stgao@chromium.org,chanli@chromium.org,liaoyuke@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  896006 
Change-Id: I18bdd6f81b3fc4a541687c38591f15bfdeeb7fc2
Reviewed-on: https://chromium-review.googlesource.com/c/1310773
Reviewed-by: Chan Li <chanli@chromium.org>
Reviewed-by: Yuke Liao <liaoyuke@chromium.org>
Commit-Queue: Chan Li <chanli@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18697}
[modify] https://crrev.com/1a3d7c60d35a934c07e6d90ac1de20e1033fe46a/appengine/findit/services/flake_detection/test/flake_report_util_test.py
[modify] https://crrev.com/1a3d7c60d35a934c07e6d90ac1de20e1033fe46a/appengine/findit/services/flake_detection/flake_report_util.py
[modify] https://crrev.com/1a3d7c60d35a934c07e6d90ac1de20e1033fe46a/appengine/findit/docs/flake_detector.md

Comment 6 by chanli@chromium.org, Dec 20

Status: Verified (was: Started)

Comment 7 by bugdroid1@chromium.org, Jan 11

Project Member

Comment 8 by bugdroid1@chromium.org, Jan 14

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/16e654e5a0353281e32c3876aa6ee13d04f54010

commit 16e654e5a0353281e32c3876aa6ee13d04f54010
Author: Chan Li <chanli@chromium.org>
Date: Mon Jan 14 23:04:28 2019

Revert "[Findit] Resume filing bugs for retry_with_patch flakes as well."

This reverts commit 26a1ad59d316419b96a98ecfd87dfe4ccb8a6513.

Reason for revert: there are still too many bugs. revert this for now until we can have more inclusive flake grouping conditions.

Original change's description:
> [Findit] Resume filing bugs for retry_with_patch flakes as well.
> 
> Bug:  896006 
> Change-Id: Ieae5286605a0cc823542056416c96501f86021e5
> Reviewed-on: https://chromium-review.googlesource.com/c/1406420
> Reviewed-by: Yuke Liao <liaoyuke@chromium.org>
> Commit-Queue: Chan Li <chanli@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#19933}

TBR=stgao@chromium.org,chanli@chromium.org,liaoyuke@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  896006 
Change-Id: I9f57e4020cfe9c840b0c4fee38865f2cbffe5bed
Reviewed-on: https://chromium-review.googlesource.com/c/1410182
Reviewed-by: Chan Li <chanli@chromium.org>
Reviewed-by: Shuotao Gao <stgao@chromium.org>
Commit-Queue: Chan Li <chanli@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19969}
[modify] https://crrev.com/16e654e5a0353281e32c3876aa6ee13d04f54010/appengine/findit/services/flake_issue_util.py
[modify] https://crrev.com/16e654e5a0353281e32c3876aa6ee13d04f54010/appengine/findit/docs/flake_detector.md
[modify] https://crrev.com/16e654e5a0353281e32c3876aa6ee13d04f54010/appengine/findit/services/test/flake_issue_util_test.py

Sign in to add a comment