[Findit] Flake Analyzer - Avoid sending notifications to code reviews if not 100% stable - > flaky |
||
Issue descriptionCurrently notifications are sent based only on confidence score. To avoid false positives, also only send notifications if the pass rates go from fully stable -> flaky, e.g. 100% or 0% to 40%, not 99% -> 40%
,
Sep 29 2017
Check for a series of fully-stable points preceding the culprit before updating. If any of the preceding points within the window are even slightly flaky (e.g. 99% passing), bail out of updating the culprit.
,
Oct 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/02fe56406735011f9db69be729ea3ce747f20752 commit 02fe56406735011f9db69be729ea3ce747f20752 Author: Jeffrey Li <lijeffrey@chromium.org> Date: Tue Oct 03 21:28:06 2017 [Findit] Flake Analyzer - Smarter logic for sending notifications to code reviews Factor out all logic to determine whether notifications should be sent into a dedicated function according to the following rules in order of precedence: 1. Never send a notification if Findit is not configured to do so. 2. Never send if the culprit has already been notified by Findit. 3. Always send if the culprit introduced a new flaky test. 4. Never send if the culprit is not preceded by a minimum number of fully-passing or fully-failing data points. 5. Never send if there is insufficient confidence that the culprit is indeed responsible. Bug: 770086 Change-Id: I8d2cdfe3a885b34a6609df1a8a0260e6c93c6dee Reviewed-on: https://chromium-review.googlesource.com/693340 Reviewed-by: Shuotao Gao <stgao@chromium.org> Commit-Queue: Jeffrey Li <lijeffrey@chromium.org> [modify] https://crrev.com/02fe56406735011f9db69be729ea3ce747f20752/appengine/findit/waterfall/flake/flake_constants.py [modify] https://crrev.com/02fe56406735011f9db69be729ea3ce747f20752/appengine/findit/waterfall/flake/test/send_notification_for_flake_culprit_pipeline_test.py [modify] https://crrev.com/02fe56406735011f9db69be729ea3ce747f20752/appengine/findit/waterfall/flake/send_notification_for_flake_culprit_pipeline.py
,
Oct 4 2017
|
||
►
Sign in to add a comment |
||
Comment 1 by lijeffrey@chromium.org
, Sep 29 2017