False positive for waterfall test failures due to flaky test |
|
Issue descriptionWhat is the bug or feature: NSS/ClientCertStoreTest/0.CertAuthorityFiltering in net_unittests has gone flakey recently (see bug 716594). Find-it identified https://codereview.chromium.org/2844963005#msg14 as the culprit. I reverted said patch, but the flake continues. Context or examples: Expected:
,
May 1 2017
I checked on the analyses and it appeared that 2 independent try jobs identified the same CL as culprit. I think there are 2 places we need to improve: 1. We need to fix the grouping. In this case, at least one analysis in group has identified the test to be flaky, a. in this case we may consider increase the iterations in swarming reruns for the other analyses in group to confirm if the failures are flaky b. evaluate the possibility of the same test is flaky on some builds and reliably fails on some other builds at the same time. If the possibility is very low, I may consider treating all failures in a group as flaky if any of the results is flaky. 2. For both analyses, in the try jobs the test on the revision before the culprit is skipped. We should consider always run the previous revision to confirm the culprit is correct.
,
May 1 2017
Make sure you follow the status of 716594. Latest theory is a potential difference in bot config, which would throw off find-it. |
|
►
Sign in to add a comment |
|
Comment 1 by st...@chromium.org
, May 1 2017Cc: -chanli@chromium.org robert...@chromium.org
Components: -Tools>Test>FindIt Tools>Test>FindIt>Waterfall
Owner: chanli@chromium.org
Status: Assigned (was: Unconfirmed)
Summary: False positive for waterfall test failures due to flaky test (was: Findit bug)