New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 717129 link

Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: ----

Blocked on:
issue 706940



Sign in to add a comment

False positive for waterfall test failures due to flaky test

Project Member Reported by sky@chromium.org, May 1 2017

Issue description

What is the bug or feature:

NSS/ClientCertStoreTest/0.CertAuthorityFiltering in net_unittests has gone flakey recently (see bug 716594). Find-it identified https://codereview.chromium.org/2844963005#msg14 as the culprit. I reverted said patch, but the flake continues.


Context or examples:


Expected:


 

Comment 1 by st...@chromium.org, May 1 2017

Blockedon: 706940
Cc: -chanli@chromium.org robert...@chromium.org
Components: -Tools>Test>FindIt Tools>Test>FindIt>Waterfall
Owner: chanli@chromium.org
Status: Assigned (was: Unconfirmed)
Summary: False positive for waterfall test failures due to flaky test (was: Findit bug)
Many thanks for the report! We will investigate this.

Chan, assign to you as this is test failure, and the original analysis result is not shown due to the failure grouping.
I checked on the analyses and it appeared that 2 independent try jobs identified the same CL as culprit. I think there are 2 places we need to improve:
1. We need to fix the grouping. In this case, at least one analysis in group has identified the test to be flaky, 
  a. in this case we may consider increase the iterations in swarming reruns for the other analyses in group to confirm if the failures are flaky
  b. evaluate the possibility of the same test is flaky on some builds and reliably fails on some other builds at the same time. If the possibility is very low, I may consider treating all failures in a group as flaky if any of the results is flaky.

2. For both analyses, in the try jobs the test on the revision before the culprit is skipped. We should consider always run the previous revision to confirm the culprit is correct.

Comment 3 by sky@chromium.org, May 1 2017

Make sure you follow the status of 716594. Latest theory is a potential difference in bot config, which would throw off find-it.

Sign in to add a comment