New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 723694 link

Starred by 1 user
Status: Fixed
Owner:
chanli@chromium.org
Closed: May 2017
Cc:
█ robert...@chromium.org
chanli@chromium.org
lijeffrey@chromium.org
st...@chromium.org
liaoyuke@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: ----
Type: ----

Blocking:
issue 725132



Sign in to add a comment

recipe findit/chromium/test doesn't handle flaky tests properly

Project Member Reported by thakis@chromium.org, May 17 2017 
Detail is https://findit-for-me.appspot.com/waterfall/culprit?key=ag9zfmZpbmRpdC1mb3ItbWVyRAsSDVdmU3VzcGVjdGVkQ0wiMWNocm9taXVtLzAyNGQ2ZjM2M2RkN2NhMWI5MWEzOWVjNjU2NWVkZmExYjc5NmVkMjQM

I think this is incorrect. https://codereview.chromium.org/2884303002/ deletes the _apk_run targets which are only used on Android, and it updates mb.py in a way that might have caused test binaries to have compile failures during build or that cause test binaries to not start at all, but I don't think it can be responsible for making a small number of tests fail.

Comment 1 by st...@chromium.org, May 17 2017

Owner: chanli@chromium.org
Status: Assigned (was: Available)
Many thanks Nico for the report!

The false positive is due to the low flakiness in PageLoadMetricsBrowserTest.HttpErrorPage. We will fix this.
The actual culprit is https://codereview.chromium.org/2887773003

chanli@, the recipe findit/chromium/test needs some adaption to deal with flaky tests. For PageLoadMetricsBrowserTest.HttpErrorPage, it failed 1 out 20 runs at revision 024d6f363dd7ca1b91a39ec6565edfa1b796ed24, but the recipe still determined it as the culprit.

Comment 2 by st...@chromium.org, May 17 2017

Summary: recipe findit/chromium/test doesn't handle flaky tests properly (was: Wrongly blame 472244)

Comment 3 by chanli@chromium.org, May 22 2017

Blocking: 725132
Project Member

Comment 4 by bugdroid1@chromium.org, May 26 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/03fd7c33f6131a4364d5793cf525d2dc0267cafe

commit 03fd7c33f6131a4364d5793cf525d2dc0267cafe
Author: Chan <chanli@chromium.org>
Date: Fri May 26 22:31:42 2017

[Findit] Remove culprit for flaky tests from result  in test recipe.

In test recipe, check if a test failure is flaky or reliable, if flaky, don't add current revision
as culprit, but keep running this test to find the culprit for it's reliable failure.

TBR=phajdan.jr@chromium.org

Bug:  723694 
Change-Id: Idcaf5f4b550fcd76d4db65726c6eeaefdafb0bd7
Reviewed-on: https://chromium-review.googlesource.com/514326
Reviewed-by: Chan Li <chanli@chromium.org>
Reviewed-by: Shuotao Gao <stgao@chromium.org>
Reviewed-by: Roberto Carrillo <robertocn@chromium.org>
Commit-Queue: Chan Li <chanli@chromium.org>

[add] https://crrev.com/03fd7c33f6131a4364d5793cf525d2dc0267cafe/scripts/slave/recipes/findit/chromium/test.expected/remove_culprits_for_flaky_failures.json
[modify] https://crrev.com/03fd7c33f6131a4364d5793cf525d2dc0267cafe/scripts/slave/recipes/findit/chromium/test.expected/findit_tests_multiple_culprits.json
[modify] https://crrev.com/03fd7c33f6131a4364d5793cf525d2dc0267cafe/scripts/slave/recipe_modules/test_utils/test_api.py
[modify] https://crrev.com/03fd7c33f6131a4364d5793cf525d2dc0267cafe/scripts/slave/recipes/findit/chromium/test.py

Comment 5 by chanli@chromium.org, May 30 2017

Status: Fixed (was: Assigned)
►

Sign in to add a comment