New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

[Umbrella Bug] New flake detection

Project Member Reported by st...@chromium.org, Apr 27 2018

Issue description

1. Understand Chromium-Try-Flakes:
   * P0 how it detects flaky tests in CQ:  https://cs.chromium.org/chromium/infra/appengine/chromium_try_flakes/status/cq_status.py?q=fetch_cq_status
   * P1 how it files/updates Monorail bugs of flaky tests
   * Ask stgao@ for docs about CTF
   * The task here is to make CTF report detected flakes to Findit even after CTF hits the limit of bug updates
2. Review the new BQ query to detect flaky tests: https://chromium-review.googlesource.com/c/infra/infra/+/1031017
   * Current query is https://cs.chromium.org/chromium/infra/appengine/findit/services/flake_detection/detect_flakes.py?q=_CQ_FLAKES_QUERY
   * The task here is to define the output schema of the query that are needed to record flake occurrences, find flaky tests, file/update bugs, and show info on UI/bugs, etc.
3. Understand the data model of the new flake detection on Findit side: https://cs.chromium.org/chromium/infra/appengine/findit/cron.yaml?q=detect-cq-flakes
   * The task here is to refine the data model to support multiple projects.
     Besides Chromium, WebRTC team has feature request to support their cases as well. Later we might want to support v8 too.
   * A design doc might be appropriate here.
4. The end goal of this is to have a better & standalone flake detection as a appengine module/service within Findit, and then deprecate CTF.
 
Blockedon: 839031

Comment 2 by st...@chromium.org, May 5 2018

Blocking: 840071
Blockedon: 840829
Blockedon: 840831
Blockedon: 840832
Blockedon: 840834
Blockedon: 841377
Blockedon: 841547
Blockedon: 844567
Blockedon: 844569
Blockedon: 844573
Summary: [Umbrella Bug] New flake detection (was: New flake detection)
Blockedon: 844655

Comment 14 by st...@chromium.org, May 19 2018

Blockedon: 844812
Blockedon: 845581
Blockedon: 845969
Blockedon: 845984
Blockedon: 846384
Blockedon: 846470
Blockedon: 846807
Blockedon: 847546
Blockedon: 848867
Blockedon: 849571
Blockedon: -846470
Blockedon: 849837
Blockedon: 850710
Blockedon: 850714
Blockedon: 852635

Comment 29 by st...@chromium.org, Jun 21 2018

Blockedon: monorail:3937
Blockedon: 856464
Blockedon: 856652
Blockedon: 857339
Blockedon: 846470
Blockedon: 857086
Blockedon: 864573
Blockedon: 856371
Blockedon: 870395
Labels: flake-detection
Blockedon: 870753
Blockedon: 859763
Blockedon: 870894
Blockedon: 872036
Blockedon: 872042
Blockedon: 872787
Blockedon: 873190
Blockedon: 873256
Blockedon: 873702
Blockedon: 873704
Blockedon: 873708
Blockedon: 874232
Blockedon: 875055
Blockedon: 875899
Blockedon: 875992
Blockedon: 876797
Blockedon: 876912
Blockedon: 877562
Blockedon: 877974
Blockedon: 878828
Blockedon: 879683
Blockedon: 879764
Blockedon: 883907
is this done now, since CTF is gone?
Owner: chanli@chromium.org
This is done from the perspective of replacing CTF. But there are a few more things we want to add or improve.

Reassigned to chanli@ to triage the blocking bugs before closing this bug.

Sign in to add a comment