Make ranked-flakes/ support a single test and show test result status for each occurrence |
||||
Issue descriptionWe have many tests and many builders. Because of this, the flakiness dashboard is (understandably) sluggish. I would like to be able to find failures for a specific test on a specific builder for a longer date range (currently I see about a day's worth), but it is not clear if or how this is supported. If I click on a specific test, then specify a specific builder, e.g.: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#builder=chromium.chromiumos%3Alinux-chromeos-rel&tests=WebBluetoothTest.NavigateWithChooserCrossOrigin I get results for every test again. It would be super helpful to be able to see just FAIL results for a specific test on a specific builder, over the last week (or month even). https://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=BluetoothApiTest.OnAdapterStateChanged&builder=chromium.chromiumos%3Alinux-chromeos-rel&result=FAIL
,
Dec 13
That's super helpful, thanks! I'd actually like to examine the flakiness of a particular test over ~6 months (or whatever period is available), but there is no indication how far back tests are analyzed, or any means to affect that: https://findit-for-me.appspot.com/waterfall/list-flakes?test_name=WebBluetoothTest.NavigateWithChooserCrossOrigin
,
Dec 13
Findit has stored all flake occurrences it detects, and it displays all flakes that have happened within the past week on https://findit-for-me.appspot.com/ranked-flakes. For your feature request, we can make a small change on the ranked-flakes dashboard to show occurrences of a single old flake if it's searched. But since Flake detection is a new system, we may not have old enough data for this particular test you mentioned in this issue. Though I believe it will be more and more useful when it runs longer and has more data,
,
Dec 13
Indeed, this is great, and should get better over time! Just showing the range in the results for list-flakes for a specific test would be very helpful. It looks like recent test failures may be missing also?
,
Dec 13
For https://findit-for-me.appspot.com/waterfall/list-flakes?test_name=WebBluetoothTest.NavigateWithChooserCrossOrigin, those are analyses to find culprit for flaky test occurrences. For each analysis, Findit looks back 5000 commits from the reported flake revision. list-flakes/ only analyzes flake failure occurrences of the test, and not consistent failures.
,
Dec 13
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/9aa8b1883089617ef0652f976de8d3be5c432f9c commit 9aa8b1883089617ef0652f976de8d3be5c432f9c Author: Chan <chanli@chromium.org> Date: Thu Dec 13 23:20:06 2018 [Findit] Show occurrences of an old flake if searched. Bug: 914962 Change-Id: I33989ee0e3bbca03697ffe476fdc0d3b8ab4ba18 Reviewed-on: https://chromium-review.googlesource.com/c/1377110 Reviewed-by: Shuotao Gao <stgao@chromium.org> Commit-Queue: Shuotao Gao <stgao@chromium.org> Cr-Commit-Position: refs/heads/master@{#19553} [modify] https://crrev.com/9aa8b1883089617ef0652f976de8d3be5c432f9c/appengine/findit/handlers/flake/detection/test/rank_flakes_test.py [modify] https://crrev.com/9aa8b1883089617ef0652f976de8d3be5c432f9c/appengine/findit/handlers/flake/detection/rank_flakes.py
,
Dec 27
|
||||
►
Sign in to add a comment |
||||
Comment 1 by st...@chromium.org
, Dec 13