New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 905330 link

Starred by 1 user
Status: Unconfirmed
Owner: ----
Cc:
erikc...@chromium.org
█ robert...@chromium.org
chanli@chromium.org
lijeffrey@chromium.org
seanmccullough@chromium.org
liaoyuke@chromium.org
st...@chromium.org
zhangtiff@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Inconsistent data between findit and flakiness dashboard

Project Member Reported by mdjones@google.com, Nov 14 
It looks like these two tools try to solve the same problem but one or both have incomplete data (see discussion in https://chromium-review.googlesource.com/c/chromium/src/+/1329161). It would be nice to have a single, unified tool to check for flakes or at least have both tools agree.

In the case here, findit reports tests as very flaky and the flake dash reports that it rarely flakes.

Flake dash: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=chrome_public_test_apk&tests=testModalNativeAppBannerCanBeTriggeredMultipleTimesCustomTab

Findit: https://findit-for-me.appspot.com/flake/occurrences?key=ag9zfmZpbmRpdC1mb3ItbWVykgELEgVGbGFrZSKGAWNocm9taXVtQGNocm9tZV9wdWJsaWNfdGVzdF9hcGtAb3JnLmNocm9taXVtLmNocm9tZS5icm93c2VyLmJhbm5lcnMuQXBwQmFubmVyTWFuYWdlclRlc3QjdGVzdEFwcEluc3RhbGxlZE1vZGFsTmF0aXZlQXBwQmFubmVyQ3VzdG9tVGFiDA

Comment 1 by chanli@chromium.org, Nov 14

Cc: erikc...@chromium.org
The reason Findit thinks org.chromium.chrome.browser.banners.AppBannerManagerTest#testAppInstalledModalNativeAppBannerCustomTab is flaky is because within 24 hours, we need to rerun CQ builds for 3 CLs to get them landed and for all three times, this test Failed with patch twice, succeeded without patch.

Although when take a closer look to those 3 failed builds, the chrome_public_test_apk on Android device Nexus 5 (with patch) steps are abnormal: in two builds there are time-out shard, in one build the failed shard has exit code 87 and unreliable_results.

add erikchen@ to see if he has some insights.

I'm not familiar with flakiness dashboard. I'd love to learn how does it to determine flakes.

Comment 2 by liaoyuke@chromium.org, Nov 14 

>> I'm not familiar with flakiness dashboard. I'd love to learn how does it to determine flakes.

I don't think flakiness dashboard 'determines' flakes, it simply presents the test results, though I'm not sure if flakiness dashboard knows how to understand the test results of retry with patch steps.

Comment 3 by seanmccullough@chromium.org, Nov 14 

re #2: Correct. "Flakiness Dashboard" is a really misleading name that everyone uses for the tests-results.appspot.com UI.  It doesn't know anything about flakes and doesn't try to do any analysis of results over time. So "Flakiness" is only in the name.

It is not very smart about retries and patch steps.
►

Sign in to add a comment