[Android WebView] Remove retry logic on instrumentation tests |
||
Issue descriptionGather some data first with scraping.. (since flakiness dashboard doesn't help due to crbug.com/609588 ) This is the failed webview tests from the release tester (https://build.chromium.org/p/chromium.linux/builders/Android%20Tests) in the last 200 runs. There are some data quality issues, but whatever, good enough. 3 org.chromium.android_webview.test.AndroidScrollIntegrationTest#testTouchScrollCanBeAlteredByUi 3 org.chromium.android_webview.test.AwContentsRenderTest#testForceDrawWhenInvisible 17 org.chromium.android_webview.test.AwQuotaManagerBridgeTest#testDeleteAllWithAppCache 74 org.chromium.android_webview.test.AwQuotaManagerBridgeTest#testDeleteOriginWithAppCache 15 org.chromium.android_webview.test.AwQuotaManagerBridgeTest#testGetResultsMatch 1 org.chromium.android_webview.test.AwQuotaManagerBridgeTest#testGetResultsMatch. 1 org.chromium.android_webview.test.AwSettingsTest#testAppCacheWithTwoViews. 2 org.chromium.android_webview.test.WebKitHitTestTest#testUnknownTypeUnrecognizedNode 3 org.chromium.android_webview.test.WebKitHitTestTest#testUnknownTypeUnrecognizedNode. So... should nuke AwQuotaManagerBridgeTest from orbit, then webview tests are probably no worse than chrome_public_apk tests. I wrote those tests /o\ Checking debug bot next.
,
May 6 2016
is fixing tests not an option, and a better option? what am I missing.
,
May 6 2016
Oh, missed like 40 builds from the debug bot because the url changed. Test name changed or something like that.
,
May 6 2016
> is fixing tests not an option, and a better option? what am I missing. Of course. But when was the last time you or anyone else on the webview team fixed a flaky test? If it's not affecting developers and getting someone yelling at you because you because a flaky test just blocked their CL from landing for the nth time, no one is going to care. And the problem just gets worse and worse. Part of the problem is discoverbility. The past 200 webview test runs that I scraped are all *green*. Even if failed tests are reported to the flakiness dashboard, who's gonna look at it unless it affects your day-to-day work flow. Also my opinion to disable flaky tests, instead of leaving them running and adding tools to workaround flakes. Not everyone agrees with that though.
,
May 6 2016
disabling is fine. and making it more discoverable too. as long as you don't remove completely.
,
May 6 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/24c4d329de36e9fba9a79f491709428da78ffd34 commit 24c4d329de36e9fba9a79f491709428da78ffd34 Author: boliu <boliu@chromium.org> Date: Fri May 06 23:32:59 2016 aw: Disable quota manager tests Too flaky. See crbug.com/609980 . BUG= 609980 , 599192 Review-Url: https://codereview.chromium.org/1960833002 Cr-Commit-Position: refs/heads/master@{#392202} [modify] https://crrev.com/24c4d329de36e9fba9a79f491709428da78ffd34/android_webview/javatests/src/org/chromium/android_webview/test/AwQuotaManagerBridgeTest.java
,
May 25 2016
Found this bug while seeking a possible explanation for the recent increase in AwContentsRenderTest#testForceDrawWhenInvisible flake (https://build.chromium.org/p/chromium.linux/builders/Android%20Tests?numbuilds=200). So I guess it's just been flaky for a while.
,
May 25 2016
,
Sep 27
|
||
►
Sign in to add a comment |
||
Comment 1 by boliu@chromium.org
, May 6 2016Same thing for debug bot: 1 org.chromium.android_webview.test.AndroidScrollIntegrationTest#testTouchScrollCanBeAlteredByUi 2 org.chromium.android_webview.test.AwContentsClientShouldOverrideUrlLoadingTest#testCallDestroyInCallback 1 org.chromium.android_webview.test.AwContentsRenderTest#testForceDrawWhenInvisible 1 org.chromium.android_webview.test.AwLegacyQuirksTest#testInitialScaleClobberQuirk 1 org.chromium.android_webview.test.AwLegacyQuirksTest#testInitialScaleClobber 24 org.chromium.android_webview.test.AwQuotaManagerBridgeTest#testDeleteAllWithAppCache 45 org.chromium.android_webview.test.AwQuotaManagerBridgeTest#testDeleteOriginWithAppCache 2 org.chromium.android_webview.test.AwQuotaManagerBridgeTest#testGetResultsMatch 1 org.chromium.android_webview.test.WebKitHitTestTest#testUnknownTypeUnrecognizedNode. Weirdly debug bot appears *less flaky* than the release bot.