New issue
Advanced search Search tips

Issue 609980 link

Starred by 2 users

Issue metadata

Status: WontFix
Owner:
Closed: Sep 27
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 3
Type: Bug

Blocked on:
issue 604568



Sign in to add a comment

[Android WebView] Remove retry logic on instrumentation tests

Project Member Reported by boliu@chromium.org, May 6 2016

Issue description

Gather some data first with scraping.. (since flakiness dashboard doesn't help due to  crbug.com/609588 )

This is the failed webview tests from the release tester (https://build.chromium.org/p/chromium.linux/builders/Android%20Tests) in the last 200 runs. There are some data quality issues, but whatever, good enough.

      3 org.chromium.android_webview.test.AndroidScrollIntegrationTest#testTouchScrollCanBeAlteredByUi
      3 org.chromium.android_webview.test.AwContentsRenderTest#testForceDrawWhenInvisible
     17 org.chromium.android_webview.test.AwQuotaManagerBridgeTest#testDeleteAllWithAppCache
     74 org.chromium.android_webview.test.AwQuotaManagerBridgeTest#testDeleteOriginWithAppCache
     15 org.chromium.android_webview.test.AwQuotaManagerBridgeTest#testGetResultsMatch
      1 org.chromium.android_webview.test.AwQuotaManagerBridgeTest#testGetResultsMatch.
      1 org.chromium.android_webview.test.AwSettingsTest#testAppCacheWithTwoViews.
      2 org.chromium.android_webview.test.WebKitHitTestTest#testUnknownTypeUnrecognizedNode
      3 org.chromium.android_webview.test.WebKitHitTestTest#testUnknownTypeUnrecognizedNode.

So... should nuke AwQuotaManagerBridgeTest from orbit, then webview tests are probably no worse than chrome_public_apk tests. I wrote those tests /o\

Checking debug bot next.
 

Comment 1 by boliu@chromium.org, May 6 2016

Same thing for debug bot:
      1 org.chromium.android_webview.test.AndroidScrollIntegrationTest#testTouchScrollCanBeAlteredByUi
      2 org.chromium.android_webview.test.AwContentsClientShouldOverrideUrlLoadingTest#testCallDestroyInCallback
      1 org.chromium.android_webview.test.AwContentsRenderTest#testForceDrawWhenInvisible
      1 org.chromium.android_webview.test.AwLegacyQuirksTest#testInitialScaleClobberQuirk
      1 org.chromium.android_webview.test.AwLegacyQuirksTest#testInitialScaleClobber
     24 org.chromium.android_webview.test.AwQuotaManagerBridgeTest#testDeleteAllWithAppCache
     45 org.chromium.android_webview.test.AwQuotaManagerBridgeTest#testDeleteOriginWithAppCache
      2 org.chromium.android_webview.test.AwQuotaManagerBridgeTest#testGetResultsMatch
      1 org.chromium.android_webview.test.WebKitHitTestTest#testUnknownTypeUnrecognizedNode.

Weirdly debug bot appears *less flaky* than the release bot.
is fixing tests not an option, and a better option? what am I missing.

Comment 3 by boliu@chromium.org, May 6 2016

Oh, missed like 40 builds from the debug bot because the url changed. Test name changed or something like that.

Comment 4 by boliu@chromium.org, May 6 2016

> is fixing tests not an option, and a better option? what am I missing.

Of course. But when was the last time you or anyone else on the webview team fixed a flaky test? If it's not affecting developers and getting someone yelling at you because you because a flaky test just blocked their CL from landing for the nth time, no one is going to care. And the problem just gets worse and worse.

Part of the problem is discoverbility. The past 200 webview test runs that I scraped are all *green*. Even if failed tests are reported to the flakiness dashboard, who's gonna look at it unless it affects your day-to-day work flow.

Also my opinion to disable flaky tests, instead of leaving them running and adding tools to workaround flakes. Not everyone agrees with that though.
disabling is fine. and making it more discoverable too. as long as you don't remove completely.

Comment 7 by roc...@chromium.org, May 25 2016

Found this bug while seeking a possible explanation for the recent increase in AwContentsRenderTest#testForceDrawWhenInvisible flake (https://build.chromium.org/p/chromium.linux/builders/Android%20Tests?numbuilds=200).

So I guess it's just been flaky for a while.
Blockedon: 604568
#7: you're looking for issue 604568
Status: WontFix (was: Assigned)

Sign in to add a comment