android_webview tests are flaky |
|||||||||||||||
Issue description"org.chromium.android_webview.test.SafeBrowsingTest#testSafeBrowsingOnSafeBrowsingHitProceed__multiprocess_mode" is flaky. This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is infrastructure-related, please add Infra-Troopers label and change issue status to Untriaged. When done, please remove the issue from Sheriff Bug Queue by removing the Sheriff-Chromium label. We have detected 3 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyeQsSBUZsYWtlIm5vcmcuY2hyb21pdW0uYW5kcm9pZF93ZWJ2aWV3LnRlc3QuU2FmZUJyb3dzaW5nVGVzdCN0ZXN0U2FmZUJyb3dzaW5nT25TYWZlQnJvd3NpbmdIaXRQcm9jZWVkX19tdWx0aXByb2Nlc3NfbW9kZQw. Flaky tests should be disabled within 30 minutes unless culprit CL is found and reverted. Please see more details here: https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/sheriffing-bug-queues#triaging-auto-filed-flakiness-bugs
,
Mar 14 2018
Trying to figure out culprit CL.
,
Mar 14 2018
Doesn't look like https://chromium-review.googlesource.com/c/chromium/src/+/959744 could have caused this. Not certain.
,
Mar 14 2018
,
Mar 14 2018
Wondering if related issue 821603 , issue 821590 , issue 821589 , issue 821529 , issue 821460
,
Mar 14 2018
yeah, there was a flurry of flakes for webview tests with no discernible pattern, so I dunno if that was some sort of infra problem, some issue in lower layers, or something else. it's been ~1 day since these were filed. maybe should check if they still are still happening
,
Mar 14 2018
last flake was 4 hours ago, so seems like it is still happening. https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyeQsSBUZsYWtlIm5vcmcuY2hyb21pdW0uYW5kcm9pZF93ZWJ2aWV3LnRlc3QuU2FmZUJyb3dzaW5nVGVzdCN0ZXN0U2FmZUJyb3dzaW5nT25TYWZlQnJvd3NpbmdIaXRQcm9jZWVkX19tdWx0aXByb2Nlc3NfbW9kZQw
,
Mar 14 2018
ah, actually it was 16 hours ago.
,
Mar 14 2018
Issue 821457 has been merged into this issue. Issue 821458 has been merged into this issue. Issue 821459 has been merged into this issue. Issue 821460 has been merged into this issue. Issue 821529 has been merged into this issue. Issue 821589 has been merged into this issue. Issue 821590 has been merged into this issue. Issue 821603 has been merged into this issue.
,
Mar 14 2018
,
Mar 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/cf978e37558334aa1451b38442b500b0a73e470e commit cf978e37558334aa1451b38442b500b0a73e470e Author: Caleb Rouleau <crouleau@chromium.org> Date: Wed Mar 14 19:02:01 2018 Disable flaky testSafeBrowsingOnSafeBrowsingHitProceed TBR=ntfschr Bug: 821697 Change-Id: I61515e735a542cf94dcb00c50263ea512d66fcea Reviewed-on: https://chromium-review.googlesource.com/963081 Reviewed-by: Caleb Rouleau <crouleau@chromium.org> Commit-Queue: Caleb Rouleau <crouleau@chromium.org> Cr-Commit-Position: refs/heads/master@{#543146} [modify] https://crrev.com/cf978e37558334aa1451b38442b500b0a73e470e/android_webview/javatests/src/org/chromium/android_webview/test/SafeBrowsingTest.java
,
Mar 14 2018
glancing through, it looks like single shards are failing a bunch of tests, and it looks like single devices will exhibit failures repeatedly (e.g. https://chromium-swarm.appspot.com/bot?id=build360-m1--device7)
,
Mar 14 2018
brief survey of the result_details screenshots indicates that the screen is either entirely black or off.
,
Mar 14 2018
Yeah pretty much all the time we get clusters of test failures like this it's the screen being off or a crash dialog being open or similar :|
,
Mar 14 2018
Just got a more recent flake https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyeQsSBUZsYWtlIm5vcmcuY2hyb21pdW0uYW5kcm9pZF93ZWJ2aWV3LnRlc3QuU2FmZUJyb3dzaW5nVGVzdCN0ZXN0U2FmZUJyb3dzaW5nT25TYWZlQnJvd3NpbmdIaXRQcm9jZWVkX19tdWx0aXByb2Nlc3NfbW9kZQw
,
Mar 14 2018
I won't be able to get to this for a few days at the earliest; releasing it.
,
Mar 14 2018
Any benefit to keeping the Safe Browsing test from comment #11 disabled? It shouldn't be especially flaky, and it depends on the screen like all other AW Safe Browsing (and most other WebView) tests. If so, when is it safe to reenable the test?
,
Mar 14 2018
Beside the test disabled on comment #11, there are many other tests failing too: webview_instrumentation_test_apk (with patch) on Android webview_instrumentation_test_apk (with patch) on Android Run on OS: 'Android' Max shard duration: 0:17:16.997980 (shard #2) failures: org.chromium.android_webview.test.WebKitHitTestTest#testImgeType__multiprocess_mode org.chromium.android_webview.test.WebKitHitTestTest#testSrcImgeAnchorTypeRelativeUrl__multiprocess_mode org.chromium.android_webview.test.WebViewAsynchronousFindApisTest#testFindAllFinds__multiprocess_mode org.chromium.android_webview.test.WebViewAsynchronousFindApisTest#testClearFindNext__multiprocess_mode org.chromium.android_webview.test.AwContentsClientFullScreenTest#testPowerSaveBlockerIsEnabledDuringEmbeddedPlayback__multiprocess_mode org.chromium.android_webview.test.AndroidViewIntegrationTest#testViewIsNotBlankInWrapContentsMode__multiprocess_mode org.chromium.android_webview.test.WebViewAsynchronousFindApisTest#testClearMatches__multiprocess_mode org.chromium.android_webview.test.SafeBrowsingTest#testSafeBrowsingOnSafeBrowsingHitProceed__multiprocess_mode org.chromium.android_webview.test.SafeBrowsingTest#testSafeBrowsingProceedQuietInterstitial__multiprocess_mode org.chromium.android_webview.test.AwSettingsTest#testLoadWithOverviewModeWithTwoViews org.chromium.android_webview.test.VisualStateTest#testVisualStateCallbackWaitsForContentsToBeOnScreen org.chromium.android_webview.test.AwContentsRenderTest#testForceDrawWhenInvisible__multiprocess_mode org.chromium.android_webview.test.AwContentsClientShouldOverrideUrlLoadingTest#testCalledOn302Redirect org.chromium.android_webview.test.WebViewAsynchronousFindApisTest#testFindAllDouble__multiprocess_mode org.chromium.android_webview.test.AndroidScrollIntegrationTest#testJsScrollCanBeAlteredByUi org.chromium.android_webview.test.SafeBrowsingTest#testSafeBrowsingWhitelistedUnsafePagesDontShowInterstitial__multiprocess_mode org.chromium.android_webview.test.AwContentsClientShouldOverrideUrlLoadingTest#testCalledOnJavaScriptLocationDelayedReplaceRedirect org.chromium.android_webview.test.AndroidScrollIntegrationTest#testFlingScroll__multiprocess_mode Here is the stack trace for one of them: I 372.624s run_tests_on_device(01eac13018c2a86d) detected failure in org.chromium.android_webview.test.WebKitHitTestTest#testImgeType__multiprocess_mode. raw output: I 372.624s run_tests_on_device(01eac13018c2a86d) INSTRUMENTATION_STATUS: numtests=1 I 372.624s run_tests_on_device(01eac13018c2a86d) INSTRUMENTATION_STATUS: stream= I 372.624s run_tests_on_device(01eac13018c2a86d) org.chromium.android_webview.test.WebKitHitTestTest: I 372.624s run_tests_on_device(01eac13018c2a86d) INSTRUMENTATION_STATUS: id=AndroidJUnitRunner I 372.624s run_tests_on_device(01eac13018c2a86d) INSTRUMENTATION_STATUS: test=testImgeType__multiprocess_mode I 372.624s run_tests_on_device(01eac13018c2a86d) INSTRUMENTATION_STATUS: class=org.chromium.android_webview.test.WebKitHitTestTest I 372.624s run_tests_on_device(01eac13018c2a86d) INSTRUMENTATION_STATUS: current=1 I 372.625s run_tests_on_device(01eac13018c2a86d) INSTRUMENTATION_STATUS_CODE: 1 I 372.625s run_tests_on_device(01eac13018c2a86d) INSTRUMENTATION_STATUS: numtests=1 I 372.625s run_tests_on_device(01eac13018c2a86d) INSTRUMENTATION_STATUS: stream= I 372.625s run_tests_on_device(01eac13018c2a86d) Error in testImgeType__multiprocess_mode(org.chromium.android_webview.test.WebKitHitTestTest): I 372.625s run_tests_on_device(01eac13018c2a86d) java.lang.AssertionError: Criteria not met in allotted time. I 372.625s run_tests_on_device(01eac13018c2a86d) at org.junit.Assert.fail(Assert.java:88) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.junit.Assert.assertTrue(Assert.java:41) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.chromium.content.browser.test.util.CriteriaHelper.pollInstrumentationThread(CriteriaHelper.java:93) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.chromium.android_webview.test.AwActivityTestRule.pollInstrumentationThread(AwActivityTestRule.java:441) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.chromium.android_webview.test.AwActivityTestRule.pollUiThread(AwActivityTestRule.java:459) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.chromium.android_webview.test.WebKitHitTestTest.pollForHitTestDataOnUiThread(WebKitHitTestTest.java:110) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.chromium.android_webview.test.WebKitHitTestTest.testImgeType(WebKitHitTestTest.java:348) I 372.625s run_tests_on_device(01eac13018c2a86d) at java.lang.reflect.Method.invoke(Native Method) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:52) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.chromium.base.test.ScreenshotOnFailureStatement.evaluate(ScreenshotOnFailureStatement.java:37) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.chromium.android_webview.test.AwActivityTestRule$1.evaluate(AwActivityTestRule.java:80) I 372.625s run_tests_on_device(01eac13018c2a86d) at android.support.test.internal.statement.UiThreadStatement.evaluate(UiThreadStatement.java:55) I 372.625s run_tests_on_device(01eac13018c2a86d) at android.support.test.rule.ActivityTestRule$ActivityStatement.evaluate(ActivityTestRule.java:270) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.junit.rules.RunRules.evaluate(RunRules.java:20) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) I 372.625s run_tests_on_device(01eac13018c2a86d) at org.chromium.base.test.BaseJUnit4ClassRunner.runChild(BaseJUnit4ClassRunner.java:201) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.chromium.android_webview.test.AwJUnit4ClassRunner.runChild(AwJUnit4ClassRunner.java:64) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.chromium.android_webview.test.AwJUnit4ClassRunner.runChild(AwJUnit4ClassRunner.java:31) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.chromium.base.test.BaseJUnit4ClassRunner.run(BaseJUnit4ClassRunner.java:183) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.Suite.runChild(Suite.java:128) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.Suite.runChild(Suite.java:27) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) I 372.626s run_tests_on_device(01eac13018c2a86d) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) I 372.626s run_tests_on_device(01eac13018c2a86d) at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:59) I 372.626s run_tests_on_device(01
,
Mar 14 2018
,
Mar 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b9d2beb8ee7990fc796457698965995935406f56 commit b9d2beb8ee7990fc796457698965995935406f56 Author: Caleb Rouleau <crouleau@chromium.org> Date: Wed Mar 14 20:18:47 2018 Revert "Disable flaky testSafeBrowsingOnSafeBrowsingHitProceed" This reverts commit cf978e37558334aa1451b38442b500b0a73e470e. Reason for revert: not more flaky than the other ones. https://bugs.chromium.org/p/chromium/issues/detail?id=821697#c17 Original change's description: > Disable flaky testSafeBrowsingOnSafeBrowsingHitProceed > > TBR=ntfschr > > Bug: 821697 > Change-Id: I61515e735a542cf94dcb00c50263ea512d66fcea > Reviewed-on: https://chromium-review.googlesource.com/963081 > Reviewed-by: Caleb Rouleau <crouleau@chromium.org> > Commit-Queue: Caleb Rouleau <crouleau@chromium.org> > Cr-Commit-Position: refs/heads/master@{#543146} TBR=crouleau@chromium.org,ntfschr@chromium.org Change-Id: I141561d5448a77c0dc12a91032850c9dd2dc2e1c No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 821697 Reviewed-on: https://chromium-review.googlesource.com/963230 Reviewed-by: Caleb Rouleau <crouleau@chromium.org> Commit-Queue: Caleb Rouleau <crouleau@chromium.org> Cr-Commit-Position: refs/heads/master@{#543180} [modify] https://crrev.com/b9d2beb8ee7990fc796457698965995935406f56/android_webview/javatests/src/org/chromium/android_webview/test/SafeBrowsingTest.java
,
Mar 14 2018
,
Mar 14 2018
so what's the next step here? someone physically wake up all the devices attached to our bots..?
,
Mar 15 2018
,
Mar 15 2018
no; the next step is investigating why the screen is off. our provisioning logic should prevent this; we need to see why it isn't. pulling from labs for now.
,
Mar 15 2018
It seems the flakiness dashboard for this test is pretty green today. Has anything been done to the bots?
,
Mar 15 2018
#25: not to my knowledge.
,
Mar 15 2018
#25: I reflashed one of the problem bots yesterday, and it looks healthier now: https://chromium-swarm.appspot.com/bot?id=build321-m1--device5 I'll reflash any other I see. And ftr, the bot logs were complaining about the "assisted_gps_enabled" setting on the devices getting reset after each and every test. So something was screwing around with the gps... Probably another symptom of whatever was wrong with the phones.
,
Mar 15 2018
Awesome, thanks Ben.
,
Mar 15 2018
|
|||||||||||||||
►
Sign in to add a comment |
|||||||||||||||
Comment 1 by crouleau@chromium.org
, Mar 14 2018