org.chromium.android_webview.test.AwSettingsTest#testLoadWithOverviewModeWithTwoViews and so on cannot run on local devices |
|||
Issue description
org.chromium.android_webview.test.AwSettingsTest#testLoadWithOverviewModeWithTwoViews
org.chromium.android_webview.test.AwSettingsTest#testLoadWithOverviewModeViewportTagWithTwoViews
org.chromium.android_webview.test.AwSettingsTest#testLoadWithOverviewModeWithTwoViews with {--webview-sandboxed-renderer}
org.chromium.android_webview.test.AwSettingsTest#testLoadWithOverviewModeViewportTagWithTwoViews with {--webview-sandboxed-renderer}
These four tests can not run on Android M and N. I didn't check L, but run on KK as trybots run.
% ninja android_webview_test_apk
% ${out}/bin/run_android_webview_test_apk -vv -f org.chromium.android_webview.test.AwSettingsTest#testLoadWithOverviewModeWithTwoViews
Error log is here:
C 60.375s Main ********************************************************************************
C 60.375s Main Detailed Logs
C 60.375s Main ********************************************************************************
C 60.375s Main [FAIL] org.chromium.android_webview.test.AwSettingsTest#testLoadWithOverviewModeWithTwoViews:
C 60.375s Main junit.framework.AssertionFailedError: expected:<1.0> but was:<0.25>
C 60.375s Main at org.chromium.android_webview.test.AwSettingsTest$AwSettingsLoadWithOverviewModeTestHelper.doEnsureSettingHasValue(AwSettingsTest.java:1348)
C 60.375s Main at org.chromium.android_webview.test.AwSettingsTest$AwSettingsLoadWithOverviewModeTestHelper.doEnsureSettingHasValue(AwSettingsTest.java:1298)
C 60.376s Main at org.chromium.android_webview.test.AwSettingsTest$AwSettingsTestHelper.ensureSettingHasValue(AwSettingsTest.java:144)
C 60.376s Main at org.chromium.android_webview.test.AwSettingsTest$AwSettingsTestHelper.ensureSettingHasInitialValue(AwSettingsTest.java:88)
C 60.376s Main at org.chromium.android_webview.test.AwSettingsTest.runPerViewSettingsTest(AwSettingsTest.java:2988)
C 60.376s Main at org.chromium.android_webview.test.AwSettingsTest.testLoadWithOverviewModeWithTwoViews(AwSettingsTest.java:2660)
C 60.376s Main at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
C 60.376s Main at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
C 60.376s Main at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
C 60.376s Main at org.chromium.base.test.BaseTestResult.runParameterized(BaseTestResult.java:161)
C 60.376s Main at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:124)
C 60.376s Main at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
C 60.376s Main at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
C 60.376s Main at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555)
C 60.376s Main at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1879)
C 60.377s Main__
C 60.377s Main [FAIL] org.chromium.android_webview.test.AwSettingsTest#testLoadWithOverviewModeWithTwoViews with {--webview-sandboxed-renderer}:
C 60.377s Main junit.framework.AssertionFailedError: expected:<1.0> but was:<0.25>
C 60.377s Main at org.chromium.android_webview.test.AwSettingsTest$AwSettingsLoadWithOverviewModeTestHelper.doEnsureSettingHasValue(AwSettingsTest.java:1348)
C 60.377s Main at org.chromium.android_webview.test.AwSettingsTest$AwSettingsLoadWithOverviewModeTestHelper.doEnsureSettingHasValue(AwSettingsTest.java:1298)
C 60.377s Main at org.chromium.android_webview.test.AwSettingsTest$AwSettingsTestHelper.ensureSettingHasValue(AwSettingsTest.java:144)
C 60.377s Main at org.chromium.android_webview.test.AwSettingsTest$AwSettingsTestHelper.ensureSettingHasInitialValue(AwSettingsTest.java:88)
C 60.377s Main at org.chromium.android_webview.test.AwSettingsTest.runPerViewSettingsTest(AwSettingsTest.java:2992)
C 60.377s Main at org.chromium.android_webview.test.AwSettingsTest.testLoadWithOverviewModeWithTwoViews(AwSettingsTest.java:2660)
C 60.377s Main at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
C 60.377s Main at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
C 60.377s Main at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
C 60.378s Main at org.chromium.base.test.BaseTestResult.runParameterized(BaseTestResult.java:161)
C 60.378s Main at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:124)
C 60.378s Main at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
C 60.378s Main at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
C 60.378s Main at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555)
C 60.378s Main at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1879)
C 60.378s Main ********************************************************************************
C 60.378s Main Summary
C 60.378s Main ********************************************************************************
C 60.379s Main [==========] 2 tests ran.
C 60.379s Main [ PASSED ] 0 tests.
C 60.379s Main [ FAILED ] 2 tests, listed below:
C 60.379s Main [ FAILED ] org.chromium.android_webview.test.AwSettingsTest#testLoadWithOverviewModeWithTwoViews
C 60.379s Main [ FAILED ] org.chromium.android_webview.test.AwSettingsTest#testLoadWithOverviewModeWithTwoViews with {--webview-sandboxed-renderer}
C 60.379s Main__
C 60.379s Main 2 FAILED TESTS
C 60.379s Main ********************************************************************************
If I remove assertions, it get to be flaky and sometime raise TimeoutException and sometime pass.
,
Aug 31 2016
I did some investigation. Here is a note from the investigation. mOnScaleChangedCallCount was different from getOnScaleChangedHelper().getCallCount() value at the beginning of the AwSettingsLoadWithOverviewModeTestHelper::doEnsureSettingHasValue() method , and following waitForCallback() call does not wait at all.
,
Aug 31 2016
I don't think these tests run on KK only, or that they are flaky: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=android_webview_test_apk&tests=AwSettingsTest%23testLoadWithOverviewModeWithTwoViews Are you specifically thinking it's failing/flaky *with* your CL applied?
,
Sep 1 2016
No, it always fails without any modification on trunk.
,
Sep 1 2016
I run Android tests on a daily use phone. That means other applications are running while I'm running tests. Do you think it could be a reason for failures? I tried Nexus 5X + Android N, and Pixel C + Android M.
,
Sep 1 2016
,
Sep 1 2016
Here is a short summary what I explained at the mail thread. Trybot failed with my patch. -> Tried to reproduce it on local machines, but failed with another error. -> Ran trunk without any modification, but it still failed. -> Ran the same binary on emulated KK env, and it passed.
,
Sep 1 2016
test devices do get configured with additional setting changes, like turn off lock screen and keeping the screen always on. Also I'm not entirely sure if root is required, but I generally have root on my devices I use since I need root for other work. there might be other config changes This bot does *not* run on a nexus devices, so there is no root, and there are more failures, but not these particular tests.. https://build.chromium.org/p/chromium.android/builders/Lollipop%20Consumer%20Tester/builds/2211
,
Sep 5 2016
Hum... one of my tried device was rooted, and emulated KK wasn't (I do not have much confidence about this, but I just used provided official image without any modification) Also, these were only tests that failed always (100%). Others in webview tests passed.
,
Sep 5 2016
Emulator images are all userdebug and have root.
,
Sep 6 2016
Oh, thank you for correction. Do these tests run correctly on you guys' devices? If they pass, I'd investigate a little more to figure out what makes these tests fail in my environment.
,
Sep 7 2016
Runs fine for me. On a nexus 4 with lollipop, and I'm at refs/heads/master@{#416938}
,
Sep 9 2016
Finally, I found a configuration on which these tests pass.
refs/heads/master@{#417594}
Nexus 5 with M, rooted, display-on => pass
Nexus 5 with M, rooted, display-off => fail
Pixel C with N, non-rooted, display-on => fail
Pixel C with N, non-rooted, display-off => fail
,
Sep 9 2016
I think majority of tests requires display being on. It's the one thing I do to all my test devices, Developer Options->Stay Awake. Not sure about pixel C, if that's a root problem, or if the test is actually broken on that device..
,
Sep 9 2016
Tried one more device. Nexus 5X with N, non-rooted, display-on => pass Nexus 5X with N, non-rooted, display-off => fail So, display seems to be the reason of my failures. Pixel C is a kind of special device. So, I'm ok to close this bug as WontFix. Also I just notice that it's explained at https://chromium.googlesource.com/chromium/src/+/master/docs/android_test_instructions.md#Screen > You MUST ensure that the screen stays on while testing: adb shell svc power stayon usb Or do this manually on the device: Settings -> Developer options -> Stay Awake. I should read this carefully first. Sorry for bothering you.
,
Sep 9 2016
|
|||
►
Sign in to add a comment |
|||
Comment 1 by toyoshim@chromium.org
, Aug 31 2016