New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 659072 link

Starred by 1 user
Status: Fixed
Owner:
█ yolandyan@chromium.org
not available
Closed: Oct 2016
Cc:
█ mikec...@chromium.org
paulmiller@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: Bug

Blocking:
issue 617655



Sign in to add a comment

CTS URLUtilTest#testGuessFileName and WebChromeClientTest#testOnJsBeforeUnload broken after CTS update

Project Member Reported by torne@chromium.org, Oct 25 2016 
After the CTS zip was updated in https://codereview.chromium.org/2444073002 two tests have failed in every build:

android.webkit.cts.URLUtilTest#testGuessFileName
android.webkit.cts.WebChromeClientTest#testOnJsBeforeUnload

Most recent example: https://build.chromium.org/p/chromium.android/builders/Android%20WebView%20CTS%20L-MR1%20%28dbg%29/builds/17193

10-25 03:21:38 I/06ad8849003b6c51: android.webkit.cts.URLUtilTest#testGuessFileName FAIL 
junit.framework.ComparisonFailure: expected:<test.jp[e]g> but was:<test.jp[]g>
at junit.framework.Assert.assertEquals(Assert.java:85)
at junit.framework.Assert.assertEquals(Assert.java:91)
at android.webkit.cts.URLUtilTest.testGuessFileName(URLUtilTest.java:147)
at java.lang.reflect.Method.invoke(Native Method)
at java.lang.reflect.Method.invoke(Method.java:372)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:115)
at junit.framework.TestResult.runProtected(TestResult.java:133)
at android.support.test.internal.runner.junit3.DelegatingTestResult.runProtected(DelegatingTestResult.java:90)
at junit.framework.TestResult.run(TestResult.java:118)
at android.support.test.internal.runner.junit3.AndroidTestResult.run(AndroidTestResult.java:52)
at junit.framework.TestCase.run(TestCase.java:124)
at android.support.test.internal.runner.junit3.NonLeakyTestSuite$NonLeakyTest.run(NonLeakyTestSuite.java:63)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at android.support.test.internal.runner.junit3.DelegatingTestSuite.run(DelegatingTestSuite.java:103)
at android.support.test.internal.runner.junit3.AndroidTestSuite.run(AndroidTestSuite.java:52)
at android.support.test.internal.runner.junit3.JUnit38ClassRunner.run(JUnit38ClassRunner.java:90)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:24)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at org.junit.runner.JUnitCore.run(JUnitCore.java:136)
at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:245)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1853)



10-25 03:21:41 I/06ad8849003b6c51: android.webkit.cts.WebChromeClientTest#testOnJsBeforeUnload FAIL 
junit.framework.ComparisonFailure: expected:<[Changes you made may not be saved.]> but was:<[testOnJsBeforeUnload]>
at junit.framework.Assert.assertEquals(Assert.java:85)
at junit.framework.Assert.assertEquals(Assert.java:91)
at android.webkit.cts.WebChromeClientTest.testOnJsBeforeUnload(WebChromeClientTest.java:224)
at java.lang.reflect.Method.invoke(Native Method)
at java.lang.reflect.Method.invoke(Method.java:372)
at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:115)
at junit.framework.TestResult.runProtected(TestResult.java:133)
at android.support.test.internal.runner.junit3.DelegatingTestResult.runProtected(DelegatingTestResult.java:90)
at junit.framework.TestResult.run(TestResult.java:118)
at android.support.test.internal.runner.junit3.AndroidTestResult.run(AndroidTestResult.java:52)
at junit.framework.TestCase.run(TestCase.java:124)
at android.support.test.internal.runner.junit3.NonLeakyTestSuite$NonLeakyTest.run(NonLeakyTestSuite.java:63)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at android.support.test.internal.runner.junit3.DelegatingTestSuite.run(DelegatingTestSuite.java:103)
at android.support.test.internal.runner.junit3.AndroidTestSuite.run(AndroidTestSuite.java:52)
at android.support.test.internal.runner.junit3.JUnit38ClassRunner.run(JUnit38ClassRunner.java:90)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:24)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at org.junit.runner.JUnitCore.run(JUnitCore.java:136)
at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:245)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1853)
Project Member

Comment 1 by bugdroid1@chromium.org, Oct 25 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b0d46dd0952923c0726d351ebb92989c1a5d2b27

commit b0d46dd0952923c0726d351ebb92989c1a5d2b27
Author: torne <torne@chromium.org>
Date: Tue Oct 25 12:09:52 2016

Suppress failing CTS tests.

testFindNext is still flaking despite the changes in
https://codereview.chromium.org/2444073002 and two tests are now failing
consistently. Suppress the failures while we investigate.

BUG= 618089 , 659072 
TBR=yolandyan@chromium.org

Review-Url: https://codereview.chromium.org/2446123002
Cr-Commit-Position: refs/heads/master@{#427325}

[modify] https://crrev.com/b0d46dd0952923c0726d351ebb92989c1a5d2b27/android_webview/tools/cts_config/expected_failure_on_bot.json

Comment 2 by gsennton@chromium.org, Oct 25 2016

Cc: paulmiller@chromium.org
testGuessFileName was changed in go/ag/600674 to expect ".jpg" (a couple of years ago). Shouldn't that change be part of the new CTS zip? (I have no idea where that zip is from ;)).

Comment 3 by yolandyan@chromium.org, Oct 26 2016 

My bad, the new zip is from android-build aosp-lollipop-cts-release, while the right place is actually aosp-lollipop-cts-mr1-release or aosp-lollipop-cts-mr1-dev.

Changing to the right zip file CL: https://codereview.chromium.org/2450103003

I think it's also best to add a field showing where the zip file is fetched from
Recipe change: https://codereview.chromium.org/2452843002

Comment 4 by gsennton@chromium.org, Oct 26 2016 

Nice, adding the file origin makes it a lot clearer, thanks Yoland! :)
Project Member

Comment 5 by bugdroid1@chromium.org, Oct 27 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/f03d1942fdbc7dca854706021ed6aaaaedad67fc

commit f03d1942fdbc7dca854706021ed6aaaaedad67fc
Author: yolandyan <yolandyan@chromium.org>
Date: Thu Oct 27 17:27:17 2016

Change CTS filepath format

Having only CTS zip file's path with the build id in the name can be confusing.
Adding "_origin" as a comment shows where the cts zip was downloaded from.

CL for json change: https://codereview.chromium.org/2450103003

BUG= 659072 

Review-Url: https://codereview.chromium.org/2452843002

[modify] https://crrev.com/f03d1942fdbc7dca854706021ed6aaaaedad67fc/scripts/slave/recipe_modules/chromium_android/api.py
[modify] https://crrev.com/f03d1942fdbc7dca854706021ed6aaaaedad67fc/scripts/slave/recipe_modules/chromium_android/example.expected/webview_cts_basic.json
[modify] https://crrev.com/f03d1942fdbc7dca854706021ed6aaaaedad67fc/scripts/slave/recipe_modules/chromium_android/example.expected/webview_cts_expected_failure.json
[modify] https://crrev.com/f03d1942fdbc7dca854706021ed6aaaaedad67fc/scripts/slave/recipe_modules/chromium_android/example.expected/webview_cts_invalid_output.json
[modify] https://crrev.com/f03d1942fdbc7dca854706021ed6aaaaedad67fc/scripts/slave/recipe_modules/chromium_android/example.expected/webview_cts_invalid_platform.json
[modify] https://crrev.com/f03d1942fdbc7dca854706021ed6aaaaedad67fc/scripts/slave/recipe_modules/chromium_android/example.expected/webview_cts_unexecuted_tests.json
[modify] https://crrev.com/f03d1942fdbc7dca854706021ed6aaaaedad67fc/scripts/slave/recipe_modules/chromium_android/example.expected/webview_cts_unexpected_class_failure.json
[modify] https://crrev.com/f03d1942fdbc7dca854706021ed6aaaaedad67fc/scripts/slave/recipe_modules/chromium_android/example.expected/webview_cts_unexpected_method_failure.json
[modify] https://crrev.com/f03d1942fdbc7dca854706021ed6aaaaedad67fc/scripts/slave/recipes/chromium.expected/full_chromium_android_Android_WebView_CTS_L_MR1__dbg_.json

Comment 7 by yolandyan@chromium.org, Oct 28 2016

Status: Fixed (was: Assigned)
CTS bot is running fine now
►

Sign in to add a comment