Issue 841098 link

Starred by 2 users

Issue metadata

Status:	Available
Owner:	----
Cc:	█ mariakho...@chromium.org █ ltian@chromium.org
Components:	Tests>Flaky
EstimatedDays:	----
NextAction:	----
OS:	Android
Pri:	1
Type:	Bug
Via-TryFlakes Test-Disabled

"org.chromium.chrome.browser.search_engines.TemplateUrlServiceTest#testSortandGetCustomSearchEngine" is flaky

Project Member Reported by chromium...@appspot.gserviceaccount.com, May 9 2018

Issue description

"org.chromium.chrome.browser.search_engines.TemplateUrlServiceTest#testSortandGetCustomSearchEngine" is flaky.

This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is infrastructure-related, please add Infra-Troopers label and change issue status to Untriaged. When done, please remove the issue from Sheriff Bug Queue by removing the Sheriff-Chromium label.

We have detected 4 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNybQsSBUZsYWtlImJvcmcuY2hyb21pdW0uY2hyb21lLmJyb3dzZXIuc2VhcmNoX2VuZ2luZXMuVGVtcGxhdGVVcmxTZXJ2aWNlVGVzdCN0ZXN0U29ydGFuZEdldEN1c3RvbVNlYXJjaEVuZ2luZQw.

Flaky tests should be disabled within 30 minutes unless culprit CL is found and reverted. Please see more details here: https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/sheriffing-bug-queues#triaging-auto-filed-flakiness-bugs

Comment 1 by vitaliii@chromium.org, May 9 2018

Sample runs:
https://ci.chromium.org/buildbot/tryserver.chromium.android/linux_android_rel_ng/546565
https://ci.chromium.org/buildbot/tryserver.chromium.android/linux_android_rel_ng/546532

log

====
org.junit.ComparisonFailure: expected:<keyword[2]> but was:<keyword[1]>
	at org.junit.Assert.assertEquals(Assert.java:115)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at org.chromium.chrome.browser.search_engines.TemplateUrlServiceTest.testSortandGetCustomSearchEngine(TemplateUrlServiceTest.java:215)
	at java.lang.reflect.Method.invokeNative(Method.java)
	at java.lang.reflect.Method.invoke(Method.java:515)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:52)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.chromium.base.test.ScreenshotOnFailureStatement.evaluate(ScreenshotOnFailureStatement.java:37)
	at org.chromium.chrome.browser.test.ChromeBrowserTestRule$1.evaluate(ChromeBrowserTestRule.java:40)
	at org.chromium.chrome.browser.test.ClearAppDataTestRule$1.evaluate(ClearAppDataTestRule.java:27)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.chromium.base.test.BaseJUnit4ClassRunner.runChild(BaseJUnit4ClassRunner.java:201)
	at org.chromium.base.test.BaseJUnit4ClassRunner.runChild(BaseJUnit4ClassRunner.java:43)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.chromium.base.test.BaseJUnit4ClassRunner.run(BaseJUnit4ClassRunner.java:183)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:59)
	at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:262)
	at org.chromium.base.test.BaseChromiumAndroidJUnitRunner.onStart(BaseChromiumAndroidJUnitRunner.java:120)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)
Searching for native crashes in: /b/swarming/w/itrS9abf/tmplpCLMm
Unknown Android release, consider passing --packed-lib.
Reading Android symbols from: /b/swarming/w/ir
Searching for Chrome symbols from within: /b/swarming/w/ir/out/Release/lib.unstripped:/b/swarming/w/ir/out/Release
org.junit.ComparisonFailure: expected:<keyword[5]> but was:<keyword[4]>
	at org.junit.Assert.assertEquals(Assert.java:115)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at org.chromium.chrome.browser.search_engines.TemplateUrlServiceTest.testSortandGetCustomSearchEngine(TemplateUrlServiceTest.java:233)
	at java.lang.reflect.Method.invokeNative(Method.java)
	at java.lang.reflect.Method.invoke(Method.java:515)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:52)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.chromium.base.test.ScreenshotOnFailureStatement.evaluate(ScreenshotOnFailureStatement.java:37)
	at org.chromium.chrome.browser.test.ChromeBrowserTestRule$1.evaluate(ChromeBrowserTestRule.java:40)
	at org.chromium.chrome.browser.test.ClearAppDataTestRule$1.evaluate(ClearAppDataTestRule.java:27)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.chromium.base.test.BaseJUnit4ClassRunner.runChild(BaseJUnit4ClassRunner.java:201)
	at org.chromium.base.test.BaseJUnit4ClassRunner.runChild(BaseJUnit4ClassRunner.java:43)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.chromium.base.test.BaseJUnit4ClassRunner.run(BaseJUnit4ClassRunner.java:183)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:59)
	at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:262)
	at org.chromium.base.test.BaseChromiumAndroidJUnitRunner.onStart(BaseChromiumAndroidJUnitRunner.java:120)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)
Searching for native crashes in: /b/swarming/w/itrS9abf/tmpTBlN8n
Unknown Android release, consider passing --packed-lib.
Reading Android symbols from: /b/swarming/w/ir
Searching for Chrome symbols from within: /b/swarming/w/ir/out/Release/lib.unstripped:/b/swarming/w/ir/out/Release
org.junit.ComparisonFailure: expected:<keyword[5]> but was:<keyword[4]>
	at org.junit.Assert.assertEquals(Assert.java:115)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at org.chromium.chrome.browser.search_engines.TemplateUrlServiceTest.testSortandGetCustomSearchEngine(TemplateUrlServiceTest.java:233)
	at java.lang.reflect.Method.invokeNative(Method.java)
	at java.lang.reflect.Method.invoke(Method.java:515)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:52)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.chromium.base.test.ScreenshotOnFailureStatement.evaluate(ScreenshotOnFailureStatement.java:37)
	at org.chromium.chrome.browser.test.ChromeBrowserTestRule$1.evaluate(ChromeBrowserTestRule.java:40)
	at org.chromium.chrome.browser.test.ClearAppDataTestRule$1.evaluate(ClearAppDataTestRule.java:27)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.chromium.base.test.BaseJUnit4ClassRunner.runChild(BaseJUnit4ClassRunner.java:201)
	at org.chromium.base.test.BaseJUnit4ClassRunner.runChild(BaseJUnit4ClassRunner.java:43)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.chromium.base.test.BaseJUnit4ClassRunner.run(BaseJUnit4ClassRunner.java:183)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:59)
	at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:262)
	at org.chromium.base.test.BaseChromiumAndroidJUnitRunner.onStart(BaseChromiumAndroidJUnitRunner.java:120)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)
Searching for native crashes in: /b/swarming/w/itrS9abf/tmpSXRlYV
Unknown Android release, consider passing --packed-lib.
Reading Android symbols from: /b/swarming/w/ir
Searching for Chrome symbols from within: /b/swarming/w/ir/out/Release/lib.unstripped:/b/swarming/w/ir/out/Release
====

Comment 2 by vitaliii@chromium.org, May 9 2018

The test was recently touched in https://chromium-review.googlesource.com/#/c/1023123/.

Comment 3 by vitaliii@chromium.org, May 9 2018

Cc: ltian@chromium.org

cc'ing the author of the CL.

Comment 4 by vitaliii@chromium.org, May 9 2018

Since I am not sure whether the CL is a culprit, disabling the test.

Comment 5 by vitaliii@chromium.org, May 9 2018

the CL disabled the test is https://chromium-review.googlesource.com/c/chromium/src/+/1051769.

Comment 6 by vitaliii@chromium.org, May 9 2018

Cc: -ltian@chromium.org mariakho...@chromium.org
Owner: ltian@chromium.org
Status: Assigned (was: Untriaged)

Hey ltian@, would you mind having a look or reassigning?

Also how can I find OWNERS of this test?
search_engines folder does not have OWNERS file as well as browser folder.

Project Member

Comment 7 by bugdroid1@chromium.org, May 9 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/47e3f813cf6b332d7995f1f009488b2abd5f8026

commit 47e3f813cf6b332d7995f1f009488b2abd5f8026
Author: vitaliii <vitaliii@chromium.org>
Date: Wed May 09 14:48:59 2018

Disabling flaky TemplateUrlServiceTest#testSortandGetCustomSearchEngine.

Disabling
org.chromium.chrome.browser.search_engines.TemplateUrlServiceTest
testSortandGetCustomSearchEngine, because flaky.

TBR=mariakhomenko@chromium.org

Bug: 841098
Change-Id: Ibae44f030880f59690d3f84114c1586d1b826d06
Reviewed-on: https://chromium-review.googlesource.com/1051769
Reviewed-by: vitaliii <vitaliii@chromium.org>
Commit-Queue: vitaliii <vitaliii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#557180}
[modify] https://crrev.com/47e3f813cf6b332d7995f1f009488b2abd5f8026/chrome/android/javatests/src/org/chromium/chrome/browser/search_engines/TemplateUrlServiceTest.java

Comment 8 by vitaliii@chromium.org, May 9 2018

Labels: -Sheriff-Chromium

Comment 9 by mariakho...@chromium.org, May 9 2018

Labels: Test-Disabled OS-Android

Project Member

Comment 10 by chromium...@appspot.gserviceaccount.com, May 10 2018

Labels: Sheriff-Chromium

Detected 11 new flakes for test/step "org.chromium.chrome.browser.search_engines.TemplateUrlServiceTest#testSortandGetCustomSearchEngine". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNybQsSBUZsYWtlImJvcmcuY2hyb21pdW0uY2hyb21lLmJyb3dzZXIuc2VhcmNoX2VuZ2luZXMuVGVtcGxhdGVVcmxTZXJ2aWNlVGVzdCN0ZXN0U29ydGFuZEdldEN1c3RvbVNlYXJjaEVuZ2luZQw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Comment 11 by rdevlin....@chromium.org, May 11 2018

Labels: -Sheriff-Chromium

assigned, -sheriff

Comment 12 by ltian@chromium.org, Jun 11 2018

Cc: ltian@chromium.org
Owner: ----
Status: Available (was: Assigned)

►

Issue 841098 link

Issue metadata

"org.chromium.chrome.browser.search_engines.TemplateUrlServiceTest#testSortandGetCustomSearchEngine" is flaky

Issue description

Comment 1 by vitaliii@chromium.org, May 9 2018

Comment 1 Deleted

Comment 2 by vitaliii@chromium.org, May 9 2018

Comment 2 Deleted

Comment 3 by vitaliii@chromium.org, May 9 2018

Comment 3 Deleted

Comment 4 by vitaliii@chromium.org, May 9 2018

Comment 4 Deleted

Comment 5 by vitaliii@chromium.org, May 9 2018

Comment 5 Deleted

Comment 6 by vitaliii@chromium.org, May 9 2018

Comment 6 Deleted

Comment 7 by bugdroid1@chromium.org, May 9 2018

Comment 7 Deleted

Comment 8 by vitaliii@chromium.org, May 9 2018

Comment 8 Deleted

Comment 9 by mariakho...@chromium.org, May 9 2018

Comment 9 Deleted

Comment 10 by chromium...@appspot.gserviceaccount.com, May 10 2018

Comment 10 Deleted

Comment 11 by rdevlin....@chromium.org, May 11 2018

Comment 11 Deleted

Comment 12 by ltian@chromium.org, Jun 11 2018

Comment 12 Deleted

Sign in to add a comment