Michael is OOO for a bit. Can this wait until he gets back, or should the team resolve this before then?

---

I see 3 categories of assertion failures:

 * "Expect: [2, 1, 2, 1, 3] Result: [2, 5, 1, 2, 1]" -- it looks like the test emits an extra "AUTOFILL_CANCEL" event. Not sure if this is a bug in the implementation or a bug in the test
 * "array lengths differed, expected.length=5 actual.length=6" -- this is caused by the same issue as above: the "AUTOFILL_CANCEL" is emitted in the beginning of the test.
   * we see 2 different failure conditions because the test is racy. The tests wait for the 5th callback [1], and after that drain the event queue. This pair of operations are not guaranteed to be atomic. Sadly, callback helper is easy to use racily.
 * waitForCallback times out -- I think this is only in testJavascriptNotTriggerSelectControlChangeNotification, and it waits for callback 0, which I would expect would always succeed.

[1] https://cs.chromium.org/chromium/src/android_webview/javatests/src/org/chromium/android_webview/test/AwAutofillTest.java?l=1885&rcl=81cadf27a8206ec3926cf170ff11840f6651e499

Thanks for taking a look; it can wait. I've marked the test as experimental in the meantime (re17a8e9db69d31a01b4f8d0c4e67d79269342cf3) so it'll keep running but won't turn the bot red.

FWIW, it was on fyi a few months ago and the suite was green at some point back then:
https://ci.chromium.org/buildbot/chromium.android.fyi/Oreo%20Phone%20Tester/2690

So maybe a recent-ish regression?

> Thanks for taking a look; it can wait.

Ack, I'll pass this to Michael then for when he gets back.

> So maybe a recent-ish regression?

The link you sent is roughly around 3ac4f597f3a298990edf25c7fa68ab96476d5ce5 (April), but the tests were added after that in 851aad4c8ea88fde4872d0919d5bcebf8488ab56 (May). I don't suppose you have any bot data for May (it looked like you sent the bot's last run). So, who knows what caused it (or if these tests ever would have passed on the bot).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9eee0111048f9c6946688530988c6f5666fd0e1e

commit 9eee0111048f9c6946688530988c6f5666fd0e1e
Author: Tao Bai <michaelbai@chromium.org>
Date: Tue Jul 31 17:39:13 2018

[Autofill] Fix test failure on Android o

Bug: 863089
Change-Id: I0e95f78799a2550aeb89b2b980b2d5cdb16142d2
Reviewed-on: https://chromium-review.googlesource.com/1156003
Reviewed-by: Changwan Ryu <changwan@chromium.org>
Commit-Queue: Tao Bai <michaelbai@chromium.org>
Cr-Commit-Position: refs/heads/master@{#579464}
[modify] https://crrev.com/9eee0111048f9c6946688530988c6f5666fd0e1e/android_webview/javatests/src/org/chromium/android_webview/test/AwAutofillTest.java

Autofill tests still failing: https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.ci/Oreo%20Phone%20Tester/786

failures:
org.chromium.android_webview.test.AwAutofillTest#testJavascriptNotTriggerSelectControlChangeNotification
org.chromium.android_webview.test.AwAutofillTest#testBasicAutofill__multiprocess_mode
org.chromium.android_webview.test.AwAutofillTest#testBasicAutofill
org.chromium.android_webview.test.AwAutofillTest#testJavascriptNotTriggerSelectControlChangeNotification__multiprocess_mode

C 1054.339s Main  [FAIL] org.chromium.android_webview.test.AwAutofillTest#testJavascriptNotTriggerSelectControlChangeNotification:
C 1054.339s Main  java.lang.AssertionError
C 1054.339s Main  	at org.chromium.base.test.util.CallbackHelper.waitForCallback(CallbackHelper.java:179)
C 1054.339s Main  	at org.chromium.base.test.util.CallbackHelper.waitForCallback(CallbackHelper.java:210)
C 1054.339s Main  	at org.chromium.android_webview.test.AwAutofillTest.waitForCallbackAndVerifyTypes(AwAutofillTest.java:1950)
C 1054.339s Main  	at org.chromium.android_webview.test.AwAutofillTest.testJavascriptNotTriggerSelectControlChangeNotification(AwAutofillTest.java:1478)
C 1054.339s Main  	at java.lang.reflect.Method.invoke(Native Method)
C 1054.339s Main  	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
C 1054.339s Main  	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
C 1054.339s Main  	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:52)
C 1054.339s Main  	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
C 1054.339s Main  	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
C 1054.340s Main  	at org.chromium.base.test.ScreenshotOnFailureStatement.evaluate(ScreenshotOnFailureStatement.java:37)
C 1054.340s Main  	at org.chromium.android_webview.test.AwActivityTestRule$1.evaluate(AwActivityTestRule.java:80)
C 1054.340s Main  	at android.support.test.internal.statement.UiThreadStatement.evaluate(UiThreadStatement.java:55)
C 1054.340s Main  	at android.support.test.rule.ActivityTestRule$ActivityStatement.evaluate(ActivityTestRule.java:270)
C 1054.340s Main  	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
C 1054.340s Main  	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
C 1054.340s Main  	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
C 1054.340s Main  	at org.chromium.base.test.BaseJUnit4ClassRunner.runChild(BaseJUnit4ClassRunner.java:237)
C 1054.340s Main  	at org.chromium.android_webview.test.AwJUnit4ClassRunner.runChild(AwJUnit4ClassRunner.java:97)
C 1054.340s Main  	at org.chromium.android_webview.test.AwJUnit4ClassRunner.runChild(AwJUnit4ClassRunner.java:34)
C 1054.340s Main  	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
C 1054.340s Main  	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
C 1054.340s Main  	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
C 1054.340s Main  	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
C 1054.340s Main  	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
C 1054.340s Main  	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
C 1054.341s Main  	at org.chromium.base.test.BaseJUnit4ClassRunner.run(BaseJUnit4ClassRunner.java:220)
C 1054.341s Main  	at org.junit.runners.Suite.runChild(Suite.java:128)
C 1054.341s Main  	at org.junit.runners.Suite.runChild(Suite.java:27)
C 1054.341s Main  	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
C 1054.341s Main  	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
C 1054.341s Main  	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
C 1054.341s Main  	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
C 1054.341s Main  	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
C 1054.341s Main  	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
C 1054.341s Main  	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
C 1054.341s Main  	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
C 1054.341s Main  	at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:59)
C 1054.341s Main  	at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:262)
C 1054.341s Main  	at org.chromium.base.test.BaseChromiumAndroidJUnitRunner.onStart(BaseChromiumAndroidJUnitRunner.java:125)
C 1054.341s Main  	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2074)

Re-opened.

This was failed because test expects no event occurs, but CallbackHelper.waitForCallback() requires at least one event should happen, it can only reproduced in debug build.