This also affected https://luci-milo.appspot.com/buildbot/chromeos/veyron_speedy-paladin/3815 (a bit disguised, but if you slick into the HWTest stdio you will see it)

+ihf

Your first link shows the test passing. But maybe autotest is confused? It shows the test twice once passing, once timing out.

on veyron_speedy-paladin/3815, the x86 version of the test was skipped, but the arm version did time out.

  cheets_CTS.6.0_r12.x86.android.core.tests.libcore.package.harmony_java_math   [  INFO  ]
  cheets_CTS.6.0_r12.x86.android.core.tests.libcore.package.harmony_java_math     TEST_NA: Skipping:  test not supported on this board.

  cheets_CTS.6.0_r12.arm.android.core.tests.libcore.package.harmony_java_math   [ FAILED ]
  cheets_CTS.6.0_r12.arm.android.core.tests.libcore.package.harmony_java_math     ABORT: Timed out, did not run.

Hmm. You are right. So why did the test get run a second time?

Well, Intel machines run x86 and arm ABI. ARM machines only arm ABI. Hence this is ok on ARM:
cheets_CTS.6.0_r12.x86.android.core.tests.libcore.package.harmony_java_math     TEST_NA: Skipping:  test not supported on this board.


I took another look at cyan. This is rather confusing:
  cheets_CTS.6.0_r12.arm.android.core.tests.libcore.package.harmony_java_math   [ FAILED ]
  cheets_CTS.6.0_r12.arm.android.core.tests.libcore.package.harmony_java_math     ABORT: Timed out, did not run.
  cheets_CTS.android.core.tests.libcore.package.harmony_java_math               [ PASSED ]

It should run 2 tests, one for arm and one for x86

arm
NAME = 'cheets_CTS.6.0_r12.arm.android.core.tests.libcore.package.harmony_java_math'
ATTRIBUTES = 'suite:arc-cts, suite:arc-cts-stable, suite:smoke, suite:bvt-cq, suite:arc-bvt-cq'

x86
NAME = 'cheets_CTS.6.0_r12.x86.android.core.tests.libcore.package.harmony_java_math'
ATTRIBUTES = 'suite:arc-cts, suite:arc-cts-stable, suite:smoke, suite:bvt-cq, suite:arc-bvt-cq'

No idea where cheets_CTS.android.core.tests.libcore.package.harmony_java_math came from.

Notice both get tagged tag='android.core.tests.libcore.package.harmony_java_math',
but I don't think tagging should matter here, but maybe we see an extra confusing bug?

To return to the arm result. The arm package gets scheduled


12/09 12:02:50.240 DEBUG|             suite:0825| Scheduling cheets_CTS.6.0_r12.arm.android.core.tests.libcore.package.harmony_java_math
12/09 12:02:50.648 DEBUG|             suite:1088| Adding job keyval for cheets_CTS.6.0_r12.arm.android.core.tests.libcore.package.harmony_java_math=89746616-chromeos-test

has logs

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/89746616-chromeos-test/chromeos4-row4-rack10-host16/debug/


tests start running
12/09 12:03:22.120 DEBUG|     dynamic_suite:0554| Waiting on suite.
12/09 12:03:48.338 DEBUG|          monitors:0191| ts_mon: sending 8 metrics to PubSub
12/09 12:21:29.561 INFO |        server_job:0153| START	89746619-chromeos-test/chromeos4-row4-rack11-host4/audio_Microphone	audio_Microphone	timestamp=1481314801	localtime=Dec 09 12:20:01	
12/09 12:21:29.562 INFO |        server_job:0153| 	GOOD	89746619-chromeos-test/chromeos4-row4-rack11-host4/audio_Microphone	audio_Microphone	timestamp=1481314829	localtime=Dec 09 12:20:29	completed successfully
12/09 12:21:29.563 INFO |        server_job:0153| END GOOD	89746619-chromeos-test/chromeos4-row4-rack11-host4/audio_Microphone	audio_Microphone	timestamp=1481314829	localtime=Dec 09 12:20:29	

cheets_CTS... starts running

12/09 12:39:05.356 DEBUG|        base_utils:0280| [stdout] 12-09 12:38:54 I/chromeos4-row4-rack10-host16:22: Test package armeabi-v7a android.core.tests.libcore.package.harmony_java_math started

log file gets truncated
12/09 12:39:33.928 DEBUG|        base_utils:0280| [stdout] 12-09 12:39:33 I/chromeos4-row4-rack10-host16:22: org.apache.harmony.tests.java.math.BigIntegerMultiplyTest#testPowNegativeNumToOddExp PASS 
12/09 12:39:33.929 DEBUG|        base_utils:0280| [stdout] 12-09 12:39:33 I/chromeos4-row4-rack10-host16:22: org.apache.harmony.tests.java.math.BigIntegerMultiplyTest#testPowNegativeNumToZeroExp PASS 
[EOF]

Is reported failing due to sql?

12/09 12:40:02.357 ERROR|                db:0025| 12:40:02 12/09/16: An operational error occurred during a database operation: (2006, 'MySQL server has gone away'); retrying, don't panic yet
12/09 12:40:24.858 INFO |        server_job:0153| START	----	veyron_speedy-paladin/R57-9066.0.0-rc1/bvt-cq/cheets_CTS.6.0_r12.arm.android.core.tests.libcore.package.harmony_java_math	timestamp=1481316024	localtime=Dec 09 12:40:24	
12/09 12:40:24.859 INFO |        server_job:0153| 	ABORT	----	veyron_speedy-paladin/R57-9066.0.0-rc1/bvt-cq/cheets_CTS.6.0_r12.arm.android.core.tests.libcore.package.harmony_java_math	timestamp=1481316024	localtime=Dec 09 12:40:24	
12/09 12:40:24.859 INFO |        server_job:0153| END ABORT	----	veyron_speedy-paladin/R57-9066.0.0-rc1/bvt-cq/cheets_CTS.6.0_r12.arm.android.core.tests.libcore.package.harmony_java_math	timestamp=1481316024	localtime=Dec 09 12:40:24	

Notice the time from tests starting running to cheets_CTS dying is about 20 minutes. Maybe we are hitting the timeout?

control.arc-bvt-cq:args_dict['max_runtime_mins'] = 20

It might be easiest to increase this to 60 minutes and see if the timeouts go away.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/autotest-cheets/+/323b880455b138c9e267ca32766c46b16a7f1446

commit 323b880455b138c9e267ca32766c46b16a7f1446
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Wed Dec 14 08:21:16 2016