Cronet test runner failing with non-actionable errors |
|||
Issue description
Cronet test runner seems to fail like so recently:
I 140.087s TimeoutThread-1-for-individual_device_tear_down(00dd45e961a6cbd5) [host]> /b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/android_tools/sdk/platform-tools/adb -s 00dd45e961a6cbd5 shell '( am clear-debug-app );echo %$?'
I 140.087s TimeoutThread-1-for-individual_device_tear_down(01e4cb3dcc2f38c2) [host]> /b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/android_tools/sdk/platform-tools/adb -s 01e4cb3dcc2f38c2 shell '( am clear-debug-app );echo %$?'
I 140.888s TimeoutThread-1-for-individual_device_tear_down(01e4cb3dcc2f38c2) [host]> /b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/android_tools/sdk/platform-tools/adb -s 01e4cb3dcc2f38c2 shell '( rm -f /data/local/tmp/chrome_timeout_scale );echo %$?'
I 140.919s TimeoutThread-1-for-individual_device_tear_down(00dd45e961a6cbd5) [host]> /b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/android_tools/sdk/platform-tools/adb -s 00dd45e961a6cbd5 shell '( rm -f /data/local/tmp/chrome_timeout_scale );echo %$?'
I 141.061s tear_down_device(00dd45e961a6cbd5) Wrote device cache: /b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/out/Release/device_cache_00dd45e961a6cbd5.json
I 141.062s tear_down_device(01e4cb3dcc2f38c2) Wrote device cache: /b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/out/Release/device_cache_01e4cb3dcc2f38c2.json
E 141.064s Main Unrecognized error occurred.
Traceback (most recent call last):
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/test_runner.py", line 987, in main
return RunTestsCommand(args)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/test_runner.py", line 731, in RunTestsCommand
return RunTestsInPlatformMode(args)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/test_runner.py", line 851, in RunTestsInPlatformMode
raw_results = test_run.RunTests()
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/local/device/local_device_test_run.py", line 124, in RunTests
run_tests_on_device, tc, try_results).pGet(None)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/catapult/devil/devil/utils/parallelizer.py", line 263, in pMap
r.pFinish(None)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/catapult/devil/devil/utils/parallelizer.py", line 135, in pFinish
self._objs.JoinAll()
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll
self._JoinAll(watcher, timeout)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 158, in _JoinAll
thread.ReraiseIfException()
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run
self._ret = self._func(*self._args, **self._kwargs)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/local/device/local_device_environment.py", line 59, in wrapper
return f(dev, *args, **kwargs)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/local/device/local_device_test_run.py", line 71, in run_tests_on_device
device=dev)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/catapult/devil/devil/android/crash_handler.py", line 32, in RetryOnSystemCrash
return f(device)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/local/device/local_device_test_run.py", line 70, in <lambda>
lambda d, t=test: self._RunTest(d, t),
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/local/device/local_device_instrumentation_test_run.py", line 472, in _RunTest
device.product_cpu_abi, self._test_instance.symbolizer)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/instrumentation/instrumentation_test_instance.py", line 922, in GenerateTestResults
start_ms, duration_ms, device_abi, symbolizer)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/instrumentation/instrumentation_test_instance.py", line 148, in GenerateTestResults
elif current_result.GetType() == base_test_result.ResultType.UNKNOWN:
AttributeError: 'NoneType' object has no attribute 'GetType'
step returned non-zero exit code: 1
Some example bot runs that failed like this:
https://build.chromium.org/p/chromium.android/builders/Android%20Cronet%20Marshmallow%2064bit%20Builder/builds/13137
https://build.chromium.org/p/chromium.android/builders/Android%20Cronet%20Marshmallow%2064bit%20Builder/builds/13117
https://build.chromium.org/p/chromium.android/builders/Android%20Cronet%20KitKat%20Builder/builds/3168
https://build.chromium.org/p/chromium.android/builders/Android%20Cronet%20Marshmallow%2064bit%20Builder/builds/13145
,
Sep 7 2017
I think the stack at the bottom is not the actual error. Looks like this test fails on the bots: I 79.889s run_tests_on_device(06596466005cab1a) detected failure in org.chromium.net.urlconnection.CronetFixedModeOutputStreamTest#testWriteAfterRequestFailed. raw output: I 79.889s run_tests_on_device(06596466005cab1a) INSTRUMENTATION_STATUS: test=testWriteAfterRequestFailed I 79.889s run_tests_on_device(06596466005cab1a) INSTRUMENTATION_STATUS: class=org.chromium.net.urlconnection.CronetFixedModeOutputStreamTest I 79.889s run_tests_on_device(06596466005cab1a) INSTRUMENTATION_STATUS: current=1 I 79.889s run_tests_on_device(06596466005cab1a) INSTRUMENTATION_STATUS_CODE: 1 I 79.889s run_tests_on_device(06596466005cab1a) INSTRUMENTATION_STATUS: numtests=1 I 79.889s run_tests_on_device(06596466005cab1a) INSTRUMENTATION_STATUS: stream=. I 79.889s run_tests_on_device(06596466005cab1a) INSTRUMENTATION_STATUS: id=AndroidJUnitRunner I 79.889s run_tests_on_device(06596466005cab1a) INSTRUMENTATION_STATUS: class=org.chromium.net.urlconnection.CronetFixedModeOutputStreamTest I 79.889s run_tests_on_device(06596466005cab1a) INSTRUMENTATION_STATUS: current=1 I 79.889s run_tests_on_device(06596466005cab1a) INSTRUMENTATION_RESULT: stream= I 79.889s run_tests_on_device(06596466005cab1a) I 79.889s run_tests_on_device(06596466005cab1a) Time: 0.273 I 79.889s run_tests_on_device(06596466005cab1a) I 79.890s run_tests_on_device(06596466005cab1a) OK (1 test) I 79.890s run_tests_on_device(06596466005cab1a) I 79.890s run_tests_on_device(06596466005cab1a) I 79.890s run_tests_on_device(06596466005cab1a) I 79.890s run_tests_on_device(06596466005cab1a) I 79.890s run_tests_on_device(06596466005cab1a) I 79.890s run_tests_on_device(06596466005cab1a) I 79.890s run_tests_on_device(06596466005cab1a) And then you get (what I assume must be close to an infinite) number of blank lines.
,
Sep 7 2017
Note: The bots are all mostly green, so it appears as though this test is maybe flaky, and the new proguard deobfuscation is causing trouble when reporting the failure.
,
Sep 7 2017
Ah - different bot shows:
Traceback (most recent call last):
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/test_runner.py", line 987, in main
return RunTestsCommand(args)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/test_runner.py", line 731, in RunTestsCommand
return RunTestsInPlatformMode(args)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/test_runner.py", line 851, in RunTestsInPlatformMode
raw_results = test_run.RunTests()
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/local/device/local_device_test_run.py", line 124, in RunTests
run_tests_on_device, tc, try_results).pGet(None)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/catapult/devil/devil/utils/parallelizer.py", line 263, in pMap
r.pFinish(None)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/catapult/devil/devil/utils/parallelizer.py", line 135, in pFinish
self._objs.JoinAll()
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll
self._JoinAll(watcher, timeout)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 158, in _JoinAll
thread.ReraiseIfException()
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run
self._ret = self._func(*self._args, **self._kwargs)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/local/device/local_device_environment.py", line 59, in wrapper
return f(dev, *args, **kwargs)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/local/device/local_device_test_run.py", line 71, in run_tests_on_device
device=dev)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/third_party/catapult/devil/devil/android/crash_handler.py", line 32, in RetryOnSystemCrash
return f(device)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/local/device/local_device_test_run.py", line 70, in <lambda>
lambda d, t=test: self._RunTest(d, t),
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/local/device/local_device_instrumentation_test_run.py", line 465, in _RunTest
output = self._test_instance.MaybeDeobfuscateLines(output)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/instrumentation/instrumentation_test_instance.py", line 848, in MaybeDeobfuscateLines
return self._deobfuscator.TransformLines(lines)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/symbols/deobfuscator.py", line 129, in TransformLines
return target_instance.TransformLines(lines)
File "/b/build/slave/Android_Cronet_Marshmallow_64bit_Builder/build/src/build/android/pylib/symbols/deobfuscator.py", line 88, in TransformLines
if self._reader_thread.is_alive():
AttributeError: 'NoneType' object has no attribute 'is_alive'
,
Sep 7 2017
,
Sep 7 2017
Probable fix here: https://chromium-review.googlesource.com/c/chromium/src/+/655121
,
Sep 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/33f43bb9c944ad7549cdf8d77f0f38f45ae39f9a commit 33f43bb9c944ad7549cdf8d77f0f38f45ae39f9a Author: Andrew Grieve <agrieve@chromium.org> Date: Fri Sep 08 01:50:36 2017 Android: Improve robustness of Deobfuscator during TearDown Was causing infinite output in one case, and crash in another. Bug: 762912 Change-Id: I8a8eb306d274f34684b06ca3df722800413fb776 Reviewed-on: https://chromium-review.googlesource.com/655121 Commit-Queue: Andrew Grieve <agrieve@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#500480} [modify] https://crrev.com/33f43bb9c944ad7549cdf8d77f0f38f45ae39f9a/build/android/pylib/symbols/deobfuscator.py
,
Sep 8 2017
Bots have been green all day. Marking as fixed. |
|||
►
Sign in to add a comment |
|||
Comment 1 by jbudorick@chromium.org
, Sep 7 2017Components: -Infra Test>Android