INVALID_TEST_RESULTS - NoDevicesError |
||
Issue descriptionWe have a lot flaky builds due to this error. Examples: https://ci.chromium.org/p/chromium/builds/b8944422251327639040 https://ci.chromium.org/p/chromium/builds/b8944524835406627696 https://ci.chromium.org/p/chromium/builds/b8944438430397728112 There could be many more: https://bigquery.cloud.google.com/results/findit-for-me:US.bquijob_698b4e0a_163dbbfa296 please use suffix the build id to "https://ci.chromium.org/p/chromium/builds/b" to load the build page.
,
Jun 7 2018
jbudorick@: assign to you for deduplicate or further triage
,
Sep 21
Some examples: https://chromium-swarm.appspot.com/task?id=400e0e1880551710&refresh=10&show_raw=1 """ I 0.000s Main command: /b/swarming/w/ir/build/android/test_runner.py instrumentation --additional-apk /b/swarming/w/ir/out/Release/apks/ChromiumNetTestAwSupport.apk --additional-apk /b/swarming/w/ir/out/Release/apks/ChromiumNetTestSupport.apk --apk-under-test /b/swarming/w/ir/out/Release/apks/WebViewInstrumentation.apk --output-directory /b/swarming/w/ir/out/Release --runtime-deps-path /b/swarming/w/ir/out/Release/gen.runtime/android_webview/test/webview_instrumentation_test_apk__test_runner_script.runtime_deps --test-apk /b/swarming/w/ir/out/Release/apks/WebViewInstrumentationTest.apk --test-jar /b/swarming/w/ir/out/Release/test.lib.java/WebViewInstrumentationTest.jar -v --store-tombstones --gs-results-bucket=chromium-result-details --recover-devices --test-launcher-summary-output=/b/swarming/w/ioO_taGb/output.json I 0.006s Main condition '<lambda>' met I 0.008s anonymous condition '<lambda>' met I 0.010s Main condition '<lambda>' met I 0.010s Main [host]> /b/swarming/w/ir/third_party/android_tools/sdk/build-tools/27.0.3/aapt dump xmltree /b/swarming/w/ir/out/Release/apks/WebViewInstrumentationTest.apk AndroidManifest.xml I 0.013s Main [host]> /b/swarming/w/ir/third_party/android_tools/sdk/build-tools/27.0.3/aapt dump xmltree /b/swarming/w/ir/out/Release/apks/WebViewInstrumentation.apk AndroidManifest.xml I 0.017s TimeoutThread-1-for-MainThread [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb devices W 3.020s Main No devices found. Will try again after restarting adb server. I 3.021s TimeoutThread-1-for-MainThread [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb kill-server I 3.023s TimeoutThread-1-for-MainThread [host]> pgrep adb I 3.025s TimeoutThread-1-for-MainThread PIDs for adb found: [] I 3.025s TimeoutThread-1-for-MainThread condition 'adb_killed' met (0.0s) I 3.025s TimeoutThread-1-for-MainThread [host]> taskset -c 0 /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb start-server I 6.030s TimeoutThread-1-for-MainThread [host]> pgrep adb I 6.032s TimeoutThread-1-for-MainThread PIDs for adb found: [11977] I 6.033s TimeoutThread-1-for-MainThread condition 'adb_started' met (3.0s) I 6.039s TimeoutThread-1-for-MainThread [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb devices I 6.042s Main Generated json results file at /b/swarming/w/iteeWAJM/tmp5HAhQB I 6.042s Main Finishing archiving output. E 6.043s Main Error occurred. Traceback (most recent call last): File "/b/swarming/w/ir/build/android/test_runner.py", line 1012, in main return RunTestsCommand(args) File "/b/swarming/w/ir/build/android/test_runner.py", line 715, in RunTestsCommand return RunTestsInPlatformMode(args) File "/b/swarming/w/ir/build/android/test_runner.py", line 839, in RunTestsInPlatformMode with json_writer(), logcats_uploader, env, test_instance, test_run: File "/b/swarming/w/ir/build/android/pylib/base/test_run.py", line 39, in __enter__ self.SetUp() File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_instrumentation_test_run.py", line 291, in SetUp self._env.parallel_devices.pMap( File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_environment.py", line 189, in parallel_devices return parallelizer.SyncParallelizer(self.devices) File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_environment.py", line 180, in devices self._InitDevices() File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_environment.py", line 137, in _InitDevices default_retries=self._max_tries - 1, device_arg=device_arg) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 2917, in HealthyDevices return _get_devices() File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 2904, in _get_devices raise device_errors.NoDevicesError() NoDevicesError: No devices attached. [W2018-09-20T04:29:02.751682Z 11922 0 butler.go:240] Butler Context was cancelled. Initiating shutdown. {"error":"context canceled"} INFO:devil.utils.timeout_retry:condition 'logdog_stopped' not met [E2018-09-20T04:29:02.751834Z 11922 0 main.go:205] Butler terminated with error. {"error":"context canceled"} [E2018-09-20T04:29:02.751856Z 11922 0 main.go:233] Failed to serve. original error: context canceled """ https://chromium-swarm.appspot.com/task?id=40109a465f41a110&refresh=10&show_raw=1 """ I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:22:36.577 3897 12351 I GoogleURLConnFactory: Using platform SSLCertificateSocketFactory I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:22:36.642 3897 12351 W ConfigFetchTask: exception on config fetch: java.net.UnknownHostException: Unable to resolve host "cloudconfig.googleapis.com": No address associated with hostname I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:22:36.647 3897 3897 I ConfigFetchService: fetch service done; releasing wakelock I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:22:36.649 3897 3897 I ConfigFetchService: stopping self I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:22:41.231 5468 5499 D Finsky : [257] AppStatesReplicator.handleContentSyncResponse: Completed 0 account content syncs with 0 successful. I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:22:41.232 5468 5468 D Finsky : [1] ContentSyncService$5.onFinished: Installation state replication succeeded. I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:22:41.301 541 541 E QCALOG : [MessageQ] ProcessNewMessage: [LOWI-SERVER] unknown deliver target [OS-Agent] I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:22:41.658 3593 3593 I ConfigService: onDestroy I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:23:26.575 3426 4282 I ClearcutLoggerApiImpl: disconnect managed GoogleApiClient I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:23:41.302 541 541 E QCALOG : [MessageQ] ProcessNewMessage: [LOWI-SERVER] unknown deliver target [OS-Agent] I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:23:54.970 12922 12922 D AndroidRuntime: >>>>>> START com.android.internal.os.RuntimeInit uid 0 <<<<<< I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:23:54.976 12922 12922 D AndroidRuntime: CheckJNI is OFF I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:23:55.025 12922 12922 D ICU : No timezone override file found: /data/misc/zoneinfo/current/icu/icu_tzdata.dat I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:23:55.075 12922 12922 I Radio-JNI: register_android_hardware_Radio DONE I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:23:55.093 12922 12922 D AndroidRuntime: Calling main entry org.chromium.android.commands.unzip.Unzip I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:24:31.288 12922 12928 I art : Background sticky concurrent mark sweep GC freed 1431(881KB) AllocSpace objects, 0(0B) LOS objects, 87% free, 143KB/1129KB, paused 34.397ms total 38.454ms I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:24:39.937 12922 12928 I art : Background sticky concurrent mark sweep GC freed 1888(873KB) AllocSpace objects, 0(0B) LOS objects, 87% free, 145KB/1128KB, paused 12.158ms total 33.914ms I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:24:41.302 541 541 E QCALOG : [MessageQ] ProcessNewMessage: [LOWI-SERVER] unknown deliver target [OS-Agent] I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:25:41.303 541 541 E QCALOG : [MessageQ] ProcessNewMessage: [LOWI-SERVER] unknown deliver target [OS-Agent] I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:26:41.304 541 541 E QCALOG : [MessageQ] ProcessNewMessage: [LOWI-SERVER] unknown deliver target [OS-Agent] I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:27:17.712 12922 12922 D AndroidRuntime: Shutting down VM I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:27:17.731 12922 12928 I art : Starting a blocking GC HeapTrim I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:27:41.304 541 541 E QCALOG : [MessageQ] ProcessNewMessage: [LOWI-SERVER] unknown deliver target [OS-Agent] I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:28:26.061 13404 13404 E chromium: [0920/162826.060842:ERROR:md5sum.cc(28)] Could not open file /data/local/tmp/forwarder/device_forwarder I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:28:27.358 11817 11817 E chromium: [0920/162827.358325:ERROR:socket.cc(357)] send: Broken pipe (32) I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:28:27.359 11817 11817 E chromium: [0920/162827.359147:ERROR:socket.cc(357)] send: Bad file descriptor (9) I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:28:27.360 11817 11818 I chromium: [0920/162827.359989:INFO:device_controller.cc(68)] Received exit notification I 414.451s individual_device_set_up(01e36dc77ba7cd17) 09-20 16:28:27.366 11817 14217 I chromium: [0920/162827.366805:INFO:device_listener.cc(89)] Received exit notification, stopped accepting clients. I 414.452s TimeoutThread-1-for-individual_device_set_up(01e36dc77ba7cd17) [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 01e36dc77ba7cd17 shell '( ps );echo %$?' I 414.583s individual_device_set_up(01e36dc77ba7cd17) Currently running device_forwarders: E 414.584s individual_device_set_up(01e36dc77ba7cd17) Shard failed: individual_device_set_up(01e36dc77ba7cd17) Traceback (most recent call last): File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_environment.py", line 66, in wrapper return f(dev, *args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function return func(*args, **kwargs) File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 349, in individual_device_set_up step() File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 340, in <lambda> return lambda: crash_handler.RetryOnSystemCrash(step, dev) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/crash_handler.py", line 32, in RetryOnSystemCrash return f(device) File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 337, in init_tool_and_start_servers s.SetUp() File "/b/swarming/w/ir/build/android/pylib/local/local_test_server_spawner.py", line 89, in SetUp [(self.port, self.port)], self._device, self._tool) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/forwarder.py", line 178, in Map formatted_output)) HostForwarderError: `/b/swarming/w/ir/out/Release/host_forwarder --adb=/b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb --serial-id=01e36dc77ba7cd17 --map 10202 10202` exited with 1: [0920/162827.132217:ERROR:host_forwarder_main.cc(150)] ERROR: could not get adb port for device. You might need to add 'adb' to your PATH or provide the device serial id. E 414.584s individual_device_set_up(01e36dc77ba7cd17) Device 01e36dc77ba7cd17 blacklisted: individual_device_set_up I 414.605s Main Generated json results file at /b/swarming/w/itFOlgy9/tmpaQwWfM I 414.605s Main Finishing archiving output. E 414.606s Main Error occurred. Traceback (most recent call last): File "/b/swarming/w/ir/build/android/test_runner.py", line 1012, in main return RunTestsCommand(args) File "/b/swarming/w/ir/build/android/test_runner.py", line 715, in RunTestsCommand return RunTestsInPlatformMode(args) File "/b/swarming/w/ir/build/android/test_runner.py", line 839, in RunTestsInPlatformMode with json_writer(), logcats_uploader, env, test_instance, test_run: File "/b/swarming/w/ir/build/android/pylib/base/test_run.py", line 39, in __enter__ self.SetUp() File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 353, in SetUp self._test_instance.GetDataDependencies()) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/parallelizer.py", line 263, in pMap r.pFinish(None) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/parallelizer.py", line 135, in pFinish self._objs.JoinAll() File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 189, in JoinAll self._JoinAll(watcher, timeout) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 161, in _JoinAll thread.ReraiseIfException() File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 84, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_environment.py", line 77, in wrapper on_failure(dev, f.__name__) File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_environment.py", line 268, in BlacklistDevice 'All devices were blacklisted due to errors') NoDevicesError: All devices were blacklisted due to errors [W2018-09-20T16:28:27.964642Z 24450 0 butler.go:240] Butler Context was cancelled. Initiating shutdown. {"error":"context canceled"} INFO:devil.utils.timeout_retry:condition 'logdog_stopped' not met [E2018-09-20T16:28:27.964800Z 24450 0 main.go:205] Butler terminated with error. {"error":"context canceled"} [E2018-09-20T16:28:27.964826Z 24450 0 main.go:233] Failed to serve. original error: context canceled """ |
||
►
Sign in to add a comment |
||
Comment 1 by st...@chromium.org
, Jun 7 2018