AdbCommandFailedError Error on chromium.perf Android Nexus5 Perf (1) |
|||
Issue descriptionLink to buildbot status page: https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Nexus5%20Perf%20%281%29/builds/4423 smoothness.sil_key_cases is failing with a HostForwarder Error: Link to log page https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Nexus5%20Perf%20%281%29/builds/4423/steps/smoothness.key_silk_cases.reference/logs/stdio Looks like the device id might be 03848d7af0e9447d Stack trace: John do you have any idea why part of the error message is printed out vertical? Traceback (most recent call last): File "/b/rr/tmpKmNEJq/w/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 86, in _RunStoryAndProcessErrorIfNeeded state.RunStory(results) File "/b/rr/tmpKmNEJq/w/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function return func(*args, **kwargs) File "/b/rr/tmpKmNEJq/w/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 309, in RunStory self._PreparePage() File "/b/rr/tmpKmNEJq/w/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function return func(*args, **kwargs) File "/b/rr/tmpKmNEJq/w/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 283, in _PreparePage set([self._current_page.serving_dir])) File "/b/rr/tmpKmNEJq/w/src/third_party/catapult/telemetry/telemetry/core/platform.py", line 407, in SetHTTPServerDirectories self.StartLocalServer(server) File "/b/rr/tmpKmNEJq/w/src/third_party/catapult/telemetry/telemetry/core/platform.py", line 374, in StartLocalServer self._local_server_controller.StartServer(server) File "/b/rr/tmpKmNEJq/w/src/third_party/catapult/telemetry/telemetry/core/local_server.py", line 164, in StartServer server.Start(self) File "/b/rr/tmpKmNEJq/w/src/third_party/catapult/telemetry/telemetry/core/local_server.py", line 93, in Start local_server_controller.GetRemotePort(http_port))) File "/b/rr/tmpKmNEJq/w/src/third_party/catapult/telemetry/telemetry/core/local_server.py", line 184, in CreateForwarder return self._platform_backend.forwarder_factory.Create(port_pair) File "/b/rr/tmpKmNEJq/w/src/third_party/catapult/telemetry/telemetry/internal/forwarders/android_forwarder.py", line 25, in Create return AndroidForwarder(self._device, port_pair) File "/b/rr/tmpKmNEJq/w/src/third_party/catapult/telemetry/telemetry/internal/forwarders/android_forwarder.py", line 60, in __init__ [(port_pair.remote_port, port_pair.local_port)], self._device) File "/b/rr/tmpKmNEJq/w/src/third_party/catapult/devil/devil/android/forwarder.py", line 162, in Map '\n'.join(output))) HostForwarderError: `/b/rr/tmpKmNEJq/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/forwarder_host --adb=/b/rr/tmpKmNEJq/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb --serial-id=03848d7af0e9447d --map 49793 49793` exited with 1: [ 1 0 1 3 / 0 1 5 2 4 2 : E R R O R : h o s t _ f o r w a r d e r _ m a i n . c c ( 4 7 7 ) ] E R R O R : C o n n e c t i o n t o d e v i c e f a i l e d . E R R O R : E x i s t i n g c o n t r o l l e r s :
,
Oct 13 2016
This is happening on several other android bots as well, in particular Android Nexus6 Perf (2) and (3) For instance same exception here: https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Nexus6%20Perf%20%283%29/builds/3745/steps/loading.mobile/logs/stdio I am wondering if this is an infra issue or if it is related to: crbug.com/655653
,
Oct 13 2016
Also seeing issue on Android One Perf (1) https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20One%20Perf%20%281%29/builds/4199 See 1) system_health.memory_mobile - host forwarder 2) blink_perf.parser - host forwarder 3) storage.indexeddb_endure - host forwarder 4) smoothness.scrolling_tough_ad_cases.reference - host forwarder 5) v8.infinite_scroll_tbmv2.reference -- host forwarder
,
Oct 18 2016
re the error: it's trying to forward a port that's already being used: INFO:devil.android.forwarder: E/chromium( 3082): [1013/085242:ERROR:socket.cc(178)] bind/listen: Address already in use INFO:devil.android.forwarder: E/chromium( 3082): [1013/085242:ERROR:device_listener.cc(30)] Device could not bind and listen to local port 49793 INFO:devil.android.forwarder: I/chromium( 3082): [1013/085242:INFO:device_controller.cc(68)] Received exit notification INFO:devil.android.forwarder: I/chromium( 3082): [1013/085242:INFO:device_listener.cc(89)] Received exit notification, stopped accepting clients. 49793 is 0xC281: INFO:devil.utils.cmd_helper:[host]> /b/rr/tmpKmNEJq/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 03848d7af0e9447d pull /proc/net/tcp /tmp/tmpVnz51R/tmp_ReadFileWithPull WARNING:root: sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode WARNING:root: 0: 0100007F:13AD 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 13219 1 00000000 100 0 0 10 -1 WARNING:root: 1: 0100007F:D939 0100007F:C280 06 00000000:00000000 03:00000A03 00000000 0 0 0 3 00000000 WARNING:root: 2: 0100007F:CF06 0100007F:DA94 06 00000000:00000000 03:000002CB 00000000 0 0 0 3 00000000 WARNING:root: 3: 0100007F:C281 0100007F:D939 06 00000000:00000000 03:00001200 00000000 0 0 0 3 00000000 re vertical: I assume we're '\n'.join()-ing a string where we expected a list of lines. Not sure where.
,
Oct 18 2016
also, I'm somewhat suspicious of the port cleanup process given the number of ports being forwarded at failure time: INFO:devil.utils.cmd_helper:[host]> /b/rr/tmpKmNEJq/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 03848d7af0e9447d forward --list WARNING:root: 0acc13b443e4b104 tcp:48276 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:39360 localabstract:chrome_device_forwarder WARNING:root: 0acbf65743e4b10a tcp:33084 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:42067 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:43478 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:47623 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:50880 localabstract:chrome_device_forwarder WARNING:root: 0d890b5f25995524 tcp:50125 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:40215 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:54649 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:37665 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:45273 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:45921 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:51116 localabstract:chrome_device_forwarder WARNING:root: 0acbf65743e4b10a tcp:60322 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:56694 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:36425 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:46872 localabstract:chrome_device_forwarder WARNING:root: 0d890b5f25995524 tcp:39154 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:51572 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:35987 localabstract:chrome_device_forwarder WARNING:root: 0acbf65743e4b10a tcp:53440 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:50471 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:49600 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:45791 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:56416 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:52288 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:44563 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:60876 localabstract:chrome_device_forwarder WARNING:root: 0d890b5f25995524 tcp:44115 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:41178 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:59002 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:44176 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:33080 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:40102 localabstract:chrome_device_forwarder WARNING:root: 0acbf65743e4b10a tcp:47192 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:56377 localabstract:chrome_device_forwarder WARNING:root: 0acbf65743e4b10a tcp:38945 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:50821 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:34074 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:53846 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:47724 localabstract:chrome_device_forwarder WARNING:root: 0d890b5f25995524 tcp:38750 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:56023 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:41438 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:38653 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:50036 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:57797 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:54064 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:55010 localabstract:chrome_device_forwarder WARNING:root: 0acbf65743e4b10a tcp:50360 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:41532 localabstract:chrome_device_forwarder WARNING:root: 0acbf65743e4b10a tcp:51984 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:34936 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:51120 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:47755 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:33876 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:34579 localabstract:chrome_device_forwarder WARNING:root: 0d890b5f25995524 tcp:58484 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:51063 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:35324 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:33890 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:37794 localabstract:chrome_device_forwarder WARNING:root: 0acbf65743e4b10a tcp:55935 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:50793 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:46861 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:39860 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:36557 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:58402 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:34802 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:60474 localabstract:chrome_devtools_remote WARNING:root: 0accace943e4af1c tcp:57951 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:56800 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:56350 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:59288 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:37152 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:49247 localabstract:chrome_device_forwarder WARNING:root: 0d890b5f25995524 tcp:56522 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:32841 localabstract:chrome_device_forwarder WARNING:root: 0d890b5f25995524 tcp:46387 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:35838 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:58294 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:56203 localabstract:chrome_device_forwarder WARNING:root: 0d890b5f25995524 tcp:54455 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:38847 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:40306 localabstract:chrome_device_forwarder WARNING:root: 0d890b5f25995524 tcp:36456 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:39301 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:57282 localabstract:chrome_devtools_remote WARNING:root: 0d890b5f25995524 tcp:58680 localabstract:chrome_devtools_remote WARNING:root: 0accace943e4af1c tcp:40399 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:48309 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:41559 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:40131 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:40652 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:44764 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:59695 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:58182 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:35377 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:38331 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:41030 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:56462 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:34116 localabstract:chrome_devtools_remote WARNING:root: 0accace943e4af1c tcp:34113 localabstract:chrome_device_forwarder WARNING:root: 0d890b5f25995524 tcp:51540 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:36428 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:36805 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:38678 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:43678 localabstract:chrome_device_forwarder WARNING:root: 0d890b5f25995524 tcp:50861 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:33758 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:58454 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:57342 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:49113 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:41140 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:57256 localabstract:chrome_device_forwarder WARNING:root: 0d890b5f25995524 tcp:46745 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:60535 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:53930 localabstract:chrome_device_forwarder WARNING:root: 0d88e86a43e4b0cc tcp:46522 localabstract:chrome_device_forwarder WARNING:root: 0accace943e4af1c tcp:60019 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:33134 localabstract:chrome_device_forwarder WARNING:root: 0acc13b443e4b104 tcp:35234 localabstract:chrome_device_forwarder WARNING:root: 0a9c636f43e4b7c9 tcp:53985 localabstract:chrome_device_forwarder WARNING:root: 0d890b5f25995524 tcp:38672 localabstract:chrome_device_forwarder WARNING:root: 03848d7af0e9447d tcp:53673 localabstract:chrome_device_forwarder WARNING:root: 0d88e8
,
Oct 20 2016
Peter, did you see anything with the device? If not, I'll take it and look more at the forwarder issues.
,
Oct 20 2016
Well, outwardly it looks fine. Let me re-flash them all first?
,
Oct 21 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c228201f3389161201cc0992b82ebbd1c7604e8c commit c228201f3389161201cc0992b82ebbd1c7604e8c Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Fri Oct 21 02:27:45 2016 Roll src/third_party/catapult/ 96f25d9aa..147f2cfdb (13 commits). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/96f25d9aa22b..147f2cfdbb17 $ git log 96f25d9aa..147f2cfdb --date=short --no-merges --format='%ad %ae %s' 2016-10-20 alexandermont Turn on eslint rule no-new-object and fix existing errors. 2016-10-20 alexandermont Turn on eslint rule yield-star-spacing and fix existing errors. 2016-10-20 alexandermont Turn on eslint rule key-spacing and fix existing errors. 2016-10-20 alexandermont Turn on eslint rule quote-props and fix existing errors. 2016-10-20 alexandermont Turn on eslint rule spaced-comment and fix existing errors. 2016-10-20 alexandermont Turn on eslint rule no-throw-literal and fix existing errors. 2016-10-20 alexandermont Turn on eslint rule array-bracket-spacing and fix existing errors. 2016-10-20 alexandermont Turn on eslint rule semi-spacing and fix existing errors. 2016-10-20 alexandermont Turn on eslint rule no-multiple-empty-lines and fix existing errors. 2016-10-20 alexandermont Turn on eslint rule one-var and fix existing errors. 2016-10-20 sullivan Add charliea and eakuefner as per-file eslint owners 2016-10-20 rnephew [Devil] Fix forwarder error message formatting. 2016-10-20 fmeawad Reland of [V8][Metric] Initial implementation of the RuntimeStatsMetric (patchset #1 id:1 of https://chromiumcodereview.appspot.com/2439953002/ ) BUG= 655625 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://chromiumcodereview.appspot.com/2438213002 Cr-Commit-Position: refs/heads/master@{#426698} [modify] https://crrev.com/c228201f3389161201cc0992b82ebbd1c7604e8c/DEPS
,
Oct 22 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/99557feac28b32cd281951050b9b7d2397534e31 commit 99557feac28b32cd281951050b9b7d2397534e31 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Sat Oct 22 01:09:34 2016 Roll src/third_party/catapult/ fe41618f8..b7d8fe883 (8 commits). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/fe41618f8d39..b7d8fe883c16 $ git log fe41618f8..b7d8fe883 --date=short --no-merges --format='%ad %ae %s' 2016-10-21 bpastene devil: Add some logging to device_monitor. 2016-10-21 jbudorick Revert of [devil] Roll to ADB 1.0.36. (patchset #1 id:1 of https://codereview.chromium.org/2436373002/ ) 2016-10-21 aiolos Add solidSelectedDots option to chart_series config. 2016-10-21 benjhayden Set text-overflow:ellipsis on Name column cells in histogram-set-table. 2016-10-21 benjhayden Fix and test metrics side panel. 2016-10-21 rnephew [Android][Telemetry] Show output of adb.ForwardList when error detected. 2016-10-21 rnephew [Devil] Fix more forwarder logging. 2016-10-21 jbudorick [devil] Roll to ADB 1.0.36. BUG=649481, 623989 , 655653 , 655625 , 623989 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://chromiumcodereview.appspot.com/2445473002 Cr-Commit-Position: refs/heads/master@{#426940} [modify] https://crrev.com/99557feac28b32cd281951050b9b7d2397534e31/DEPS
,
Nov 15 2016
Now having AdbCommandFailedError.
,
Nov 15 2016
#10: can you include a link to the build/step to which you're referring?
,
Jan 5 2017
Marking this as archived as we have for sure since have fiddled with these devices in other bugs since. |
|||
►
Sign in to add a comment |
|||
Comment 1 by pschm...@google.com
, Oct 13 2016Status: Assigned (was: Untriaged)