New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 635884 link

Starred by 2 users
Status: WontFix
Owner:
█ rnep...@chromium.org
Last visit > 30 days ago
Closed: Feb 2017
Cc:
jbudorick@chromium.org
█ benhenry@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

single-device host_forwarder failures on N5X perf

Project Member Reported by nedngu...@google.com, Aug 9 2016 
All the benchmarks in https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20%283%29/builds/1487 that got assigned to device 3 are failing because of fowarders error:

(INFO) 2016-08-09 03:10:03,494 forwarder._KillDeviceLocked:335  Killing device_forwarder.
(INFO) 2016-08-09 03:10:03,495 cmd_helper._ValidateAndLogCommand:153  [host]> /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2ea7a27b7aa05 shell '( test -e /data/local/tmp/forwarder/device_forwarder );echo %$?'
(INFO) 2016-08-09 03:10:03,603 cmd_helper._ValidateAndLogCommand:153  [host]> /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2ea7a27b7aa05 shell '( LD_LIBRARY_PATH=/data/local/tmp/forwarder/  /data/local/tmp/forwarder/device_forwarder --kill-server );echo %$?'
(INFO) 2016-08-09 03:10:03,814 cmd_helper._ValidateAndLogCommand:153  [host]> /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2ea7a27b7aa05 shell '( a=/data/local/tmp/md5sum/md5sum_bin;! [[ $(ls -l $a) = *534360* ]]&&exit 2;export LD_LIBRARY_PATH=/data/local/tmp/md5sum;$a /data/local/tmp/forwarder/device_forwarder;: );echo %$?'
(INFO) 2016-08-09 03:10:03,815 cmd_helper._ValidateAndLogCommand:153  [host]> /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/md5sum_host /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/android/arm64-v8a/bin/forwarder_device
(INFO) 2016-08-09 03:10:04,315 cmd_helper._ValidateAndLogCommand:153  [host]> /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2ea7a27b7aa05 shell '( LD_LIBRARY_PATH=/data/local/tmp/forwarder/  /data/local/tmp/forwarder/device_forwarder );echo %$?'
(INFO) 2016-08-09 03:10:04,483 forwarder.Map:94  Forwarding using commands: [[u'--adb=/b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb', '--serial-id=00b2ea7a27b7aa05', '--map', '0', '44033'], [u'--adb=/b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb', '--serial-id=00b2ea7a27b7aa05', '--map', '0', '46180']]
(INFO) 2016-08-09 03:10:04,484 cmd_helper._ValidateAndLogCommand:153  [host]> /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/forwarder_host --adb=/b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb --serial-id=00b2ea7a27b7aa05 --map 0 44033
(CRITICAL) 2016-08-09 03:10:04,521 cmd_helper.GetCmdStatusAndOutput:175  STDERR: [0809/031004:ERROR:host_forwarder_main.cc(390)] ERROR: Connection to device failed.ERROR: Existing controllers:ERROR:   53547:8001ERROR:   42243:43584ERROR:   54334:42697ERROR:   42243:45588

(INFO) 2016-08-09 03:10:04,521 forwarder._KillDeviceLocked:335  Killing device_forwarder.
(INFO) 2016-08-09 03:10:04,522 cmd_helper._ValidateAndLogCommand:153  [host]> /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2ea7a27b7aa05 shell '( test -e /data/local/tmp/forwarder/device_forwarder );echo %$?'
(INFO) 2016-08-09 03:10:04,629 cmd_helper._ValidateAndLogCommand:153  [host]> /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2ea7a27b7aa05 shell '( LD_LIBRARY_PATH=/data/local/tmp/forwarder/  /data/local/tmp/forwarder/device_forwarder --kill-server );echo %$?'
(INFO) 2016-08-09 03:10:04,794 cmd_helper._ValidateAndLogCommand:153  [host]> /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2ea7a27b7aa05 shell '( ps );echo %$?'
(INFO) 2016-08-09 03:10:05,026 forwarder.Map:110  Currently running device_forwarders:
(WARNING) 2016-08-09 03:10:05,026 android_forwarder.Create:28  Failed to create forwarder. Currently forwarded connections:
(INFO) 2016-08-09 03:10:05,026 cmd_helper._ValidateAndLogCommand:153  [host]> /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2ea7a27b7aa05 forward --list
(WARNING) 2016-08-09 03:10:05,090 android_forwarder.Create:31    00e2164b278fb20b tcp:36227 localabstract:chrome_devtools_remote
(WARNING) 2016-08-09 03:10:05,090 android_forwarder.Create:31    00b88e248060cd28 tcp:53547 localabstract:chrome_device_forwarder
(WARNING) 2016-08-09 03:10:05,090 android_forwarder.Create:31    00b2b9cb898332e1 tcp:54334 localabstract:chrome_device_forwarder
(WARNING) 2016-08-09 03:10:05,090 android_forwarder.Create:31    00e2164b278fb20b tcp:37825 localabstract:chrome_device_forwarder
(WARNING) 2016-08-09 03:10:05,090 android_forwarder.Create:31    00b2b9cb898332e1 tcp:51717 localabstract:chrome_devtools_remote
(WARNING) 2016-08-09 03:10:05,090 android_forwarder.Create:31    00b88e248060cd28 tcp:52675 localabstract:chrome_devtools_remote
(WARNING) 2016-08-09 03:10:05,090 android_forwarder.Create:31    00497c1989bb1ee6 tcp:42243 localabstract:chrome_device_forwarder
(WARNING) 2016-08-09 03:10:05,090 android_forwarder.Create:31    00b675f2273fbc0b tcp:46776 localabstract:chrome_devtools_remote
(WARNING) 2016-08-09 03:10:05,090 android_forwarder.Create:31    00497c1989bb1ee6 tcp:41731 localabstract:chrome_devtools_remote
(WARNING) 2016-08-09 03:10:05,090 android_forwarder.Create:31    
(WARNING) 2016-08-09 03:10:05,090 android_forwarder.Create:35  Device tcp sockets in use:
(INFO) 2016-08-09 03:10:05,091 cmd_helper._ValidateAndLogCommand:153  [host]> /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2ea7a27b7aa05 pull /proc/net/tcp /tmp/tmpnVmwSd/tmp_ReadFileWithPull
(WARNING) 2016-08-09 03:10:05,298 android_forwarder.Create:39      sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode                                                     
(WARNING) 2016-08-09 03:10:05,298 android_forwarder.Create:39       0: 00000000:C36E 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21685 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,298 android_forwarder.Create:39       1: 00000000:C36F 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21689 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,298 android_forwarder.Create:39       2: 00000000:C370 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21693 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39       3: 00000000:C371 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21697 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39       4: 00000000:C372 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21701 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39       5: 00000000:C373 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21705 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39       6: 00000000:C374 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21709 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39       7: 00000000:C375 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21713 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39       8: 00000000:C376 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21717 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39       9: 00000000:C377 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21721 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      10: 00000000:C378 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21725 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      11: 00000000:C379 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21729 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      12: 00000000:C37A 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21733 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      13: 00000000:C35A 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21605 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      14: 00000000:C37B 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21737 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      15: 00000000:C35B 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21609 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      16: 00000000:C37C 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21741 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      17: 00000000:C35C 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21613 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      18: 00000000:C37D 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21745 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      19: 00000000:C35D 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21617 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      20: 00000000:C37E 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21749 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      21: 00000000:C35E 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21621 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      22: 00000000:C37F 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21753 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      23: 00000000:C35F 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21625 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      24: 00000000:7D00 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21809 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      25: 00000000:C380 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21757 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      26: 00000000:C360 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21629 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      27: 00000000:7D01 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21813 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,299 android_forwarder.Create:39      28: 00000000:C381 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21761 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      29: 00000000:C361 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21633 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      30: 00000000:7D02 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21817 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      31: 00000000:C382 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21765 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      32: 00000000:C362 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21637 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      33: 00000000:7D03 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21821 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      34: 00000000:C383 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21769 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      35: 00000000:C363 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21641 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      36: 00000000:7D04 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21825 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      37: 00000000:C384 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21773 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      38: 00000000:C364 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21645 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      39: 00000000:13C4 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21601 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      40: 00000000:7D05 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21829 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      41: 00000000:C385 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21777 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      42: 00000000:C365 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21649 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      43: 00000000:7D06 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21833 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      44: 00000000:C386 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21781 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      45: 00000000:C366 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21653 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      46: 00000000:7D07 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21837 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      47: 00000000:C387 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21785 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      48: 00000000:C367 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21657 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      49: 00000000:7D08 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21841 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      50: 00000000:C388 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21789 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      51: 00000000:C368 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21661 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      52: 00000000:7D09 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21845 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      53: 00000000:C389 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21793 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,300 android_forwarder.Create:39      54: 00000000:C369 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21665 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,301 android_forwarder.Create:39      55: 00000000:7D0A 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21849 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,301 android_forwarder.Create:39      56: 00000000:C38A 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21797 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,301 android_forwarder.Create:39      57: 00000000:C36A 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21669 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,301 android_forwarder.Create:39      58: 00000000:7D0B 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21853 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,301 android_forwarder.Create:39      59: 00000000:C38B 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21801 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,301 android_forwarder.Create:39      60: 00000000:C36B 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21673 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,301 android_forwarder.Create:39      61: 00000000:C38C 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21805 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,301 android_forwarder.Create:39      62: 00000000:C36C 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21677 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,301 android_forwarder.Create:39      63: 0100007F:13AD 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 28958 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,301 android_forwarder.Create:39      64: 00000000:C36D 00000000:0000 8A 00000000:00000000 00:00000000 00000000     0        0 21681 1 0000000000000000 100 0 0 10 0                     
(WARNING) 2016-08-09 03:10:05,301 android_forwarder.Create:43  Alive webpagereplay instances:
(WARNING) 2016-08-09 03:10:05,310 shared_page_state.DumpStateUponFailure:136  Cannot dump browser state: No browser.
(INFO) 2016-08-09 03:10:05,310 cmd_helper._ValidateAndLogCommand:153  [host]> /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2ea7a27b7aa05 shell '( /system/bin/screencap -p /data/local/tmp/temp_file-c03ce77ae590b.png );echo %$?'
(INFO) 2016-08-09 03:10:08,920 cmd_helper._ValidateAndLogCommand:153  [host]> /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2ea7a27b7aa05 pull /data/local/tmp/temp_file-c03ce77ae590b.png /tmp/tmpLj4bwG.png
(INFO) 2016-08-09 03:10:10,033 cmd_helper._ValidateAndLogCommand:153  [host]> /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00b2ea7a27b7aa05 shell 'rm -f /data/local/tmp/temp_file-c03ce77ae590b.png'
Traceback (most recent call last):
  File "/b/rr/tmptjOH9x/w/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 78, in _RunStoryAndProcessErrorIfNeeded
    state.WillRunStory(story)
  File "/b/rr/tmptjOH9x/w/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 212, in WillRunStory
    archive_path, page.make_javascript_deterministic)
  File "/b/rr/tmptjOH9x/w/src/third_party/catapult/telemetry/telemetry/core/network_controller.py", line 27, in StartReplay
    archive_path, make_javascript_deterministic)
  File "/b/rr/tmptjOH9x/w/src/third_party/catapult/telemetry/telemetry/internal/platform/network_controller_backend.py", line 192, in StartReplay
    self._StartForwarder(local_ports)
  File "/b/rr/tmptjOH9x/w/src/third_party/catapult/telemetry/telemetry/internal/platform/network_controller_backend.py", line 245, in _StartForwarder
    self._wpr_port_pairs.remote_ports))
  File "/b/rr/tmptjOH9x/w/src/third_party/catapult/telemetry/telemetry/internal/forwarders/android_forwarder.py", line 25, in Create
    return AndroidForwarder(self._device, port_pairs)
  File "/b/rr/tmptjOH9x/w/src/third_party/catapult/telemetry/telemetry/internal/forwarders/android_forwarder.py", line 60, in __init__
    for p in port_pairs if p], self._device)
  File "/b/rr/tmptjOH9x/w/src/third_party/catapult/devil/devil/android/forwarder.py", line 116, in Map
    exit_code, '\n'.join(output)))
HostForwarderError: /b/rr/tmptjOH9x/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/forwarder_host exited with 1:


I am not sure whether this bug should be addressed by telemetry/devil being more resilient about this type of failures, or whether this needs help from the lab infra (and should be a purple failure)

Comment 1 by sullivan@chromium.org, Aug 9 2016

Components: Infra>Client>Perf

Comment 2 by jbudorick@chromium.org, Aug 9 2016 

I'm guessing that port 44033 may already be in use on the host? Hard to say, as we're only logging all ports in use on the device but only the ports that the forwarder is using on the host.

Comment 3 by nedngu...@google.com, Aug 9 2016 

Is port 44033 fixed? If this is a port collision problem, I would expect only 1 benchmark fails at most, not all the benchmarks that are assigned to run on device 3.

Comment 4 by jbudorick@chromium.org, Aug 9 2016 

Hm, no, it's not. Just looked at the other failure logs & forwarder mapping is failing for a bunch of other ports, too.

Comment 5 by rnep...@chromium.org, Aug 9 2016 

Device status does not use the host forwarder. This failure has nothing to do with device status so the fix probably lies somewhere else.

Comment 6 by jbudorick@chromium.org, Aug 9 2016

Summary: single-device host_forwarder failures on N5X perf (was: device_status_check does not check for HostForwarderError )
Project Member

Comment 7 by sheriffbot@chromium.org, Aug 10 2016

Labels: Hotlist-Google

Comment 8 by benhenry@chromium.org, Aug 16 2016

Status: Available (was: Untriaged)
Chrome Infra side won't have cycles to debug this for at least a couple of weeks unless it's a higher priority.

Comment 9 by rnep...@chromium.org, Aug 16 2016 

I'm not really sure what the correct thing to do here is. The test is failing while actually running the benchmark via run_benchmark so I am hesitant to mark it an infra issue. 

I dont think being resilient to this type of failure is possible though, because WPR will not work if the host forwarder is malfunctioning. The correct course of action would probably be to find a way to recover from this type of failure, but with the forwarder thats easier said than done. Since it passes in the next run it looks like something that restarting the host fixes, but that isn't an option for mid-run recovery. We probably have to selectively do something on the host though if we detect this issue; which would probably put the fix somewhere in the run_benchmark or android test runner realm.

Comment 10 by sullivan@chromium.org, Aug 16 2016 

Looks like all the devices are okay now on this host? Should we close this issue?

Comment 11 by nedngu...@google.com, Aug 17 2016 

Randy: can you explain why did the host forwarder fail in this case?

Comment 12 by rnep...@chromium.org, Aug 17 2016 

No clue unfortunately.

Comment 13 by jbudorick@chromium.org, Aug 17 2016 

I'd like to look at this a little more before we close it.

Comment 14 by benhenry@chromium.org, Aug 18 2016

Status: Assigned (was: Available)

Comment 15 by benhenry@chromium.org, Feb 13 2017

Status: WontFix (was: Assigned)
►

Sign in to add a comment