Nexus 5 (1) and (2) and Nexus 6 (1) taking very long to build on chromium.perf |
||||||
Issue descriptionLink to buildbot status page: https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf%20%281%29 (build 4359 running for 27 hours) John, can you help triage?
,
Oct 3 2016
,
Oct 3 2016
Nexus 6 (2) running for 51 hours: https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus6%20Perf%20%281%29/builds/4160
,
Oct 3 2016
Turns out this is forwarder-related. :( I'm debugging on the N5.1 bot, please do not intervene until further notice.
,
Oct 3 2016
A forwarder --unmap call is hanging. The host controller for the given device appears unable to stop its internal thread and is hung in the thread dtor. Still investigating why. relevant stack traces: #0 0x00007f29ee39c66b in pthread_join (threadid=139818013406976, thread_return=0x0) at pthread_join.c:92 #1 0x000000000042da19 in Join () at ../../base/threading/platform_thread_posix.cc:221 #2 0x000000000042e1f3 in Stop () at ../../base/threading/thread.cc:183 #3 ~Thread () at ../../base/threading/thread.cc:64 #4 0x000000000040c115 in ~HostController () at ../../tools/android/forwarder2/host_controller.cc:63 #5 0x0000000000410182 in depart () at ../../base/memory/linked_ptr.h:144 #6 ~linked_ptr () at ../../base/memory/linked_ptr.h:82 #7 DeleteRefCountedValueInMapFromIterator<std::unordered_map<std::basic_string<char>, linked_ptr<forwarder2::HostController>, base_hash::hash<std::basic_string<char> >, std::equal_to<std::basic_string<char> >, std::allocator<std::pair<std::basic_string<char> const, linked_ptr<forwarder2::HostController> > > >, std::__detail::_Hashtable_iterator<std::pair<std::basic_string<char> const, linked_ptr<forwarder2::HostController> >, false, false> > () at ../../tools/android/forwarder2/util.h:32 #8 0x0000000000410098 in DeleteRefCountedValueInMap<std::unordered_map<std::basic_string<char>, linked_ptr<forwarder2::HostController>, base_hash::hash<std::basic_string<char> >, std::equal_to<std::basic_string<char> >, std::allocator<std::pair<std::basic_string<char> const, linked_ptr<forwarder2::HostController> > > >, std::basic_string<char> > () at ../../tools/android/forwarder2/util.h:22 #9 0x000000000040ed2d in Unmap () at ../../tools/android/forwarder2/host_forwarder_main.cc:218 #10 HandleRequestOnInternalThread () at ../../tools/android/forwarder2/host_forwarder_main.cc:274 child thread (I think): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00000000004480ca in TimedWait () at ../../base/synchronization/waitable_event_posix.cc:219 #2 0x0000000000447f81 in base::WaitableEvent::Wait() () at ../../base/synchronization/waitable_event_posix.cc:156 #3 0x000000000043f90f in Run () at ../../base/message_loop/message_pump_default.cc:55 #4 0x000000000044787e in Run () at ../../base/run_loop.cc:35
,
Oct 3 2016
the last message for that host_controller in the host forwarder log: [1002/041956:ERROR:host_controller.cc(149)] 0d890b5f25995524: Device could not handle the new Adb Data Connection.
,
Oct 3 2016
Killed off the host forwarder on N5.1 to unstick the bot.
,
Oct 3 2016
N5.2 is in a similar scenario, though the child thread is in a different state (and is indistinguishable from other HostController threads).
,
Oct 3 2016
(and more tests are timing out)
,
Oct 3 2016
N5.2 also had a single instance of "Device could not handle the new Adb Data Connection." in its host forwarder log.
,
Oct 3 2016
N6.1 is stuck in an --unmap-all on ZX1G22KGK7 w/ the same error message.
,
Oct 3 2016
Current theory: in all of the above cases, the forwarder hit an error in HostController::StartForwarder [1]. (That it's the same error in StartForwarder is interesting but not relevant to this issue AFAICT.) However, the calling function, ReadCommandOnInternalThread [2], always calls HostForwarder::ReadNextCommandSoon [3] after StartForwarder returns, even in the error case. ReadNextCommandSoon posts a task to the internal thread, but on an error case, I don't that command ever comes, so the task is left to wait indefinitely. [1] https://codesearch.chromium.org/chromium/src/tools/android/forwarder2/host_controller.cc?rcl=0&l=130 [2] https://codesearch.chromium.org/chromium/src/tools/android/forwarder2/host_controller.cc?rcl=0&l=126 [3] https://codesearch.chromium.org/chromium/src/tools/android/forwarder2/host_controller.cc?rcl=0&l=88
,
Oct 3 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/bacf00e3bd031eaef1facd16335e355c722cd9fe commit bacf00e3bd031eaef1facd16335e355c722cd9fe Author: jbudorick <jbudorick@chromium.org> Date: Mon Oct 03 17:23:55 2016 [android] Don't attempt to read a command on failure to start the forwarder. BUG= 652250 Review-Url: https://codereview.chromium.org/2385143003 Cr-Commit-Position: refs/heads/master@{#422449} [modify] https://crrev.com/bacf00e3bd031eaef1facd16335e355c722cd9fe/tools/android/forwarder2/host_controller.cc [modify] https://crrev.com/bacf00e3bd031eaef1facd16335e355c722cd9fe/tools/android/forwarder2/host_controller.h
,
Oct 3 2016
Will upgrade the binary configurations momentarily. In the meantime, I've also filed https://bugs.chromium.org/p/chromium/issues/detail?id=652337 to add strict script-side timeouts to invocations of the forwarder.
,
Oct 3 2016
,
Oct 3 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d9a51c08d84af40ad378387ff64fcb0edd7ed2b3 commit d9a51c08d84af40ad378387ff64fcb0edd7ed2b3 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Mon Oct 03 19:51:23 2016 Roll src/third_party/catapult/ c159bab23..1b48ca5d8 (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/c159bab23ac4..1b48ca5d8ef0 $ git log c159bab23..1b48ca5d8 --date=short --no-merges --format='%ad %ae %s' 2016-10-03 jbudorick [devil][telemetry] Roll forwarder binaries. BUG= 652250 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2385223003 Cr-Commit-Position: refs/heads/master@{#422497} [modify] https://crrev.com/d9a51c08d84af40ad378387ff64fcb0edd7ed2b3/DEPS
,
Oct 3 2016
,
Oct 4 2016
N6 (1): after my change in #13/#16 :( https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus6%20Perf%20%281%29/builds/4163
,
Oct 4 2016
Hanging while attempting to receive a command from the adb socket: (gdb) info thread Id Target Id Frame 3 Thread 0x7f758866d700 (LWP 6204) "HostControllers" 0x00007f75899aa66b in pthread_join (threadid=140142709200640, thread_return=0x0) at pthread_join.c:92 * 2 Thread 0x7f7584665700 (LWP 7086) "HostControllerT" 0x00007f7588bb3da3 in select () at ../sysdeps/unix/syscall-template.S:81 1 Thread 0x7f7589dc0780 (LWP 6200) "forwarder_host" 0x00007f7588bb3da3 in select () at ../sysdeps/unix/syscall-template.S:81 (gdb) thread 2 [Switching to thread 2 (Thread 0x7f7584665700 (LWP 7086))] #0 0x00007f7588bb3da3 in select () at ../sysdeps/unix/syscall-template.S:81 81 in ../sysdeps/unix/syscall-template.S (gdb) bt #0 0x00007f7588bb3da3 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000412352 in WaitForEvent () at ../../tools/android/forwarder2/socket.cc:422 #2 0x00000000004125a6 in Read () at ../../tools/android/forwarder2/socket.cc:315 #3 0x0000000000412557 in forwarder2::Socket::ReadNumBytes(void*, unsigned long) () at ../../tools/android/forwarder2/socket.cc:300 #4 0x0000000000409aa1 in ReadCommand () at ../../tools/android/forwarder2/command.cc:50 #5 0x0000000000409e3c in forwarder2::ReceivedCommand(forwarder2::command::Type, forwarder2::Socket*) () at ../../tools/android/forwarder2/command.cc:91 #6 0x000000000040ca38 in UnmapPortOnDevice () at ../../tools/android/forwarder2/host_controller.cc:179 #7 0x000000000040c5ed in OnInternalThreadError () at ../../tools/android/forwarder2/host_controller.cc:161 #8 ReadCommandOnInternalThread () at ../../tools/android/forwarder2/host_controller.cc:100 #9 0x0000000000450a61 in Run () at ../../base/callback.h:64 #10 RunTask () at ../../base/debug/task_annotator.cc:54 #11 0x000000000043e00b in RunTask () at ../../base/message_loop/message_loop.cc:405 #12 0x000000000043e2f8 in DeferOrRunPendingTask () at ../../base/message_loop/message_loop.cc:414 #13 0x000000000043e64b in DoWork () at ../../base/message_loop/message_loop.cc:513 #14 0x000000000043f93a in Run () at ../../base/message_loop/message_pump_default.cc:35 #15 0x000000000044788e in Run () at ../../base/run_loop.cc:35 #16 0x000000000042e765 in ThreadMain () at ../../base/threading/thread.cc:333 #17 0x000000000042db13 in ThreadFunc () at ../../base/threading/platform_thread_posix.cc:71 #18 0x00007f75899a9182 in start_thread (arg=0x7f7584665700) at pthread_create.c:312 #19 0x00007f7588bbc47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 (gdb)
,
Oct 4 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/6b5d14dc03df1b90ff9794d0887a491b2c7bad6c commit 6b5d14dc03df1b90ff9794d0887a491b2c7bad6c Author: jbudorick <jbudorick@chromium.org> Date: Tue Oct 04 23:58:55 2016 [android] Support not passing --adb-path. This is a workaround for the perf bots, which currently are using chromium's adb because of this in the perf test runner while using devil's adb while running benchmarks. This should be backed out once telemetry supports configurable adb paths. BUG= 652250 Review-Url: https://codereview.chromium.org/2391163002 [modify] https://crrev.com/6b5d14dc03df1b90ff9794d0887a491b2c7bad6c/scripts/slave/recipe_modules/chromium_android/api.py
,
Oct 5 2016
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/41917491b2cad984d0e50d6b5b93a7045174d3fa commit 41917491b2cad984d0e50d6b5b93a7045174d3fa Author: recipe-roller <recipe-roller@chromium.org> Date: Wed Oct 05 00:15:15 2016 Roll recipe dependencies (trivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) build: https://crrev.com/a1ec1d60129d55b93340a46b2bb389b86ea2fb95 Add gen/ui to generated bindings paths (rockot@chromium.org) https://crrev.com/6b5d14dc03df1b90ff9794d0887a491b2c7bad6c [android] Support not passing --adb-path. (jbudorick@chromium.org) TBR=martiniss@chromium.org,phajdan.jr@chromium.org BUG= 652250 ,652761 Recipe-Tryjob-Bypass-Reason: Autoroller Bugdroid-Send-Email: False Review-Url: https://codereview.chromium.org/2394473003 [modify] https://crrev.com/41917491b2cad984d0e50d6b5b93a7045174d3fa/infra/config/recipes.cfg
,
Oct 5 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1ea8ac8f3ef56a4315d4003a5fbdc5925deb4a16 commit 1ea8ac8f3ef56a4315d4003a5fbdc5925deb4a16 Author: recipe-roller <recipe-roller@chromium.org> Date: Wed Oct 05 00:30:31 2016 Roll recipe dependencies (trivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) build: https://crrev.com/a1ec1d60129d55b93340a46b2bb389b86ea2fb95 Add gen/ui to generated bindings paths (rockot@chromium.org) https://crrev.com/6b5d14dc03df1b90ff9794d0887a491b2c7bad6c [android] Support not passing --adb-path. (jbudorick@chromium.org) TBR=martiniss@chromium.org,phajdan.jr@chromium.org BUG= 652250 ,652761 Recipe-Tryjob-Bypass-Reason: Autoroller Bugdroid-Send-Email: False Review-Url: https://codereview.chromium.org/2396713002 Cr-Commit-Position: refs/heads/master@{#423020} [modify] https://crrev.com/1ea8ac8f3ef56a4315d4003a5fbdc5925deb4a16/infra/config/recipes.cfg
,
Oct 5 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/e4a7e1fbb7fe9781401cd01d78226256193fabca commit e4a7e1fbb7fe9781401cd01d78226256193fabca Author: jbudorick <jbudorick@chromium.org> Date: Wed Oct 05 15:48:56 2016 [android] Add pass_adb_path to chromium_tests.steps.DynamicPerfTests. Sequel to https://codereview.chromium.org/2391163002 BUG= 652250 Review-Url: https://codereview.chromium.org/2393843004 [modify] https://crrev.com/e4a7e1fbb7fe9781401cd01d78226256193fabca/scripts/slave/recipe_modules/chromium_tests/steps.py
,
Oct 5 2016
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/490dc2ad4febaa921fad93e5c234122cce4ae838 commit 490dc2ad4febaa921fad93e5c234122cce4ae838 Author: recipe-roller <recipe-roller@chromium.org> Date: Wed Oct 05 16:04:44 2016 Roll recipe dependencies (trivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) build: https://crrev.com/e4a7e1fbb7fe9781401cd01d78226256193fabca [android] Add pass_adb_path to chromium_tests.steps.DynamicPerfTests. (jbudorick@chromium.org) TBR=martiniss@chromium.org,phajdan.jr@chromium.org BUG= 652250 Recipe-Tryjob-Bypass-Reason: Autoroller Bugdroid-Send-Email: False Review-Url: https://codereview.chromium.org/2398673002 [modify] https://crrev.com/490dc2ad4febaa921fad93e5c234122cce4ae838/infra/config/recipes.cfg
,
Oct 5 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/def04ed853af32be10b954b0435409ed4e5e100e commit def04ed853af32be10b954b0435409ed4e5e100e Author: recipe-roller <recipe-roller@chromium.org> Date: Wed Oct 05 16:02:50 2016 Roll recipe dependencies (trivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) build: https://crrev.com/e4a7e1fbb7fe9781401cd01d78226256193fabca [android] Add pass_adb_path to chromium_tests.steps.DynamicPerfTests. (jbudorick@chromium.org) TBR=martiniss@chromium.org,phajdan.jr@chromium.org BUG= 652250 Recipe-Tryjob-Bypass-Reason: Autoroller Bugdroid-Send-Email: False Review-Url: https://codereview.chromium.org/2389193005 Cr-Commit-Position: refs/heads/master@{#423174} [modify] https://crrev.com/def04ed853af32be10b954b0435409ed4e5e100e/infra/config/recipes.cfg
,
Oct 5 2016
S5 (2): https://build.chromium.org/p/chromium.perf/builders/Android%20Galaxy%20S5%20Perf%20%282%29/builds/3590 $ ps aux | head -n 1; ps aux | grep adb USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND chrome-+ 1480 0.0 0.0 3880 752 ? S Oct04 0:01 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb -s 32081d5f765c510d logcat -v threadtime chrome-+ 2343 0.0 0.0 3880 756 ? S Oct04 0:12 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb -s 3208611192be51c3 logcat -v threadtime chrome-+ 2597 0.0 0.0 3880 752 ? S Oct04 0:08 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb -s 3208531995be5145 logcat -v threadtime chrome-+ 9985 0.0 0.0 78752 4784 ? Ssl Oct04 0:56 /b/rr/tmpwyqju7/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/forwarder_host --adb=/b/rr/tmpwyqju7/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb --serial-id=32081d5f765c510d --map 0 43284 chrome-+ 13988 0.0 0.0 28340 5588 ? S Oct04 0:00 /usr/bin/python /b/rr/tmpwyqju7/rw/checkout/infra/.recipe_deps/build/scripts/slave/daemonizer.py -- /b/rr/tmpwyqju7/w/src/build/android/adb_logcat_monitor.py /b/rr/tmpwyqju7/w/src/out/logcat /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb chrome-+ 13989 0.0 0.0 27332 6144 ? S Oct04 0:09 python /b/rr/tmpwyqju7/w/src/build/android/adb_logcat_monitor.py /b/rr/tmpwyqju7/w/src/out/logcat /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb chrome-+ 19543 0.0 0.0 28340 5664 ? S Oct04 0:00 python -u /b/rr/tmpwyqju7/rw/checkout/infra/.recipe_deps/build/scripts/slave/daemonizer.py --action restart --pid-file-path /tmp/device_monitor.pid -- /b/rr/tmpwyqju7/rw/checkout/infra/.recipe_deps/build/scripts/slave/recipe_modules/chromium_android/resources/spawn_device_monitor.py /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb ["32081d5f765c510d", "3208611192be51c3", "3208531995be5145", "3208dd33a9c25169", "32085d1787be514b", "3208521990be511d", "320851777626611b"] --blacklist-file /b/rr/tmpwyqju7/w/src/out/bad_devices.json chrome-+ 19544 0.0 0.0 30840 7680 ? S Oct04 0:05 python /b/rr/tmpwyqju7/rw/checkout/infra/.recipe_deps/build/scripts/slave/recipe_modules/chromium_android/resources/spawn_device_monitor.py /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb ["32081d5f765c510d", "3208611192be51c3", "3208531995be5145", "3208dd33a9c25169", "32085d1787be514b", "3208521990be511d", "320851777626611b"] --blacklist-file /b/rr/tmpwyqju7/w/src/out/bad_devices.json chrome-+ 26212 0.0 0.0 3880 752 ? S Oct04 0:10 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb -s 32085d1787be514b logcat -v threadtime chrome-+ 26843 0.2 0.1 2023844 19404 ? Sl Oct04 4:32 python -u /b/rr/tmpwyqju7/w/src/build/android/test_runner.py perf --release --verbose --steps /tmp/tmpNS9Xl1.json --blacklist-file /b/rr/tmpwyqju7/w/src/out/bad_devices.json --collect-chartjson-data --max-battery-temp 350 --known-devices-file /home/chrome-bot/.android/known_devices.json --enable-platform-mode --adb-path /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb chrome-+ 27004 0.0 0.0 3880 752 ? S 12:44 0:00 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb devices chrome-+ 27024 0.0 0.0 3880 752 ? S 12:44 0:00 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb devices chrome-+ 27033 0.0 0.0 3880 752 ? S 12:44 0:00 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb devices chrome-+ 27049 0.0 0.0 3880 752 ? S 12:45 0:00 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb devices chrome-+ 27061 0.0 0.0 3880 752 ? S 12:45 0:00 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb devices chrome-+ 27066 0.0 0.0 3880 752 ? S 12:45 0:00 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb devices chrome-+ 27085 0.0 0.0 3880 752 ? S 12:46 0:00 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb devices chrome-+ 27087 0.0 0.0 3880 752 ? S 12:46 0:00 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb devices chrome-+ 27108 0.0 0.0 3880 752 ? S 12:46 0:00 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb devices chrome-+ 27119 0.0 0.0 3880 752 ? S 12:46 0:00 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb devices chrome-+ 27123 0.0 0.0 0 0 ? Z 12:46 0:00 [adb] <defunct> chrome-+ 27124 0.0 0.0 3880 752 ? S 12:46 0:00 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb devices chrome-+ 27125 0.0 0.0 3876 756 ? S 12:46 0:00 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb -s 3208521990be511d shell dumpsys battery chrome-+ 27129 0.0 0.0 13736 932 pts/0 S+ 12:46 0:00 grep --color=auto adb chrome-+ 28347 0.0 0.0 3880 756 ? S Oct04 0:00 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb devices chrome-+ 28364 0.0 0.0 3880 752 ? S Oct04 0:00 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb devices chrome-+ 30274 94.9 0.0 150528 1620 ? Rl Oct04 1388:45 adb -P 5037 fork-server server chrome-+ 30683 0.0 0.0 3880 752 ? S Oct04 0:02 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb -s 3208dd33a9c25169 logcat -v threadtime chrome-+ 30685 0.0 0.0 3880 752 ? S Oct04 0:01 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb -s 3208521990be511d logcat -v threadtime chrome-+ 30686 0.0 0.0 3880 752 ? S Oct04 0:01 /b/rr/tmpwyqju7/w/src/third_party/android_tools/sdk/platform-tools/adb -s 320851777626611b logcat -v threadtime chrome-+ 31438 0.0 0.0 24424 1776 ? S Oct04 0:00 /b/rr/tmpwyqju7/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/forwarder_host --adb=/b/rr/tmpwyqju7/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb --serial-id=320851777626611b --unmap 49254
,
Oct 5 2016
Of particular note in the above, beyond the sheer number of things using adb and the fact that the forwarder is using a different binary than everything else, is the adb server, which is using almost 95% of CPU. chrome-+ 30274 94.9 0.0 150528 1620 ? Rl Oct04 1388:45 adb -P 5037 fork-server server
,
Oct 6 2016
from https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20%282%29/builds/1760 #0 0x00007fcf5e19ada3 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000412352 in WaitForEvent () at ../../tools/android/forwarder2/socket.cc:422 #2 0x00000000004125a6 in Read () at ../../tools/android/forwarder2/socket.cc:315 #3 0x0000000000412557 in forwarder2::Socket::ReadNumBytes(void*, unsigned long) () at ../../tools/android/forwarder2/socket.cc:300 #4 0x0000000000409aa1 in ReadCommand () at ../../tools/android/forwarder2/command.cc:50 #5 0x0000000000409e3c in forwarder2::ReceivedCommand(forwarder2::command::Type, forwarder2::Socket*) () at ../../tools/android/forwarder2/command.cc:91 #6 0x000000000040ca38 in UnmapPortOnDevice () at ../../tools/android/forwarder2/host_controller.cc:179 #7 0x000000000040c5ed in OnInternalThreadError () at ../../tools/android/forwarder2/host_controller.cc:161 #8 ReadCommandOnInternalThread () at ../../tools/android/forwarder2/host_controller.cc:100 #9 0x0000000000450a61 in Run () at ../../base/callback.h:64 #10 RunTask () at ../../base/debug/task_annotator.cc:54 #11 0x000000000043e00b in RunTask () at ../../base/message_loop/message_loop.cc:405 #12 0x000000000043e2f8 in DeferOrRunPendingTask () at ../../base/message_loop/message_loop.cc:414 #13 0x000000000043e64b in DoWork () at ../../base/message_loop/message_loop.cc:513 #14 0x000000000043f93a in Run () at ../../base/message_loop/message_pump_default.cc:35 #15 0x000000000044788e in Run () at ../../base/run_loop.cc:35 #16 0x000000000042e765 in ThreadMain () at ../../base/threading/thread.cc:333 #17 0x000000000042db13 in ThreadFunc () at ../../base/threading/platform_thread_posix.cc:71 #18 0x00007fcf5ef90182 in start_thread (arg=0x7fcf5bc50700) at pthread_create.c:312 #19 0x00007fcf5e1a347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 I've seen this stack multiple times; it's the next hang I'll address.
,
Oct 7 2016
from https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf%20%283%29/builds/3362 device-side: Id Target Id Frame * 1 Thread 2249.2249 "device_forwarde" 0xb6e556d4 in ?? () 2 Thread 2249.2250 "controller_thre" 0xb6e56904 in ?? () 3 Thread 2249.8763 "ForwardersManag" 0xb6e55ea4 in ?? () 4 Thread 2249.8764 "DeviceListener" 0xb6e56904 in ?? () (gdb) thread 1 [Switching to thread 1 (Thread 2249.2249)] #-1 0xb6e556d4 in ?? () (gdb) bt #-1 0xb6e556d4 in ?? () Backtrace stopped: Cannot access memory at address 0x7469724e (gdb) thread 2 [Switching to thread 2 (Thread 2249.2250)] #0 0xb6e56904 in ?? () (gdb) bt #0 0xb6e56904 in ?? () #1 0xb6e43ec8 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) thread 3 [Switching to thread 3 (Thread 2249.8763)] #0 0xb6e55ea4 in ?? () (gdb) bt #0 0xb6e55ea4 in ?? () #1 0xb6ec7f86 in ?? () #2 0xb6ec1566 in ?? () #3 0xb6ed37f4 in ?? () #4 0xb6ed4294 in ?? () #5 0xb6ec0f98 in ?? () #6 0xb6eba4ca in ?? () #7 0xb6ebb932 in ?? () #8 0xb6ec4eea in ?? () #9 0xb6eb05d6 in ?? () #10 0xb6eb4568 in ?? () #11 0xb6eb0a38 in ?? () #12 0xb6eb6f46 in ?? () #13 0xb6ebab5a in ?? () #14 0xb6ebafc8 in ?? () #15 0xb6ebb242 in ?? () #16 0xb6ebbd4c in ?? () #17 0xb6ec1780 in ?? () #18 0xb6ec7272 in ?? () #19 0xb6ec6c84 in ?? () #20 0xb6e42174 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) thread 4 [Switching to thread 4 (Thread 2249.8764)] #0 0xb6e56904 in ?? () (gdb) bt #0 0xb6e56904 in ?? () #1 0xb6e43ec8 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb)
,
Oct 8 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/06a677a5adc5a85a4c6516f822a3930fe2427128 commit 06a677a5adc5a85a4c6516f822a3930fe2427128 Author: jbudorick <jbudorick@chromium.org> Date: Sat Oct 08 02:05:41 2016 [Android] Add timeout to unmap reply handling. BUG= 652250 Review-Url: https://codereview.chromium.org/2402793002 Cr-Commit-Position: refs/heads/master@{#424043} [modify] https://crrev.com/06a677a5adc5a85a4c6516f822a3930fe2427128/tools/android/forwarder2/command.cc [modify] https://crrev.com/06a677a5adc5a85a4c6516f822a3930fe2427128/tools/android/forwarder2/command.h [modify] https://crrev.com/06a677a5adc5a85a4c6516f822a3930fe2427128/tools/android/forwarder2/host_controller.cc [modify] https://crrev.com/06a677a5adc5a85a4c6516f822a3930fe2427128/tools/android/forwarder2/socket.cc [modify] https://crrev.com/06a677a5adc5a85a4c6516f822a3930fe2427128/tools/android/forwarder2/socket.h
,
Oct 8 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/312de254d8040722741baa13fa1f13d266c208d3 commit 312de254d8040722741baa13fa1f13d266c208d3 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Sat Oct 08 05:56:17 2016 Roll src/third_party/catapult/ 455c70b71..2e8df65af (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/455c70b716be..2e8df65af29f $ git log 455c70b71..2e8df65af --date=short --no-merges --format='%ad %ae %s' 2016-10-07 jbudorick [devil][telemetry] Roll forwarder2 to pick up unmap timeouts. BUG= 652250 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2404613003 Cr-Commit-Position: refs/heads/master@{#424073} [modify] https://crrev.com/312de254d8040722741baa13fa1f13d266c208d3/DEPS
,
Oct 11 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/540552d9f246614e7afc613e648c77263bf63aa0 commit 540552d9f246614e7afc613e648c77263bf63aa0 Author: recipe-roller <recipe-roller@chromium.org> Date: Tue Oct 11 21:49:23 2016
,
Oct 11 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/ba74db9f1be6a4fe27b4ba54dddbd195865f3ec1 commit ba74db9f1be6a4fe27b4ba54dddbd195865f3ec1 Author: recipe-roller <recipe-roller@chromium.org> Date: Tue Oct 11 22:53:53 2016
,
Oct 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/def04ed853af32be10b954b0435409ed4e5e100e commit def04ed853af32be10b954b0435409ed4e5e100e Author: recipe-roller <recipe-roller@chromium.org> Date: Wed Oct 05 16:02:50 2016 Roll recipe dependencies (trivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) build: https://crrev.com/e4a7e1fbb7fe9781401cd01d78226256193fabca [android] Add pass_adb_path to chromium_tests.steps.DynamicPerfTests. (jbudorick@chromium.org) TBR=martiniss@chromium.org,phajdan.jr@chromium.org BUG= 652250 Recipe-Tryjob-Bypass-Reason: Autoroller Bugdroid-Send-Email: False Review-Url: https://codereview.chromium.org/2389193005 Cr-Commit-Position: refs/heads/master@{#423174} [modify] https://crrev.com/def04ed853af32be10b954b0435409ed4e5e100e/infra/config/recipes.cfg
,
Oct 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3e6279a8320141e10ee76679b1fa69097bca86e9 commit 3e6279a8320141e10ee76679b1fa69097bca86e9 Author: jbudorick <jbudorick@chromium.org> Date: Thu Oct 06 19:41:04 2016 [Android] Don't pass adb path to sharded perf tests. BUG= 652250 Review-Url: https://codereview.chromium.org/2392423002 Cr-Commit-Position: refs/heads/master@{#423634} [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Galaxy_S5_Perf__1_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Galaxy_S5_Perf__2_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Galaxy_S5_Perf__3_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus5X_Perf__1_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus5X_Perf__2_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus5X_Perf__3_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus5X_WebView_Perf__1_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus5X_WebView_Perf__2_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus5X_WebView_Perf__3_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus5_Perf__1_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus5_Perf__2_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus5_Perf__3_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus6_Perf__1_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus6_Perf__2_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus6_Perf__3_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus6_WebView_Perf__1_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus6_WebView_Perf__2_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus6_WebView_Perf__3_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus7v2_Perf__1_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus7v2_Perf__2_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus7v2_Perf__3_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus9_Perf__1_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus9_Perf__2_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_Nexus9_Perf__3_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_One_Perf__1_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_One_Perf__2_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_Android_One_Perf__3_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/full_chromium_perf_fyi_Android_Power_Nexus_5X_Perf__1_.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/get_perf_test_list_old_data.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/host_info_failure.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/missing_device.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/src_side_script_fails.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.expected/test_failure.json [modify] https://crrev.com/3e6279a8320141e10ee76679b1fa69097bca86e9/infra/recipes/android/perf.py
,
Nov 4 2016
[Automated comment] removing mislabelled merge-merged-2840
,
Nov 16 2016
Perf bothealth Sheriff Ping: Pri-1 bugs should be pinged daily, and checked to make sure someone is following up. John@ any update on this?
,
Nov 16 2016
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by jbudorick@chromium.org
, Oct 3 2016