That's probably due to the fact that a step/test won't complete until all of its child processes are either killed or released. Searching for the string 'all_pids_killed' in the linked log shows a stack trace and a failed attempt to do so. Not sure why though. Maybe the test isn't properly cleaning up after itself?

It was definitely running, not sure what was hanging though. Take a look at the full 'steps' stdio:

command timed out: 2400 seconds without output, attempting to kill
process killed by signal 9
program finished with exit code -1

Hmhh, so it's possible that telemetry hang after it finish running the tests. Probably at the steps where the atexit is called?

168 tests run in 179.0s, 0 failures.
(INFO) 2016-04-01 06:27:05,260 cmd_helper._ValidateAndLogCommand:153  [host]> /b/build/slave/android/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 06b9b9f700622636 shell '( dumpsys battery );echo %$?'
(INFO) 2016-04-01 06:27:05,362 cmd_helper._ValidateAndLogCommand:153  [host]> /b/build/slave/android/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 060752dc13c85a2b shell '( dumpsys battery );echo %$?'
(INFO) 2016-04-01 06:27:05,432 cmd_helper._ValidateAndLogCommand:153  [host]> /b/build/slave/android/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 05fcbefa0ae59f81 shell '( dumpsys battery );echo %$?'
(INFO) 2016-04-01 06:27:05,503 cmd_helper._ValidateAndLogCommand:153  [host]> /b/build/slave/android/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 05fc05a60ae4ffc7 shell '( dumpsys battery );echo %$?'
(INFO) 2016-04-01 06:27:05,621 cmd_helper._ValidateAndLogCommand:153  [host]> /b/build/slave/android/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 03f9fb2900621d4e shell '( dumpsys battery );echo %$?'
(INFO) 2016-04-01 06:27:05,723 cmd_helper._ValidateAndLogCommand:153  [host]> /b/build/slave/android/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 03d27fe9006adbae shell '( dumpsys battery );echo %$?'
(INFO) 2016-04-01 06:27:05,825 cmd_helper._ValidateAndLogCommand:153  [host]> /b/build/slave/android/build/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 03d27f97006adbb7 shell '( dumpsys battery );echo %$?'
[Running on builder: "linux_android_rel_ng"]


Randy: this maybe caused by the atexit hooks we added in the past to fix the problem of telemetry failed to cleanup the power measurement. Can you take a look?

Note that linux_android_rel_ng is very flaky according https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV0ZWxlbWV0cnlfcGVyZl91bml0dGVzdHMgKHdpdGggcGF0Y2gpDA

My guess is that if it is it would be this use of atexit:
https://code.google.com/p/chromium/codesearch#chromium/src/third_party/catapult/telemetry/telemetry/internal/platform/power_monitor/android_power_monitor_controller.py&q=f:telemetry%20atexit%5C.register&sq=package:chromium&type=cs

But, I dont know why those would hang. Adding a logging message to the end of that atexit call to make sure it is exiting.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cd1275746f3b9b8bb1f729bbcaedd9d6452f3533

commit cd1275746f3b9b8bb1f729bbcaedd9d6452f3533
Author: catapult-deps-roller <catapult-deps-roller@chromium.org>
Date: Wed Apr 06 00:04:47 2016

Roll src/third_party/catapult/ 0c61248dd..2557cf067 (2 commits).

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/0c61248dd7da..2557cf06771e

$ git log 0c61248dd..2557cf067 --date=short --no-merges --format='%ad %ae %s'

BUG= 599926 

TBR=catapult-sheriff@chromium.org

Review URL: https://codereview.chromium.org/1864793002

Cr-Commit-Position: refs/heads/master@{#385333}

[modify] https://crrev.com/cd1275746f3b9b8bb1f729bbcaedd9d6452f3533/DEPS

It looks like they are exiting the atexit function, so I do not think that is what is causing the issue. 

https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/49334/steps/telemetry_perf_unittests%20%28with%20patch%29/logs/stdio

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8a22f2f85b4cbbae4cfbd0e27170bfd4f3ff49a6

commit 8a22f2f85b4cbbae4cfbd0e27170bfd4f3ff49a6
Author: catapult-deps-roller <catapult-deps-roller@chromium.org>
Date: Sat Apr 09 16:37:53 2016

Roll src/third_party/catapult/ ea94b86fc..6120720f1 (1 commit).

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/ea94b86fc8da..6120720f1a8c

$ git log ea94b86fc..6120720f1 --date=short --no-merges --format='%ad %ae %s'

BUG= 599926 

TBR=catapult-sheriff@chromium.org

Review URL: https://codereview.chromium.org/1872513006

Cr-Commit-Position: refs/heads/master@{#386299}

[modify] https://crrev.com/8a22f2f85b4cbbae4cfbd0e27170bfd4f3ff49a6/DEPS

https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/54468/steps/telemetry_perf_unittests%20%28with%20patch%29/logs/stdio is the latest failure of this type.

Looking at the log, there are 10 atexit calls that happened after all telemetry tests have run & all of them finished (ctrl+f, search for "Try running" & "Did run").

So I am really unsure what is causing the test to stuck here.

telemetry_perf_unittest no longer fails in the last 200 builds: https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng?numbuilds=200