Look like a devil problem. John, can you take a look?

I    0.003s Main  command: /b/build/slave/Android_One_Perf__3_/build/src/build/android/test_runner.py perf --release --verbose --steps /tmp/tmpuJehme.json --blacklist-file /b/build/slave/Android_One_Perf__3_/build/src/out/bad_devices.json --collect-chartjson-data --max-battery-temp 350 --known-devices-file /b/build/site_config/.known_devices
I    0.004s TimeoutThread-1-for-MainThread  [host]> /b/build/slave/Android_One_Perf__3_/build/src/third_party/android_tools/sdk/platform-tools/adb devices
I    0.020s Main  Killing 15 lighttpd 2132
E    0.040s Main  Failed killing lighttpd 2132
Traceback (most recent call last):
  File "/b/build/slave/Android_One_Perf__3_/build/src/build/android/pylib/utils/test_environment.py", line 22, in _KillWebServers
    p.send_signal(s)
  File "/usr/lib/python2.7/dist-packages/psutil/__init__.py", line 174, in wrapper
    return fun(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/psutil/__init__.py", line 865, in send_signal
    raise AccessDenied(self.pid, name)
AccessDenied: (pid=2132)
I    0.059s Main  Killing 2 lighttpd 2132
E    0.059s Main  Failed killing lighttpd 2132
Traceback (most recent call last):
  File "/b/build/slave/Android_One_Perf__3_/build/src/build/android/pylib/utils/test_environment.py", line 22, in _KillWebServers
    p.send_signal(s)
  File "/usr/lib/python2.7/dist-packages/psutil/__init__.py", line 174, in wrapper
    return fun(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/psutil/__init__.py", line 865, in send_signal
    raise AccessDenied(self.pid, name)
AccessDenied: (pid=2132)
I    0.070s Main  Killing 3 lighttpd 2132
E    0.070s Main  Failed killing lighttpd 2132
Traceback (most recent call last):
  File "/b/build/slave/Android_One_Perf__3_/build/src/build/android/pylib/utils/test_environment.py", line 22, in _KillWebServers
    p.send_signal(s)
  File "/usr/lib/python2.7/dist-packages/psutil/__init__.py", line 174, in wrapper
    return fun(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/psutil/__init__.py", line 865, in send_signal
    raise AccessDenied(self.pid, name)
AccessDenied: (pid=2132)

(https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20One%20Perf%20(3)/builds/1379/steps/Sharded%20Perf%20Tests/logs/stdio)

lighttpd, my nemesis.

I'll take a look at this tomorrow.

This hasn't happened in more than two days, and I'm working toward getting rid of our lighttpd dependency elsewhere. In light of those two factors, I'm dropping the priority here and will continue to monitor the bot.

Update: "Sharded Perf Tests" has failed on May 4, with the same error:
https://build.chromium.org/p/chromium.perf/builders/Android%20One%20Perf%20%283%29/builds/1814. But it is not failing for most builds.

While the lighttpd errors are still present, that's not what killed the run.


I   14.611s TimeoutThread-1-for-cleanup_device(AG860440GCAC4GC)  [host]> /b/build/slave/Android_One_Perf__3_/build/src/third_party/android_tools/sdk/platform-tools/adb -s AG860440GCAC4GC shell '( pm path android );echo %$?'
I   15.069s TimeoutThread-1-for-cleanup_device(AG860440GCAC4GC)  condition 'pm_ready' met (0.5s)
I   15.069s TimeoutThread-1-for-cleanup_device(AG860440GCAC4GC)  [host]> /b/build/slave/Android_One_Perf__3_/build/src/third_party/android_tools/sdk/platform-tools/adb -s AG860440GCAC4GC shell '( getprop sys.boot_completed );echo %$?'
I   15.126s TimeoutThread-1-for-cleanup_device(AG860440GCAC4GC)  condition 'boot_completed' met (0.6s)
C   35.200s cleanup_device(AG8604420CI46GC)  ********************************************************************************
C   35.200s cleanup_device(AG8604420CI46GC)  Exception on thread TimeoutThread-1-for-cleanup_device(AG8604420CI46GC) (attempt 1 of 3)
C   35.200s cleanup_device(AG8604420CI46GC)  ********************************************************************************
C   35.201s cleanup_device(AG8604420CI46GC)  Traceback (most recent call last):
C   35.201s cleanup_device(AG8604420CI46GC)    File "/b/build/slave/Android_One_Perf__3_/build/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 167, in Run
C   35.201s cleanup_device(AG8604420CI46GC)      error_log_func=error_log_func)
C   35.201s cleanup_device(AG8604420CI46GC)    File "/b/build/slave/Android_One_Perf__3_/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll
C   35.201s cleanup_device(AG8604420CI46GC)      self._JoinAll(watcher, timeout)
C   35.202s cleanup_device(AG8604420CI46GC)    File "/b/build/slave/Android_One_Perf__3_/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 158, in _JoinAll
C   35.202s cleanup_device(AG8604420CI46GC)      thread.ReraiseIfException()
C   35.202s cleanup_device(AG8604420CI46GC)    File "/b/build/slave/Android_One_Perf__3_/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run
C   35.202s cleanup_device(AG8604420CI46GC)      self._ret = self._func(*self._args, **self._kwargs)
C   35.202s cleanup_device(AG8604420CI46GC)    File "/b/build/slave/Android_One_Perf__3_/build/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 160, in <lambda>
C   35.202s cleanup_device(AG8604420CI46GC)      child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
C   35.202s cleanup_device(AG8604420CI46GC)    File "/b/build/slave/Android_One_Perf__3_/build/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
C   35.202s cleanup_device(AG8604420CI46GC)      return f(*args, **kwargs)
C   35.202s cleanup_device(AG8604420CI46GC)    File "/b/build/slave/Android_One_Perf__3_/build/src/third_party/catapult/devil/devil/android/device_utils.py", line 515, in WaitUntilFullyBooted
C   35.202s cleanup_device(AG8604420CI46GC)      self.adb.WaitForDevice()
C   35.202s cleanup_device(AG8604420CI46GC)    File "/b/build/slave/Android_One_Perf__3_/build/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 723, in WaitForDevice
C   35.202s cleanup_device(AG8604420CI46GC)      self._RunDeviceAdbCmd(['wait-for-device'], timeout, retries)
C   35.202s cleanup_device(AG8604420CI46GC)    File "/b/build/slave/Android_One_Perf__3_/build/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 259, in _RunDeviceAdbCmd
C   35.202s cleanup_device(AG8604420CI46GC)      check_error=check_error)
C   35.202s cleanup_device(AG8604420CI46GC)    File "/b/build/slave/Android_One_Perf__3_/build/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
C   35.202s cleanup_device(AG8604420CI46GC)      return impl()
C   35.202s cleanup_device(AG8604420CI46GC)    File "/b/build/slave/Android_One_Perf__3_/build/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
C   35.203s cleanup_device(AG8604420CI46GC)      return f(*args, **kwargs)
C   35.203s cleanup_device(AG8604420CI46GC)    File "/b/build/slave/Android_One_Perf__3_/build/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 226, in _RunAdbCmd
C   35.203s cleanup_device(AG8604420CI46GC)      timeout_retry.CurrentTimeoutThreadGroup().GetRemainingTime())
C   35.203s cleanup_device(AG8604420CI46GC)    File "/b/build/slave/Android_One_Perf__3_/build/src/third_party/catapult/devil/devil/utils/cmd_helper.py", line 273, in GetCmdStatusAndOutputWithTimeout
C   35.203s cleanup_device(AG8604420CI46GC)      raise TimeoutError(output.getvalue())
C   35.203s cleanup_device(AG8604420CI46GC)  CommandTimeoutError
C   35.203s cleanup_device(AG8604420CI46GC)  ********************************************************************************
I   35.203s TimeoutThread-2-for-cleanup_device(AG8604420CI46GC)  [host]> /b/build/slave/Android_One_Perf__3_/build/src/third_party/android_tools/sdk/platform-tools/adb -s AG8604420CI46GC wait-for-device

Looks like all the Android One bots are failing consistently due to this.  Eg:

https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20One%20Perf%20%281%29/builds/2790/steps/Sharded%20Perf%20Tests/logs/stdio

https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20One%20Perf%20%282%29/builds/3104/steps/Sharded%20Perf%20Tests/logs/stdio

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3cee1c9cd4b3d25e0c983a503c0938aebb034ac8

commit 3cee1c9cd4b3d25e0c983a503c0938aebb034ac8
Author: jbudorick <jbudorick@chromium.org>
Date: Thu May 26 17:49:24 2016

[Android] Attempt to continue after single-device failure in cleanup_device.

BUG= 596974 

Review-Url: https://codereview.chromium.org/2015723003
Cr-Commit-Position: refs/heads/master@{#396222}

[modify] https://crrev.com/3cee1c9cd4b3d25e0c983a503c0938aebb034ac8/build/android/pylib/utils/test_environment.py

Seems fixed. Closing this bug.

I'd like to look at https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20One%20Perf%20%281%29/builds/2848/steps/Sharded%20Perf%20Tests/logs/stdio before closing this out.

Dropping priority to reflect reduced frequency, though.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/776800974c2d5834279964ea6634425e1447f031

commit 776800974c2d5834279964ea6634425e1447f031
Author: jbudorick <jbudorick@chromium.org>
Date: Thu Jun 02 15:42:06 2016

[Android] Attempt to continue on timeout in perf test runner setup.

BUG= 596974 

Review-Url: https://codereview.chromium.org/2029833003
Cr-Commit-Position: refs/heads/master@{#397420}

[modify] https://crrev.com/776800974c2d5834279964ea6634425e1447f031/build/android/pylib/utils/test_environment.py

#13 should address the remaining sharded perf test failures that have occurred on the A1 bot over the last few days.