Failed steps failed list perf tests (Builder: Android Nexus5X Perf (1)) |
||||||||||||
Issue description
Log:
CRITICAL:root:(TimeoutThread-1-for-MainThread) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['devices'], retries=2, timeout=30), attempt 1 of 3: AdbCommandFailedError("adb devices: failed with exit status 1 and output:\n- cannot bind 'tcp:5037': Address already in use\n- ADB server didn't ACK\n- * failed to start daemon *\n- error: \n- error: \n- List of devices attached\n- adb server is out of date. killing...\n",)
CRITICAL:root:(TimeoutThread-2-for-MainThread) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['devices'], retries=2, timeout=30), attempt 2 of 3: AdbCommandFailedError("adb devices: failed with exit status 1 and output:\n- cannot bind 'tcp:5037': Address already in use\n- ADB server didn't ACK\n- * failed to start daemon *\n- error: \n- error: \n- List of devices attached\n- adb server is out of date. killing...\n",)
CRITICAL:root:(TimeoutThread-1-for-MainThread) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['devices'], retries=2, timeout=30), attempt 1 of 3: AdbCommandFailedError('adb devices: failed with exit status 1 and output:\n- ** daemon still not running\n- error: cannot connect to daemon: Connection refused\n- error: cannot connect to daemon: Connection refused\n- List of devices attached\n- adb server is out of date. killing...\n- * daemon started successfully *\n',)
CRITICAL:root:(TimeoutThread-1-for-MainThread) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['devices'], retries=2, timeout=30), attempt 1 of 3: AdbCommandFailedError("adb devices: failed with exit status 1 and output:\n- cannot bind 'tcp:5037': Address already in use\n- ADB server didn't ACK\n- * failed to start daemon *\n- error: \n- error: \n- List of devices attached\n- adb server is out of date. killing...\n",)
CRITICAL:root:(TimeoutThread-2-for-MainThread) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['devices'], retries=2, timeout=30), attempt 2 of 3: AdbCommandFailedError("adb devices: failed with exit status 1 and output:\n- cannot bind 'tcp:5037': Address already in use\n- ADB server didn't ACK\n- * failed to start daemon *\n- error: \n- error: \n- List of devices attached\n- adb server is out of date. killing...\n",)
CRITICAL:root:(TimeoutThread-1-for-MainThread) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['devices'], retries=2, timeout=30), attempt 1 of 3: AdbCommandFailedError('adb devices: failed with exit status 1 and output:\n- ** daemon still not running\n- error: cannot connect to daemon: Connection refused\n- error: cannot connect to daemon: Connection refused\n- List of devices attached\n- adb server is out of date. killing...\n- * daemon started successfully *\n',)
CRITICAL:root:(TimeoutThread-1-for-MainThread) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['devices'], retries=2, timeout=30), attempt 1 of 3: AdbCommandFailedError("adb devices: failed with exit status 1 and output:\n- cannot bind 'tcp:5037': Address already in use\n- ADB server didn't ACK\n- * failed to start daemon *\n- error: \n- error: \n- List of devices attached\n- adb server is out of date. killing...\n",)
CRITICAL:root:(TimeoutThread-2-for-MainThread) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['devices'], retries=2, timeout=30), attempt 2 of 3: AdbCommandFailedError("adb devices: failed with exit status 1 and output:\n- cannot bind 'tcp:5037': Address already in use\n- ADB server didn't ACK\n- * failed to start daemon *\n- error: \n- error: \n- List of devices attached\n- adb server is out of date. killing...\n",)
CRITICAL:root:(TimeoutThread-1-for-MainThread) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['devices'], retries=2, timeout=30), attempt 1 of 3: AdbCommandFailedError('adb devices: failed with exit status 1 and output:\n- ** daemon still not running\n- error: cannot connect to daemon: Connection refused\n- error: cannot connect to daemon: Connection refused\n- List of devices attached\n- adb server is out of date. killing...\n- * daemon started successfully *\n',)
CRITICAL:root:(TimeoutThread-1-for-MainThread) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['devices'], retries=2, timeout=30), attempt 1 of 3: AdbCommandFailedError("adb devices: failed with exit status 1 and output:\n- cannot bind 'tcp:5037': Address already in use\n- ADB server didn't ACK\n- * failed to start daemon *\n- error: \n- error: \n- List of devices attached\n- adb server is out of date. killing...\n",)
CRITICAL:root:(TimeoutThread-2-for-MainThread) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['devices'], retries=2, timeout=30), attempt 2 of 3: AdbCommandFailedError("adb devices: failed with exit status 1 and output:\n- cannot bind 'tcp:5037': Address already in use\n- ADB server didn't ACK\n- * failed to start daemon *\n- error: \n- error: \n- List of devices attached\n- adb server is out of date. killing...\n",)
Traceback (most recent call last):
<module> at /b/rr/tmp6ITCQw/w/src/tools/perf/run_benchmark:22
sys.exit(main())
main at /b/rr/tmp6ITCQw/w/src/tools/perf/run_benchmark:19
return benchmark_runner.main(config, [trybot_command.Trybot])
main at /b/rr/tmp6ITCQw/w/src/third_party/catapult/telemetry/telemetry/benchmark_runner.py:428
return command_instance.Run(options)
Run at /b/rr/tmp6ITCQw/w/src/third_party/catapult/telemetry/telemetry/benchmark_runner.py:157
possible_reference_browser = browser_finder.FindBrowser(args)
Cacher at /b/rr/tmp6ITCQw/w/src/third_party/catapult/telemetry/telemetry/decorators.py:34
cacher.__cache[key] = obj(*args, **kwargs)
FindBrowser at /b/rr/tmp6ITCQw/w/src/third_party/catapult/telemetry/telemetry/internal/browser/browser_finder.py:62
devices = device_finder.GetDevicesMatchingOptions(options)
GetDevicesMatchingOptions at /b/rr/tmp6ITCQw/w/src/third_party/catapult/telemetry/telemetry/internal/platform/device_finder.py:38
devices = _GetAllAvailableDevices(options)
_GetAllAvailableDevices at /b/rr/tmp6ITCQw/w/src/third_party/catapult/telemetry/telemetry/internal/platform/device_finder.py:24
devices.extend(device.FindAllAvailableDevices(options))
FindAllAvailableDevices at /b/rr/tmp6ITCQw/w/src/third_party/catapult/telemetry/telemetry/internal/platform/android_device.py:179
devices = AndroidDevice.GetAllConnectedDevices(blacklist)
GetAllConnectedDevices at /b/rr/tmp6ITCQw/w/src/third_party/catapult/telemetry/telemetry/internal/platform/android_device.py:37
device_serials = GetDeviceSerials(blacklist)
GetDeviceSerials at /b/rr/tmp6ITCQw/w/src/third_party/catapult/telemetry/telemetry/internal/platform/android_device.py:62
device_serials = _ListSerialsOfHealthyOnlineDevices(blacklist)
_ListSerialsOfHealthyOnlineDevices at /b/rr/tmp6ITCQw/w/src/third_party/catapult/telemetry/telemetry/internal/platform/android_device.py:51
for d in device_utils.DeviceUtils.HealthyDevices(blacklist)
HealthyDevices at /b/rr/tmp6ITCQw/w/src/third_party/catapult/devil/devil/android/device_utils.py:2389
for adb in adb_wrapper.AdbWrapper.Devices():
Devices at /b/rr/tmp6ITCQw/w/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py:360
retries=retries)
_RawDevices at /b/rr/tmp6ITCQw/w/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py:380
output = cls._RunAdbCmd(cmd, timeout=timeout, retries=retries)
timeout_retry_wrapper at /b/rr/tmp6ITCQw/w/src/third_party/catapult/devil/devil/android/decorators.py:57
retry_if_func=retry_if_func)
Run at /b/rr/tmp6ITCQw/w/src/third_party/catapult/devil/devil/utils/timeout_retry.py:159
error_log_func=error_log_func)
JoinAll at /b/rr/tmp6ITCQw/w/src/third_party/catapult/devil/devil/utils/reraiser_thread.py:186
self._JoinAll(watcher, timeout)
_JoinAll at /b/rr/tmp6ITCQw/w/src/third_party/catapult/devil/devil/utils/reraiser_thread.py:158
thread.ReraiseIfException()
run at /b/rr/tmp6ITCQw/w/src/third_party/catapult/devil/devil/utils/reraiser_thread.py:81
self._ret = self._func(*self._args, **self._kwargs)
<lambda> at /b/rr/tmp6ITCQw/w/src/third_party/catapult/devil/devil/utils/timeout_retry.py:152
child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
impl at /b/rr/tmp6ITCQw/w/src/third_party/catapult/devil/devil/android/decorators.py:47
return f(*args, **kwargs)
_RunAdbCmd at /b/rr/tmp6ITCQw/w/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py:258
args, output, status, device_serial)
AdbCommandFailedError: adb devices: failed with exit status 1 and output:
- cannot bind 'tcp:5037': Address already in use
- ADB server didn't ACK
- * failed to start daemon *
- error:
- error:
- List of devices attached
- adb server is out of date. killing...
Locals:
args : ['devices']
check_error : True
cls : <class 'devil.android.sdk.adb_wrapper.AdbWrapper'>
cpu_affinity : None
device_serial : None
output : "cannot bind 'tcp:5037': Address already in use\nADB server didn't ACK\n* failed to start daemon *\nerror: \nerror: \nList of devices attached\nadb server is out of date. killing...\n"
retries : 2
status : 1
timeout : 30
step returned non-zero exit code: 1
@@@STEP_LOG_LINE@json.output (invalid)@null@@@
@@@STEP_LINK@logdog-->json.output (invalid)@https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5X_Perf__1_%2F3464%2F%2B%2Frecipes%2Fsteps%2FList_Perf_Tests%2F0%2Flogs%2Fjson.output__invalid_%2F0@@@
@@@STEP_LOG_END@json.output (invalid)@@@
@@@STEP_FAILURE@@@
,
Oct 15 2016
This is apparently affecting all the android bots.
,
Oct 15 2016
,
Oct 15 2016
,
Oct 15 2016
,
Oct 15 2016
Actually I am not confident that Randy's CL is the cause. It seems like adb is having problem connecting to the device. Can Infra also take a look?
,
Oct 15 2016
I suspect that there is some stray adb process that block telemetry from using adb. Hence I am making a hacky fix in https://codereview.chromium.org/2419073003/
,
Oct 15 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/746078e0517a27285eb253fc847453e301e98a3e commit 746078e0517a27285eb253fc847453e301e98a3e Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Sat Oct 15 02:29:24 2016 Roll src/third_party/catapult/ d63fac302..c1cc0281f (2 commits). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/d63fac3022e0..c1cc0281f791 $ git log d63fac302..c1cc0281f --date=short --no-merges --format='%ad %ae %s' 2016-10-14 nednguyen Revert of [Android] Check list length before printing forwarder information. (patchset #2 id:20001 of https://codereview.chromium.org/2419023003/ ) 2016-10-14 eakuefner [Node] Roll Node to v6.7.0 BUG= 656224 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2419263002 Cr-Commit-Position: refs/heads/master@{#425538} [modify] https://crrev.com/746078e0517a27285eb253fc847453e301e98a3e/DEPS
,
Oct 15 2016
Issue 656218 has been merged into this issue.
,
Oct 15 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f19bbc8b5de9b224d7c4fa0c993cb1e3e2009fd4 commit f19bbc8b5de9b224d7c4fa0c993cb1e3e2009fd4 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Sat Oct 15 12:36:55 2016 Roll src/third_party/catapult/ c1cc0281f..3684d40e7 (3 commits). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/c1cc0281f791..3684d40e7803 $ git log c1cc0281f..3684d40e7 --date=short --no-merges --format='%ad %ae %s' 2016-10-15 nednguyen [Telemetry] Only kill stray adb process when telemetry fails to list healthy devices 2016-10-15 nednguyen Reland of [Android] Check list length before printing forwarder information. (patchset #1 id:1 of https://codereview.chromium.org/2425473002/ ) 2016-10-14 nednguyen [Telemetry] Kill stray adb processes before gathering device list BUG= 656224 , 656224 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2414363004 Cr-Commit-Position: refs/heads/master@{#425561} [modify] https://crrev.com/f19bbc8b5de9b224d7c4fa0c993cb1e3e2009fd4/DEPS
,
Oct 15 2016
,
Oct 15 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/548e01d4ba62fc9ac54458e7d03aabf1dc26abfa commit 548e01d4ba62fc9ac54458e7d03aabf1dc26abfa Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Sat Oct 15 15:09:19 2016 Roll src/third_party/catapult/ 3684d40e7..ed924969a (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/3684d40e7803..ed924969adb8 $ git log 3684d40e7..ed924969a --date=short --no-merges --format='%ad %ae %s' 2016-10-15 nednguyen Fix typo: p.exitcode -> p.returncode BUG= 656224 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2419303002 Cr-Commit-Position: refs/heads/master@{#425563} [modify] https://crrev.com/548e01d4ba62fc9ac54458e7d03aabf1dc26abfa/DEPS
,
Oct 15 2016
- cannot bind 'tcp:5037': Address already in use - ADB server didn't ACK - * failed to start daemon * - error: - adb server is out of date. killing... It looks like something is occupying port 5037 (default port to be used by adb daemon server), hence I make this bandaid fix to pick a random port if we found port 5037 is used: https://codereview.chromium.org/2417403003/
,
Oct 15 2016
,
Oct 15 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d1621e4a72bc880507475c4c4634ac67086314e5 commit d1621e4a72bc880507475c4c4634ac67086314e5 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Sat Oct 15 20:00:23 2016 Roll src/third_party/catapult/ ed924969a..6591274a9 (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/ed924969adb8..6591274a9ba8 $ git log ed924969a..6591274a9 --date=short --no-merges --format='%ad %ae %s' 2016-10-15 nednguyen [devil] Pick a free port for adb daemon server when port 5037 is in used BUG= 656224 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2420313002 Cr-Commit-Position: refs/heads/master@{#425572} [modify] https://crrev.com/d1621e4a72bc880507475c4c4634ac67086314e5/DEPS
,
Oct 16 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d9b24004899c07810f7449cc68fac2c218e3867e commit d9b24004899c07810f7449cc68fac2c218e3867e Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Sun Oct 16 00:05:06 2016 Roll src/third_party/catapult/ 6591274a9..07c81af59 (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/6591274a9ba8..07c81af59775 $ git log 6591274a9..07c81af59 --date=short --no-merges --format='%ad %ae %s' 2016-10-15 nednguyen Revert of [devil] Pick a free port for adb daemon server when port 5037 is in used (patchset #6 id:100001 of https://codereview.chromium.org/2417403003/ ) BUG= 656224 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2421123002 Cr-Commit-Position: refs/heads/master@{#425579} [modify] https://crrev.com/d9b24004899c07810f7449cc68fac2c218e3867e/DEPS
,
Oct 16 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/707130a4d99e9bf9f27376743147a637160bb917 commit 707130a4d99e9bf9f27376743147a637160bb917 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Sun Oct 16 18:43:39 2016 Roll src/third_party/catapult/ 07c81af59..5e0af8f7d (2 commits). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/07c81af59775..5e0af8f7dc50 $ git log 07c81af59..5e0af8f7d --date=short --no-merges --format='%ad %ae %s' 2016-10-16 eakuefner [Node] Fix and re-enable eslint smoke test on Windows 2016-10-16 nednguyen [telemetry] Log all running processes when telemetry fails to create browser on Android BUG= 656224 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2425533003 Cr-Commit-Position: refs/heads/master@{#425593} [modify] https://crrev.com/707130a4d99e9bf9f27376743147a637160bb917/DEPS
,
Oct 16 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/4aa4e8225ccf1e272b8d787e81847141d980f6d2 commit 4aa4e8225ccf1e272b8d787e81847141d980f6d2 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Sun Oct 16 21:58:16 2016 Roll src/third_party/catapult/ 5e0af8f7d..60871179a (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/5e0af8f7dc50..60871179ac16 $ git log 5e0af8f7d..60871179a --date=short --no-merges --format='%ad %ae %s' 2016-10-16 nednguyen [Telemetry] Make add more details for ongoing processes logging BUG= 656224 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2426453002 Cr-Commit-Position: refs/heads/master@{#425596} [modify] https://crrev.com/4aa4e8225ccf1e272b8d787e81847141d980f6d2/DEPS
,
Oct 16 2016
Access failed list perf tests (Builder: Android Nexus5X Perf (1)) https://chromium.googlesource.com/chromium/src.git/+/4aa4e8225ccf1e272b8d787e81847141d980f6d2 https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/5e0af8f7dc50..60871179ac16 %s' processes logging CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
,
Oct 16 2016
Here is the log from the bot: chrome-+ 3757 3753 0 15:48 ? 00:00:00 /usr/bin/python /b/.remote_run_cipd/recipes.py --verbose remote --repository https://chromium.googlesource.com/chromium/src.git --revision 0d24e8746d06cfb185af5eb54cdbaa557c311148 --workdir /b/rr/tmpUpiLTT/rw --use-gitiles -- --verbose run --properties-file /b/rr/tmpUpiLTT/remote_run_properties.json --workdir /b/rr/tmpUpiLTT/w --output-result-json /b/rr/tmpUpiLTT/recipe_result.json android/perf chrome-+ 3765 3757 0 15:48 ? 00:00:00 /usr/bin/python /b/rr/tmpUpiLTT/rw/checkout/infra/recipes.py --verbose run --properties-file /b/rr/tmpUpiLTT/remote_run_properties.json --workdir /b/rr/tmpUpiLTT/w --output-result-json /b/rr/tmpUpiLTT/recipe_result.json android/perf chrome-+ 3778 3765 0 15:48 ? 00:00:05 /usr/bin/python -u /b/rr/tmpUpiLTT/rw/checkout/infra/.recipe_deps/recipe_engine/recipes.py --package /b/rr/tmpUpiLTT/rw/checkout/infra/config/recipes.cfg --bootstrap-script /b/rr/tmpUpiLTT/rw/checkout/infra/recipes.py --verbose run --properties-file /b/rr/tmpUpiLTT/remote_run_properties.json --workdir /b/rr/tmpUpiLTT/w --output-result-json /b/rr/tmpUpiLTT/recipe_result.json android/perf root 6578 2 0 15:51 ? 00:00:00 [kworker/u16:0] chrome-+ 13914 1 0 15:57 ? 00:00:00 /usr/bin/python /b/rr/tmpUpiLTT/rw/checkout/infra/.recipe_deps/build/scripts/slave/daemonizer.py -- /b/rr/tmpUpiLTT/w/src/build/android/adb_logcat_monitor.py /b/rr/tmpUpiLTT/w/src/out/logcat /b/rr/tmpUpiLTT/w/src/third_party/android_tools/sdk/platform-tools/adb chrome-+ 13915 13914 0 15:57 ? 00:00:00 python /b/rr/tmpUpiLTT/w/src/build/android/adb_logcat_monitor.py /b/rr/tmpUpiLTT/w/src/out/logcat /b/rr/tmpUpiLTT/w/src/third_party/android_tools/sdk/platform-tools/adb chrome-+ 17659 3778 1 16:12 ? 00:00:01 python -u /b/rr/tmpUpiLTT/w/src/tools/perf/run_benchmark list --browser android-chromium --json-output /tmp/tmpoITGei.json --num-shards 21 --device HT4B6JT02626 chrome-+ 18927 13915 0 16:13 ? 00:00:00 /b/rr/tmpUpiLTT/w/src/third_party/android_tools/sdk/platform-tools/adb -s HT4B6JT02626 logcat -v threadtime chrome-+ 18962 18927 0 16:13 ? 00:00:00 [adb] <defunct> chrome-+ 18989 18927 2 16:13 ? 00:00:00 [adb] <defunct> chrome-+ 19005 13915 0 16:13 ? 00:00:00 [adb] <defunct> chrome-+ 19006 13915 0 16:13 ? 00:00:00 [adb] <defunct> chrome-+ 19007 13915 0 16:13 ? 00:00:00 [adb] <defunct> chrome-+ 19009 13915 0 16:13 ? 00:00:00 [adb] <defunct> chrome-+ 19010 13915 0 16:13 ? 00:00:00 [adb] <defunct> chrome-+ 19021 1 0 16:13 ? 00:00:00 adb -P 5037 fork-server server chrome-+ 19106 17659 0 16:13 ? 00:00:00 ps -ef (https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Nexus9%20Perf%20%281%29/builds/3596/steps/List%20Perf%20Tests/logs/stdio) Based on the log, at the time telemetry is running, there is also adb_logcat_monitor.py running at the same time (PID = 13914) which probably has cause the race with using adb (it uses a different binary of adb from telemetry) & block telemetry from using adb.
,
Oct 16 2016
Can someone from infra takes a look, at this point, I highly suspect that some infra change has caused this outage.
,
Oct 16 2016
In particular, I am suspecting https://codereview.chromium.org/2402303003 is causing this failure. bpastene@, can you investigate & reassign to me if you think otherwise?
,
Oct 17 2016
Actually, I think it's stip@'s CL https://chromium.googlesource.com/chromium/src/+/e3c21739bcc8d5df70d2dd011c3f575570085b6d that trigger the failure since that update the adb used by adb_logcat_monitor.py . I will revert stip@'s CL to see if it fixes it.
,
Oct 17 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3d5a0fb164db4c2c24e83fe859463aa4268918a4 commit 3d5a0fb164db4c2c24e83fe859463aa4268918a4 Author: nednguyen <nednguyen@google.com> Date: Mon Oct 17 01:36:35 2016 Revert of Upgrade to new Android SDK. (patchset #16 id:300001 of https://codereview.chromium.org/2350013004/ ) Reason for revert: This may have changed the adb used by adb_logcat_monitor.py & creates a conflict with the one used by telemetry. BUG= 656224 Original issue's description: > Upgrade to new Android SDK. > > BUG= 623989 > > Committed: https://crrev.com/e3c21739bcc8d5df70d2dd011c3f575570085b6d > Cr-Commit-Position: refs/heads/master@{#425423} TBR=jbudorick@chromium.org,xunjieli@chromium.org,agrieve@chromium.org,stip@chromium.org # Not skipping CQ checks because original CL landed more than 1 days ago. BUG= 623989 Review-Url: https://codereview.chromium.org/2425563002 Cr-Commit-Position: refs/heads/master@{#425604} [modify] https://crrev.com/3d5a0fb164db4c2c24e83fe859463aa4268918a4/DEPS [modify] https://crrev.com/3d5a0fb164db4c2c24e83fe859463aa4268918a4/build/android/BUILD.gn [modify] https://crrev.com/3d5a0fb164db4c2c24e83fe859463aa4268918a4/build/android/devil_chromium.json [modify] https://crrev.com/3d5a0fb164db4c2c24e83fe859463aa4268918a4/build/android/lint/suppressions.xml [modify] https://crrev.com/3d5a0fb164db4c2c24e83fe859463aa4268918a4/build/android/pylib/constants/__init__.py [modify] https://crrev.com/3d5a0fb164db4c2c24e83fe859463aa4268918a4/build/config/android/config.gni [modify] https://crrev.com/3d5a0fb164db4c2c24e83fe859463aa4268918a4/components/cronet/android/api/build.xml [modify] https://crrev.com/3d5a0fb164db4c2c24e83fe859463aa4268918a4/components/cronet/tools/generate_javadoc.py [modify] https://crrev.com/3d5a0fb164db4c2c24e83fe859463aa4268918a4/docs/android_test_instructions.md [modify] https://crrev.com/3d5a0fb164db4c2c24e83fe859463aa4268918a4/tools/android/eclipse/.classpath
,
Oct 17 2016
The "List perf tests" step is now green!!! https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus6%20WebView%20Perf%20%281%29/builds/639/ This bug can now be de-prioritize, but left opened to track stip@'s relanding his CL.
,
Oct 17 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d1621e4a72bc880507475c4c4634ac67086314e5 commit d1621e4a72bc880507475c4c4634ac67086314e5 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Sat Oct 15 20:00:23 2016 Roll src/third_party/catapult/ ed924969a..6591274a9 (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/ed924969adb8..6591274a9ba8 $ git log ed924969a..6591274a9 --date=short --no-merges --format='%ad %ae %s' 2016-10-15 nednguyen [devil] Pick a free port for adb daemon server when port 5037 is in used BUG= 656224 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2420313002 Cr-Commit-Position: refs/heads/master@{#425572} [modify] https://crrev.com/d1621e4a72bc880507475c4c4634ac67086314e5/DEPS
,
Oct 17 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d9b24004899c07810f7449cc68fac2c218e3867e commit d9b24004899c07810f7449cc68fac2c218e3867e Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Sun Oct 16 00:05:06 2016 Roll src/third_party/catapult/ 6591274a9..07c81af59 (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/6591274a9ba8..07c81af59775 $ git log 6591274a9..07c81af59 --date=short --no-merges --format='%ad %ae %s' 2016-10-15 nednguyen Revert of [devil] Pick a free port for adb daemon server when port 5037 is in used (patchset #6 id:100001 of https://codereview.chromium.org/2417403003/ ) BUG= 656224 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2421123002 Cr-Commit-Position: refs/heads/master@{#425579} [modify] https://crrev.com/d9b24004899c07810f7449cc68fac2c218e3867e/DEPS
,
Nov 7 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/4711d631a805e410531fbba09e9c69278b2d17c0 commit 4711d631a805e410531fbba09e9c69278b2d17c0 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Mon Nov 07 16:01:00 2016 Roll src/third_party/catapult/ 8ac279fd8..0adb6ef9b (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/8ac279fd8db3..0adb6ef9b5dd $ git log 8ac279fd8..0adb6ef9b --date=short --no-merges --format='%ad %ae %s' 2016-11-07 nednguyen [Telemetry] Remove the logic of killing stray adb processes BUG= 656224 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2477243003 Cr-Commit-Position: refs/heads/master@{#430279} [modify] https://crrev.com/4711d631a805e410531fbba09e9c69278b2d17c0/DEPS
,
Nov 7 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0a50d3db988b7e3154017d39fbc900980c51d4a3 commit 0a50d3db988b7e3154017d39fbc900980c51d4a3 Author: scheib <scheib@chromium.org> Date: Mon Nov 07 18:52:54 2016 Revert of Roll src/third_party/catapult/ 8ac279fd8..0adb6ef9b (1 commit). (patchset #1 id:1 of https://codereview.chromium.org/2477243003/ ) Reason for revert: Failed compile on official.desktop.continuous for win trunk https://bugs.chromium.org/p/chromium/issues/detail?id=662998 Link to Build: ------------- https://uberchromegw.corp.google.com/i/official.desktop.continuous/builders/win%20trunk/builds/49769 Link to Log Path: ------------------ https://uberchromegw.corp.google.com/i/official.desktop.continuous/builders/win%20trunk/builds/49769/steps/compile/logs/stdio Error log : FAILED: net_unittests.exe net_unittests.exe.pdb C:/b/depot_tools/python276_bin/python.exe ../../build/toolchain/win/tool_wrapper.py link-wrapper environment.x64 False link.exe /nologo /OUT:./net_unittests.exe /PDB:./net_unittests.exe.pdb @./net_unittests.exe.rsp c:\b\build\slave\win_trunk\build\src\v8\src\heap\mark-compact.cc(1278) : fatal error C1001: An internal error has occurred in the compiler. (compiler file 'f:\dd\vctools\compiler\utc\src\p2\main.c', line 255) To work around this problem, try simplifying or changing the program near the locations listed above. Please choose the Technical Support command on the Visual C++ Help menu, or open the Technical Support help file for more information scheib@ could you please look into this. thank you Original issue's description: > Roll src/third_party/catapult/ 8ac279fd8..0adb6ef9b (1 commit). > > https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/8ac279fd8db3..0adb6ef9b5dd > > $ git log 8ac279fd8..0adb6ef9b --date=short --no-merges --format='%ad %ae %s' > 2016-11-07 nednguyen [Telemetry] Remove the logic of killing stray adb processes > > BUG= 656224 > > Documentation for the AutoRoller is here: > https://skia.googlesource.com/buildbot/+/master/autoroll/README.md > > If the roll is causing failures, see: > http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls > > CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel > TBR=catapult-sheriff@chromium.org > > Committed: https://crrev.com/4711d631a805e410531fbba09e9c69278b2d17c0 > Cr-Commit-Position: refs/heads/master@{#430279} TBR=catapult-sheriff@chromium.org,catapult-deps-roller@chromium.org # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG= 656224 Review-Url: https://codereview.chromium.org/2484833002 Cr-Commit-Position: refs/heads/master@{#430328} [modify] https://crrev.com/0a50d3db988b7e3154017d39fbc900980c51d4a3/DEPS
,
Nov 8 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/cfa43000d1547324bc97b8c1bf6a1b9303e6e8fb commit cfa43000d1547324bc97b8c1bf6a1b9303e6e8fb Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Tue Nov 08 00:10:04 2016 Roll src/third_party/catapult/ 8ac279fd8..d18e525a0 (6 commits). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/8ac279fd8db3..d18e525a0681 $ git log 8ac279fd8..d18e525a0 --date=short --no-merges --format='%ad %ae %s' 2016-11-07 aiolos Revert of [Telemetry] Remove the logic of killing stray adb processes (patchset #1 id:1 of https://codereview.chromium.org/2424843002/ ) 2016-11-07 aiolos Add missing import in model track. 2016-11-07 jbudorick [devil + systrace] Set up logging in profile_chrome. 2016-11-07 jbudorick [devil] Add explicit None checking to device_temp_file. 2016-11-07 perezju [Dashboard] Fix regex for complex story names 2016-11-07 nednguyen [Telemetry] Remove the logic of killing stray adb processes BUG=662301, 662472 , 656224 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2477333004 Cr-Commit-Position: refs/heads/master@{#430433} [modify] https://crrev.com/cfa43000d1547324bc97b8c1bf6a1b9303e6e8fb/DEPS
,
Nov 8 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/17c528dcce792753c035a812fe4dcfc41367f14e commit 17c528dcce792753c035a812fe4dcfc41367f14e Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Tue Nov 08 19:08:29 2016 Roll src/third_party/catapult/ 2f5014c3a..ac18e509a (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/2f5014c3a1f0..ac18e509a982 $ git log 2f5014c3a..ac18e509a --date=short --no-merges --format='%ad %ae %s' 2016-11-08 aiolos Reland of [Telemetry] Remove the logic of killing stray adb processes (patchset #1 id:1 of https://codereview.chromium.org/2487493002/ ) BUG= 656224 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2483083003 Cr-Commit-Position: refs/heads/master@{#430678} [modify] https://crrev.com/17c528dcce792753c035a812fe4dcfc41367f14e/DEPS
,
Dec 16 2016
Closing since this is green, but we still need to figure out how to properly unbreak the bisect bot for adb rolls. |
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by nedngu...@google.com
, Oct 15 2016