Issue 916824 link

Starred by 1 user

Issue metadata

Status:	Duplicate
Merged:	issue 915429
Owner:	ynovikov@chromium.org
Closed:	Dec 20
Cc:	chrome-gpu-perf-owners@chromium.org zhanliang@google.com jmad...@chromium.org
Components:	Internals>GPU>ANGLE
EstimatedDays:	----
NextAction:	----
OS:	----
Pri:	2
Type:	----
Performance-Sheriff-BotHealth Filed-Via-SoM

Blocked on:
issue 917124

angle_perftests failing on chromium.perf/Android Nexus5 Perf

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, Dec 20

Issue description

Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of zhanliang@google.com

angle_perftests failing on chromium.perf/Android Nexus5 Perf

Builders failed on: 
- Android Nexus5 Perf: 
  https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/Android%20Nexus5%20Perf

Comment 1 by crouleau@chromium.org, Dec 20

Culprit revision range is 616705 to 616802.

Comment 2 by crouleau@chromium.org, Dec 20

Cc: chrome-gpu-perf-owners@chromium.org jmad...@chromium.org

Comment 3 by crouleau@chromium.org, Dec 20

Components: Internals>GPU>ANGLE

Comment 4 by crouleau@chromium.org, Dec 20

Blockedon: 917124

Comment 5 by crouleau@chromium.org, Dec 20

I can't start a bisection job because of issue 917124.

Comment 6 by jmadill@google.com, Dec 20

Mergedinto: 915429
Owner: ynovikov@chromium.org
Status: Duplicate (was: Available)

Comment 7 by crouleau@chromium.org, Dec 20

It's also failing on https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-nexus5x-perf/1207 nexus 5x

Comment 8 by jmadill@google.com, Dec 20

Crouleau, looks like a differnet problem. Related to a timeout. Passed the next build:

https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-nexus5x-perf/1208

Comment 9 by crouleau@chromium.org, Dec 20

For nexusx it may or may not be a different root cause... Regardless, it is flaky or failing on both devices:

https://chromium-review.googlesource.com/c/chromium/src/+/1387832

Comment 10 by crouleau@chromium.org, Dec 20

There's a timeout error log in all the stack traces that I have seen: 

Traceback (most recent call last):
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 183, in Run
    self._component, extras=extras, raw=False, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run
    error_log_func=error_log_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1342, in StartInstrumentation
    large_output=True)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1161, in RunShellCommand
    output = handle_large_output(cmd, large_output)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1126, in handle_large_output
    handle_large_command(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1112, in handle_large_command
    return handle_check_return(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1103, in handle_check_return
    return run(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1099, in run
    return self.adb.Shell(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 533, in Shell
    output = self._RunDeviceAdbCmd(args, timeout, retries, check_error=False)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd
    check_error=check_error)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 277, in _RunAdbCmd
    timeout, env=cls._ADB_ENV)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/cmd_helper.py", line 425, in GetCmdStatusAndOutputWithTimeout
    raise TimeoutError(output.getvalue())
CommandTimeoutError: Timeout
D   50.938s TimeoutThread-1-for-list_tests(05b4266a0a4ebe51)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05b4266a0a4ebe51 shell '( TZ=utc ls -a -l /storage/emulated/legacy/ );echo %$?'
D   50.969s TimeoutThread-1-for-delete_temporary_file(05b4266a0a4ebe51)  STDOUT+STDERR: 
D   50.977s TimeoutThread-1-for-list_tests(05b4266a0a4ebe51)  STDOUT+STDERR: drwxrwx--- root     sdcard_r          2018-12-14 05:00 Alarms
drwxrwx--x root     sdcard_r          2018-12-14 05:00 Android
drwxrwx--- root     sdcard_r          2018-12-14 05:00 DCIM
drwxrwx--- root     sdcard_r          2018-12-14 05:00 Documents
drwxrwx--- root     sdcard_r          2018-12-14 05:00 Download
drwxrwx--- root     sdcard_r          2018-12-14 05:00 Movies
drwxrwx--- root     sdcard_r          2018-12-14 05:00 Music
drwxrwx--- root     sdcard_r          2018-12-14 05:00 Notifications
drwxrwx--- root     sdcard_r          2018-12-14 05:00 Pictures
drwxrwx--- root     sdcard_r          2018-12-14 05:00 Podcasts
drwxrwx--- root     sdcard_r          2018-12-14 05:00 Ringtones
drwxrwx--- root     sdcard_r          2018-12-14 05:08 chromium_tests_root
%0
D   51.003s TimeoutThread-1-for-delete_temporary_file(05b4266a0a4ebe51)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05b4266a0a4ebe51 shell 'rm -f /storage/emulated/legacy/temp_file-d610d9dc44a2.gtest_out'
E   51.004s list_tests(05b4266a0a4ebe51)  Shard failed: list_tests(05b4266a0a4ebe51)
Traceback (most recent call last):
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_environment.py", line 66, in wrapper
    return f(dev, *args, **kwargs)
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 423, in list_tests
    device=dev)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/crash_handler.py", line 32, in RetryOnSystemCrash
    return f(device)
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 422, in <lambda>
    None, d, flags=' '.join(flags), timeout=timeout),
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 196, in Run
    stdout_file.name, device.StatPath(stdout_file.name).get('st_size', 0))
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 2134, in StatPath
    'Cannot find file or directory: %r' % device_path, str(self))
CommandFailedError: (device: 05b4266a0a4ebe51) Cannot find file or directory: '/storage/emulated/legacy/temp_file-d610d9dc44a2.gtest_out'
D   51.005s TimeoutThread-1-for-delete_temporary_file(05b4266a0a4ebe51)  [host]> /b/swarming/w/ir/third_party/android_tools/sdk/platform-tools/adb -s 05b4266a0a4ebe51 shell 'rm -f /data/local/tmp/temp_file-2d49c248a2a9a'
E   51.005s list_tests(05b4266a0a4ebe51)  Device 05b4266a0a4ebe51 blacklisted: list_tests
I   51.007s Main  Generated json results file at /b/swarming/w/itKmxqQn/tmp6S_NH8
I   51.007s Main  Finishing archiving output.
E   51.008s Main  Error occurred.
Traceback (most recent call last):
  File "/b/swarming/w/ir/build/android/test_runner.py", line 1037, in main
    return RunTestsCommand(args)
  File "/b/swarming/w/ir/build/android/test_runner.py", line 723, in RunTestsCommand
    return RunTestsInPlatformMode(args)
  File "/b/swarming/w/ir/build/android/test_runner.py", line 869, in RunTestsInPlatformMode
    test_run.RunTests(raw_results)
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_test_run.py", line 58, in RunTests
    tests = self._GetTests()
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 440, in _GetTests
    test_lists = self._env.parallel_devices.pMap(list_tests).pGet(None)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/parallelizer.py", line 263, in pMap
    r.pFinish(None)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/parallelizer.py", line 135, in pFinish
    self._objs.JoinAll()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_environment.py", line 77, in wrapper
    on_failure(dev, f.__name__)
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_environment.py", line 277, in BlacklistDevice
    'All devices were blacklisted due to errors')

Project Member

Comment 11 by bugdroid1@chromium.org, Dec 21

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bd1da8c13f676843ae4ab9fd015803ba10b8a899

commit bd1da8c13f676843ae4ab9fd015803ba10b8a899
Author: Caleb Rouleau <crouleau@chromium.org>
Date: Fri Dec 21 23:31:20 2018

Disable angle_perftests on Nexus5.

It has been failing for several days.

To reenable, just re-add the lines I removed to perf_data_generator.py
and then rerun tools/perf/generate_perf_data
TBR=jbudorick@chromium.org

Bug:  916824 , 915429
Change-Id: I46bb0cd44b1ebf68ccbd6eae79a718c4cf49ee73
Reviewed-on: https://chromium-review.googlesource.com/c/1387832
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Cr-Commit-Position: refs/heads/master@{#618668}
[modify] https://crrev.com/bd1da8c13f676843ae4ab9fd015803ba10b8a899/testing/buildbot/chromium.perf.json
[modify] https://crrev.com/bd1da8c13f676843ae4ab9fd015803ba10b8a899/tools/perf/core/perf_data_generator.py

►

Issue 916824 link

Issue metadata

angle_perftests failing on chromium.perf/Android Nexus5 Perf

Issue description

Comment 1 by crouleau@chromium.org, Dec 20

Comment 1 Deleted

Comment 2 by crouleau@chromium.org, Dec 20

Comment 2 Deleted

Comment 3 by crouleau@chromium.org, Dec 20

Comment 3 Deleted

Comment 4 by crouleau@chromium.org, Dec 20

Comment 4 Deleted

Comment 5 by crouleau@chromium.org, Dec 20

Comment 5 Deleted

Comment 6 by jmadill@google.com, Dec 20

Comment 6 Deleted

Comment 7 by crouleau@chromium.org, Dec 20

Comment 7 Deleted

Comment 8 by jmadill@google.com, Dec 20

Comment 8 Deleted

Comment 9 by crouleau@chromium.org, Dec 20

Comment 9 Deleted

Comment 10 by crouleau@chromium.org, Dec 20

Comment 10 Deleted

Comment 11 by bugdroid1@chromium.org, Dec 21

Comment 11 Deleted

Sign in to add a comment