cc_perftests failing on chromium.perf/Android Nexus7v2 Perf (2) |
||||||
Issue descriptioncc_perftests failing on chromium.perf/Android Nexus7v2 Perf (2) Builders failed on: - Android Nexus7v2 Perf (2): https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus7v2%20Perf%20%282%29 Looks like it's failing with a devil issue? John, can you help triage? I 215.432s run_tests_on_device(0585ea58) <<ScopedMainEntryLogger I 260.587s run_tests_on_device(0585c0cd) Still working on StartInstrumentation(0585c0cd, org.chromium.native_test/org.chromium.native_test.NativeTestInstrumentationTestRunner, retries=0, extras={'org.chromium.native_test.NativeTestInstrumentationTestRunner.ShardNanoTimeout': 120000000000, 'org.chromium.native_test.NativeTestInstrumentationTestRunner.NativeTestActivity': 'org.chromium.native_test.NativeUnitTestActivity', 'org.chromium.native_test.NativeTestInstrumentationTestRunner.Test': 'CCSerializationPerfTest.DelegatedFrame_ManyQuads_10_100000', 'org.chromium.native_test.NativeTestInstrumentationTestRunner.StdoutFile': '/storage/emulated/legacy/temp_file-2eed043736c1e.gtest_out'}, timeout=120.0, raw=False) C 320.562s run_tests_on_device(0585c0cd) Timed out. Dumping threads. C 320.562s run_tests_on_device(0585c0cd) ******************************************************************************** C 320.562s run_tests_on_device(0585c0cd) Stack dump for thread 'TimeoutThread-1-for-run_tests_on_device(0585c0cd)' C 320.562s run_tests_on_device(0585c0cd) ******************************************************************************** C 320.563s run_tests_on_device(0585c0cd) File: "/usr/lib/python2.7/threading.py", line 783, in __bootstrap C 320.563s run_tests_on_device(0585c0cd) self.__bootstrap_inner() C 320.563s run_tests_on_device(0585c0cd) File: "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner C 320.563s run_tests_on_device(0585c0cd) self.run() C 320.563s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run C 320.563s run_tests_on_device(0585c0cd) self._ret = self._func(*self._args, **self._kwargs) C 320.563s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 152, in <lambda> C 320.563s run_tests_on_device(0585c0cd) child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), C 320.563s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl C 320.563s run_tests_on_device(0585c0cd) return f(*args, **kwargs) C 320.563s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/device_utils.py", line 1077, in StartInstrumentation C 320.563s run_tests_on_device(0585c0cd) large_output=True) C 320.564s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper C 320.564s run_tests_on_device(0585c0cd) return impl() C 320.564s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl C 320.564s run_tests_on_device(0585c0cd) return f(*args, **kwargs) C 320.564s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/device_utils.py", line 923, in RunShellCommand C 320.564s run_tests_on_device(0585c0cd) output = handle_large_output(cmd, large_output) C 320.564s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/device_utils.py", line 894, in handle_large_output C 320.564s run_tests_on_device(0585c0cd) handle_large_command(cmd) C 320.564s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/device_utils.py", line 886, in handle_large_command C 320.564s run_tests_on_device(0585c0cd) return handle_check_return('sh %s' % script.name_quoted) C 320.564s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/device_utils.py", line 871, in handle_check_return C 320.564s run_tests_on_device(0585c0cd) return run(cmd) C 320.564s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/device_utils.py", line 867, in run C 320.564s run_tests_on_device(0585c0cd) return self.adb.Shell(cmd) C 320.564s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 480, in Shell C 320.564s run_tests_on_device(0585c0cd) output = self._RunDeviceAdbCmd(args, timeout, retries, check_error=False) C 320.564s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 282, in _RunDeviceAdbCmd C 320.565s run_tests_on_device(0585c0cd) check_error=check_error) C 320.565s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper C 320.565s run_tests_on_device(0585c0cd) return impl() C 320.565s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl C 320.565s run_tests_on_device(0585c0cd) return f(*args, **kwargs) C 320.565s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 249, in _RunAdbCmd C 320.565s run_tests_on_device(0585c0cd) timeout_retry.CurrentTimeoutThreadGroup().GetRemainingTime()) C 320.565s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/utils/cmd_helper.py", line 273, in GetCmdStatusAndOutputWithTimeout C 320.565s run_tests_on_device(0585c0cd) for data in _IterProcessStdout(process, timeout=timeout): C 320.565s run_tests_on_device(0585c0cd) File: "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/utils/cmd_helper.py", line 229, in _IterProcessStdout C 320.565s run_tests_on_device(0585c0cd) read_fds, _, _ = select.select([child_fd], [], [], poll_interval) C 320.565s run_tests_on_device(0585c0cd) ******************************************************************************** E 320.565s run_tests_on_device(0585c0cd) gtest shard timed out. Traceback (most recent call last): File "/b/rr/tmpztKUe9/w/src/build/android/pylib/local/device/local_device_gtest_run.py", line 169, in Run self._component, extras=extras, raw=False, **kwargs) File "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 159, in Run error_log_func=error_log_func) File "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll self._JoinAll(watcher, timeout) File "/b/rr/tmpztKUe9/w/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 151, in _JoinAll (len(alive_threads), len(self._threads))) CommandTimeoutError: Timed out waiting for 1 of 1 threads. I 320.566s TimeoutThread-1-for-run_tests_on_device(0585c0cd) [host]> /b/rr/tmpztKUe9/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0585c0cd shell '( TZ=utc ls -a -l /storage/emulated/legacy/ );echo %$?' I 320.684s TimeoutThread-1-for-run_tests_on_device(0585c0cd) [host]> /b/rr/tmpztKUe9/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0585c0cd shell '( cat /storage/emulated/legacy/temp_file-2eed043736c1e.gtest_out );echo %$?' I 320.775s TimeoutThread-1-for-delete_temporary_file(0585c0cd) [host]> /b/rr/tmpztKUe9/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0585c0cd shell 'rm -f /storage/emulated/legacy/temp_file-2eed043736c1e.gtest_out' I 320.778s TimeoutThread-1-for-run_tests_on_device(0585c0cd) [host]> /b/rr/tmpztKUe9/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0585c0cd shell '( pm clear org.chromium.native_test );echo %$?' I 320.780s TimeoutThread-1-for-delete_temporary_file(0585c0cd) [host]> /b/rr/tmpztKUe9/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0585c0cd shell 'rm -f /data/data/org.chromium.native_test/temp_file-2cd37fa649e60.xml' I 320.944s TimeoutThread-1-for-delete_temporary_file(0585c0cd) [host]> /b/rr/tmpztKUe9/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0585c0cd shell 'rm -f /data/local/tmp/temp_file-768e7ad5d74be.sh' I 320.945s TimeoutThread-1-for-delete_temporary_file(0585c0cd) [host]> /b/rr/tmpztKUe9/w/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 0585c0cd shell 'rm -f /data/local/tmp/temp_file-f3e131123c596' I 321.743s run_tests_on_device(0585c0cd) >>ScopedMainEntryLogger I 321.743s run_tests_on_device(0585c0cd) Note: Google Test filter = CCSerializationPerfTest.DelegatedFrame_ManyQuads_10_100000 I 321.743s run_tests_on_device(0585c0cd) [==========] Running 1 test from 1 test case. I 321.744s run_tests_on_device(0585c0cd) [----------] Global test environment set-up. I 321.744s run_tests_on_device(0585c0cd) [----------] 1 test from CCSerializationPerfTest I 321.744s run_tests_on_device(0585c0cd) [ RUN ] CCSerializationPerfTest.DelegatedFrame_ManyQuads_10_100000 I 321.744s run_tests_on_device(0585c0cd) *RESULT StructTraits serialization min_frame_serialization_time_per_render_pass_shared_quad_state: DelegatedFrame_ManyQuads_10_100000= 1979119.9000000001 us I 321.744s run_tests_on_device(0585c0cd) *RESULT StructTraits serialization: num runs in 2 seconds_per_render_pass_shared_quad_state: DelegatedFrame_ManyQuads_10_100000= 1 I 321.744s run_tests_on_device(0585c0cd) *RESULT StructTraits deserialization min_frame_deserialization_time_per_render_pass_shared_quad_state: DelegatedFrame_ManyQuads_10_100000= 1401702.9 us I 321.744s run_tests_on_device(0585c0cd) *RESULT StructTraits deserialization: num runs in 2 seconds_per_render_pass_shared_quad_state: DelegatedFrame_ManyQuads_10_100000= 1 I 321.744s run_tests_on_device(0585ea58) Finished running tests on this device. I 321.745s run_tests_on_device(0585c0cd) Finished running tests on this device. I 321.745s run_tests_on_device(06d8a77c) Finished running tests on this device. I 321.745s run_tests_on_device(07a1ae29) Finished running tests on this device. I 321.745s run_tests_on_device(059959cd) Finished running tests on this device. I 321.746s run_tests_on_device(07086a3b) Finished running tests on this device. I 321.746s run_tests_on_device(07a04273) Finished running tests on this device. I 321.748s Main FINISHED TRY #2/3
,
Feb 13 2017
Thanks for letting me know. Is it possible to disable this test for android perf bots for now?
,
Feb 13 2017
as it's a gtest, you should be able to disable it with something like this: https://codesearch.chromium.org/chromium/src/ipc/ipc_fuzzing_tests.cc?rcl=1ffa882da6a9795878cbdd6bba4542d0158298a9&l=97
,
Feb 13 2017
martiniss, can you disable?
,
Feb 13 2017
I could do what jbudorick suggested in #3. I'm not a chromium src committer though. But glad to do it. I can disable the entire test suite on the android perf bots, as well. I wouldn't think we'd want to do that...?
,
Feb 13 2017
Your thinking is correct. Let's stick with just disabling the test. Not being a committer should be fine for now, though we should look at changing that separately if you've got enough src/-side CLs.
,
Feb 13 2017
If you need someone to LGTM the CL I can do that.
,
Feb 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0dbe48ffd68565abfb1eb87dc922fe6fcc47b35d commit 0dbe48ffd68565abfb1eb87dc922fe6fcc47b35d Author: martiniss <martiniss@chromium.org> Date: Mon Feb 13 22:49:54 2017 Disable test in cc_perftest Disable DelegatedFrame_ManyQuads_10_100000 on android. The test is taking too long. BUG=691730 CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_trusty_blink_rel Review-Url: https://codereview.chromium.org/2691943003 Cr-Commit-Position: refs/heads/master@{#450130} [modify] https://crrev.com/0dbe48ffd68565abfb1eb87dc922fe6fcc47b35d/cc/ipc/cc_serialization_perftest.cc
,
Feb 14 2017
Ok, test disabled. Downgrading priority.
,
Dec 4 2017
Moving Infra>Client>Android -> Infra>Client>Chrome+OS=Android
,
Dec 4 2017
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by jbudorick@chromium.org
, Feb 13 2017Owner: weiliangc@chromium.org
Status: Assigned (was: Available)