base_perftests failing on chromium.perf/android-nexus5x-perf |
|||||||||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of sullivan@google.com base_perftests failing on chromium.perf/android-nexus5x-perf Builders failed on: - android-nexus5x-perf: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-nexus5x-perf This seems to have been failing since build 946: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-nexus5x-perf/946 I can't find it running on builds before that. I don't see any logs for the test, and the perf dashboard does show some data for the test on that bot. Ned, Gab, Sami, any idea what's going on here?
,
Nov 26
Some of these tests are still failing now: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-nexus5x-perf looks like some tests are consistent failures and some are flaky failures logs from https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-nexus5x-perf/1063 C 258.197s Main ******************************************************************************** C 258.197s Main Detailed Logs C 258.197s Main ******************************************************************************** C 258.197s Main [UNKNOWN] ObserverListPerfTest/0.MAYBE_NotifyPerformance: C 258.197s Main [ RUN ] ObserverListPerfTest/0.MAYBE_NotifyPerformance C 258.197s Main ../../base/test/test_suite.cc:85: Failure C 258.197s Main Value of: IsMarkedMaybe(test_info) C 258.197s Main Actual: true C 258.197s Main Expected: false C 258.197s Main Probably the OS #ifdefs don't include all of the necessary platforms. C 258.197s Main Please ensure that no tests have the MAYBE_ prefix after the code is preprocessed. C 258.197s Main [ FAILED ] ObserverListPerfTest/0.MAYBE_NotifyPerformance, where TypeParam = <type> (0 ms) C 258.197s Main [----------] 1 test from ObserverListPerfTest/0 (41 ms total) C 258.197s Main C 258.197s Main [----------] 1 test from ObserverListPerfTest/1, where TypeParam = <type> C 258.197s Main C 258.197s Main [UNKNOWN] ObserverListPerfTest/1.MAYBE_NotifyPerformance: C 258.197s Main [ RUN ] ObserverListPerfTest/1.MAYBE_NotifyPerformance C 258.197s Main ../../base/test/test_suite.cc:85: Failure C 258.197s Main Value of: IsMarkedMaybe(test_info) C 258.197s Main Actual: true C 258.197s Main Expected: false C 258.197s Main Probably the OS #ifdefs don't include all of the necessary platforms. C 258.197s Main Please ensure that no tests have the MAYBE_ prefix after the code is preprocessed. C 258.197s Main [ FAILED ] ObserverListPerfTest/1.MAYBE_NotifyPerformance, where TypeParam = <type> (1 ms) C 258.197s Main [----------] 1 test from ObserverListPerfTest/1 (30 ms total) C 258.197s Main C 258.197s Main [----------] 7 tests from TaskSchedulerPerfTest C 258.197s Main C 258.198s Main [UNKNOWN] SequenceManagerPerfTest.PostDelayedTasks_OneQueue/0: C 258.198s Main [ RUN ] SequenceManagerPerfTest.PostDelayedTasks_OneQueue/0 C 258.198s Main ******************************************************************************** C 258.198s Main Summary C 258.198s Main ******************************************************************************** C 258.198s Main [==========] 160 tests ran. C 258.198s Main [ PASSED ] 157 tests. C 258.198s Main [ FAILED ] 3 tests, listed below: C 258.198s Main [ FAILED ] ObserverListPerfTest/0.MAYBE_NotifyPerformance (UNKNOWN) C 258.198s Main [ FAILED ] ObserverListPerfTest/1.MAYBE_NotifyPerformance (UNKNOWN) C 258.198s Main [ FAILED ] SequenceManagerPerfTest.PostDelayedTasks_OneQueue/0 (UNKNOWN) logs from https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-nexus5x-perf/1062 C 272.490s Main ******************************************************************************** C 272.490s Main Detailed Logs C 272.490s Main ******************************************************************************** C 272.490s Main [UNKNOWN] IntegratedPostTaskPerfTest.OneHundredTasksPerReload: C 272.490s Main [ RUN ] IntegratedPostTaskPerfTest.OneHundredTasksPerReload C 272.490s Main C 272.490s Main [UNKNOWN] ObserverListPerfTest/0.MAYBE_NotifyPerformance: C 272.490s Main [ RUN ] ObserverListPerfTest/0.MAYBE_NotifyPerformance C 272.490s Main ../../base/test/test_suite.cc:85: Failure C 272.490s Main Value of: IsMarkedMaybe(test_info) C 272.490s Main Actual: true C 272.491s Main Expected: false C 272.491s Main Probably the OS #ifdefs don't include all of the necessary platforms. C 272.491s Main Please ensure that no tests have the MAYBE_ prefix after the code is preprocessed. C 272.491s Main [ FAILED ] ObserverListPerfTest/0.MAYBE_NotifyPerformance, where TypeParam = <type> (0 ms) C 272.491s Main [----------] 1 test from ObserverListPerfTest/0 (23 ms total) C 272.491s Main C 272.491s Main [----------] 1 test from ObserverListPerfTest/1, where TypeParam = <type> C 272.491s Main C 272.491s Main [UNKNOWN] ObserverListPerfTest/1.MAYBE_NotifyPerformance: C 272.491s Main [ RUN ] ObserverListPerfTest/1.MAYBE_NotifyPerformance C 272.491s Main ../../base/test/test_suite.cc:85: Failure C 272.491s Main Value of: IsMarkedMaybe(test_info) C 272.491s Main Actual: true C 272.491s Main Expected: false C 272.491s Main Probably the OS #ifdefs don't include all of the necessary platforms. C 272.491s Main Please ensure that no tests have the MAYBE_ prefix after the code is preprocessed. C 272.491s Main [ FAILED ] ObserverListPerfTest/1.MAYBE_NotifyPerformance, where TypeParam = <type> (0 ms) C 272.491s Main [----------] 1 test from ObserverListPerfTest/1 (18 ms total) C 272.491s Main C 272.491s Main [----------] 7 tests from TaskSchedulerPerfTest C 272.491s Main ******************************************************************************** C 272.491s Main Summary C 272.491s Main ******************************************************************************** C 272.491s Main [==========] 160 tests ran. C 272.491s Main [ PASSED ] 157 tests. C 272.491s Main [ FAILED ] 3 tests, listed below: C 272.491s Main [ FAILED ] IntegratedPostTaskPerfTest.OneHundredTasksPerReload (UNKNOWN) C 272.491s Main [ FAILED ] ObserverListPerfTest/0.MAYBE_NotifyPerformance (UNKNOWN) C 272.491s Main [ FAILED ] ObserverListPerfTest/1.MAYBE_NotifyPerformance (UNKNOWN) C 272.491s Main C 272.491s Main 3 FAILED TESTS C 272.492s Main ********************************************************************************
,
Nov 26
,
Nov 27
,
Nov 27
r610618 doesn't work since TYPED_TEST(ObserverListPerfTest, MAYBE_NotifyPerformance) { is preprocessed to something_MAYBE_NotifyPerformance So then the directive #define MAYBE_NotifyPerformance DISABLED_NotifyPerformance is not applied.
,
Nov 27
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/158390508c619b9c04a79ec3868be57eff702d12 commit 158390508c619b9c04a79ec3868be57eff702d12 Author: Caleb Rouleau <crouleau@chromium.org> Date: Tue Nov 27 23:19:05 2018 [go/perfbot-sheriff][base_perftests] Properly disable NotifyPerformance This change should be committed right away per go/perfbot-sheriff since this is failing on the waterfall. r610618 didn't work since TYPED_TEST(ObserverListPerfTest, MAYBE_NotifyPerformance) { is preprocessed to something_MAYBE_NotifyPerformance So then the directive #define MAYBE_NotifyPerformance DISABLED_NotifyPerformance is not applied. This will fix both 908490 and 906686. Bug: 906686 , 908490 Change-Id: Ib2adc016fbef79bb889a076e452534f893fb09b5 Reviewed-on: https://chromium-review.googlesource.com/c/1352481 Reviewed-by: Lei Zhang <thestig@chromium.org> Reviewed-by: John Rummell <jrummell@chromium.org> Commit-Queue: Caleb Rouleau <crouleau@chromium.org> Cr-Commit-Position: refs/heads/master@{#611397} [modify] https://crrev.com/158390508c619b9c04a79ec3868be57eff702d12/base/observer_list_perftest.cc
,
Nov 28
Thanks for investigating and fixing this! I completely overlooked the fact that this was a TYPED_TEST. Both Mac and Android bots look happy with this test now.
,
Nov 28
Actually SequenceManagerPerfTest seems to be still failing -- let me look into it.
,
Nov 28
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/151a6653e40000
,
Nov 28
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/151a6653e40000 All of the runs failed. The most common error (1/20 runs) was: IOError: [Errno 2] No such file or directory: '/b/swarming/w/it1CcM9Q/tmpw2wjFgtelemetry/histograms.json'
,
Nov 28
It seems like the failure is a different test each time: SequenceManagerPerfTest.PostDelayedTasks_FourQueues MessageLoopPerfTest.PostTaskRate but the stack trace is the same each time https://logs.chromium.org/logs/chrome/buildbucket/cr-buildbucket.appspot.com/8928617438908335456/+/steps/base_perftests_on_Android_device_Nexus_5X/0/stdout https://logs.chromium.org/logs/chrome/buildbucket/cr-buildbucket.appspot.com/8928602146646224288/+/steps/base_perftests_on_Android_device_Nexus_5X/0/stdout E 171.262s run_tests_on_device(020ad055c25b1129) gtest shard timed out. Traceback (most recent call last): File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 190, in Run self._component, extras=extras, raw=False, **kwargs) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run error_log_func=error_log_func) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll self._JoinAll(watcher, timeout) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll thread.ReraiseIfException() File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1340, in StartInstrumentation large_output=True) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1159, in RunShellCommand output = handle_large_output(cmd, large_output) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1124, in handle_large_output handle_large_command(cmd) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1116, in handle_large_command return handle_check_return('sh %s' % script.name_quoted) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1101, in handle_check_return return run(cmd) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1097, in run return self.adb.Shell(cmd) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 533, in Shell output = self._RunDeviceAdbCmd(args, timeout, retries, check_error=False) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd check_error=check_error) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 277, in _RunAdbCmd timeout, env=cls._ADB_ENV) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/cmd_helper.py", line 420, in GetCmdStatusAndOutputWithTimeout raise TimeoutError(output.getvalue()) CommandTimeoutError: Timeout So I think therefore that this is a bug in devil https://github.com/catapult-project/catapult/tree/master/devil John, what team owns Devil right now? Is it CCI team or just you personally?
,
Nov 28
Juan and I are the primary owners at this point, though we're growing Ben as an owner too. I'm hesitant to say that CCI owns it. I'm not sure I agree with your analysis, but I'll look into it more.
,
Nov 28
,
Nov 28
My analysis could definitely be wrong :) . Thanks for looking into it though!
,
Nov 29
I've got a patch here to increase the timeout: https://chromium-review.googlesource.com/c/chromium/src/+/1353922
,
Nov 29
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/21bf578c37e818d29bd788bf720606f7069cefd3 commit 21bf578c37e818d29bd788bf720606f7069cefd3 Author: Sami Kyostila <skyostil@chromium.org> Date: Thu Nov 29 16:41:48 2018 base_perftests: Increase shard timeout The base perftests can take longer than the default 120 seconds to execute -- especially on low end Android devices. Increase the timeout on Android to 10 minutes. Bug: 906686 Change-Id: Iacfe08e3cbd1d6e7ef756a1a8c1028655a5d3918 TBR=fdoray@chromium.org Change-Id: Iacfe08e3cbd1d6e7ef756a1a8c1028655a5d3918 Reviewed-on: https://chromium-review.googlesource.com/c/1353922 Reviewed-by: Sami Kyöstilä <skyostil@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Sami Kyöstilä <skyostil@chromium.org> Cr-Commit-Position: refs/heads/master@{#612228} [modify] https://crrev.com/21bf578c37e818d29bd788bf720606f7069cefd3/base/BUILD.gn
,
Nov 30
Timeout tweak seems to have worked!
,
Dec 14
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c61e1a789bd71a1bf03432218ebc127d9e252a09 commit c61e1a789bd71a1bf03432218ebc127d9e252a09 Author: Sami Kyostila <skyostil@chromium.org> Date: Fri Dec 14 11:21:39 2018 Revert "base: Disable some perf tests that are failing on Android" This reverts commit 15a28138fef95c32556efa516aae023dfc6a47ed. Reason for revert: The test timeout has been increased with https://chromium-review.googlesource.com/c/1353922, so let's see if these pass reliably now. Original change's description: > base: Disable some perf tests that are failing on Android > > The following perf tests are often timing out on Android: > > - JSONPerfTest.StressTest > - ObserverListPerfTest/1.NotifyPerformance > - ScheduleWorkTest.* > > This patch disables them on Android until they are be investigated. > > TBR=gab@chromium.org > > Bug: 906686 > Change-Id: Iad78d71e1b3293a75834b06a2b7b4b13b7ac5e75 > Reviewed-on: https://chromium-review.googlesource.com/c/1349711 > Reviewed-by: Eric Seckler <eseckler@chromium.org> > Commit-Queue: Sami Kyöstilä <skyostil@chromium.org> > Cr-Commit-Position: refs/heads/master@{#610618} TBR=gab@chromium.org,skyostil@chromium.org,perezju@chromium.org,eseckler@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. Bug: 906686 Change-Id: I23a585253860366456fe7d8b7a27d5cd949840d9 Reviewed-on: https://chromium-review.googlesource.com/c/1377851 Reviewed-by: Sami Kyöstilä <skyostil@chromium.org> Reviewed-by: Eric Seckler <eseckler@chromium.org> Commit-Queue: Sami Kyöstilä <skyostil@chromium.org> Cr-Commit-Position: refs/heads/master@{#616635} [modify] https://crrev.com/c61e1a789bd71a1bf03432218ebc127d9e252a09/base/json/json_perftest.cc [modify] https://crrev.com/c61e1a789bd71a1bf03432218ebc127d9e252a09/base/message_loop/message_pump_perftest.cc [modify] https://crrev.com/c61e1a789bd71a1bf03432218ebc127d9e252a09/base/observer_list_perftest.cc
,
Dec 14
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/dacd59c9179d75f89a27be539dde8892984612bc commit dacd59c9179d75f89a27be539dde8892984612bc Author: Sami Kyostila <skyostil@chromium.org> Date: Fri Dec 14 18:32:52 2018 base: Disable JSONPerfTest.StressTest Disable JSONPerfTest.StressTest as it is still timing out even after increasing the timeout. TBR=gab@chromium.org Bug: 906686 Change-Id: I67434e2405f86fd7f0fe9d82a901e618ac0b2b9e Reviewed-on: https://chromium-review.googlesource.com/c/1378144 Reviewed-by: Sami Kyöstilä <skyostil@chromium.org> Commit-Queue: Sami Kyöstilä <skyostil@chromium.org> Cr-Commit-Position: refs/heads/master@{#616756} [modify] https://crrev.com/dacd59c9179d75f89a27be539dde8892984612bc/base/json/json_perftest.cc |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by bugdroid1@chromium.org
, Nov 23