New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Nov 30
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 1
Type: ----



Sign in to add a comment
link

Issue 906686: base_perftests failing on chromium.perf/android-nexus5x-perf

Reported by sheriff-...@appspot.gserviceaccount.com, Nov 19 Project Member

Issue description

Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of sullivan@google.com

base_perftests failing on chromium.perf/android-nexus5x-perf

Builders failed on: 
- android-nexus5x-perf: 
  https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-nexus5x-perf

This seems to have been failing since build 946: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-nexus5x-perf/946

I can't find it running on builds before that. I don't see any logs for the test, and the perf dashboard does show some data for the test on that bot. Ned, Gab, Sami, any idea what's going on here?
 

Comment 1 by bugdroid1@chromium.org, Nov 23

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/15a28138fef95c32556efa516aae023dfc6a47ed

commit 15a28138fef95c32556efa516aae023dfc6a47ed
Author: Sami Kyostila <skyostil@chromium.org>
Date: Fri Nov 23 15:59:38 2018

base: Disable some perf tests that are failing on Android

The following perf tests are often timing out on Android:

- JSONPerfTest.StressTest
- ObserverListPerfTest/1.NotifyPerformance
- ScheduleWorkTest.*

This patch disables them on Android until they are be investigated.

TBR=gab@chromium.org

Bug:  906686 
Change-Id: Iad78d71e1b3293a75834b06a2b7b4b13b7ac5e75
Reviewed-on: https://chromium-review.googlesource.com/c/1349711
Reviewed-by: Eric Seckler <eseckler@chromium.org>
Commit-Queue: Sami Kyöstilä <skyostil@chromium.org>
Cr-Commit-Position: refs/heads/master@{#610618}
[modify] https://crrev.com/15a28138fef95c32556efa516aae023dfc6a47ed/base/json/json_perftest.cc
[modify] https://crrev.com/15a28138fef95c32556efa516aae023dfc6a47ed/base/message_loop/message_pump_perftest.cc
[modify] https://crrev.com/15a28138fef95c32556efa516aae023dfc6a47ed/base/observer_list_perftest.cc

Comment 2 by crouleau@google.com, Nov 26

Owner: skyos...@chromium.org
Status: Assigned (was: Available)
Some of these tests are still failing now:
https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-nexus5x-perf

looks like some tests are consistent failures and some are flaky failures


logs from https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-nexus5x-perf/1063


C  258.197s Main  ********************************************************************************
C  258.197s Main  Detailed Logs
C  258.197s Main  ********************************************************************************
C  258.197s Main  [UNKNOWN] ObserverListPerfTest/0.MAYBE_NotifyPerformance:
C  258.197s Main  [ RUN      ] ObserverListPerfTest/0.MAYBE_NotifyPerformance
C  258.197s Main  ../../base/test/test_suite.cc:85: Failure
C  258.197s Main  Value of: IsMarkedMaybe(test_info)
C  258.197s Main    Actual: true
C  258.197s Main  Expected: false
C  258.197s Main  Probably the OS #ifdefs don't include all of the necessary platforms.
C  258.197s Main  Please ensure that no tests have the MAYBE_ prefix after the code is preprocessed.
C  258.197s Main  [  FAILED  ] ObserverListPerfTest/0.MAYBE_NotifyPerformance, where TypeParam = <type> (0 ms)
C  258.197s Main  [----------] 1 test from ObserverListPerfTest/0 (41 ms total)
C  258.197s Main  
C  258.197s Main  [----------] 1 test from ObserverListPerfTest/1, where TypeParam = <type>
C  258.197s Main  
C  258.197s Main  [UNKNOWN] ObserverListPerfTest/1.MAYBE_NotifyPerformance:
C  258.197s Main  [ RUN      ] ObserverListPerfTest/1.MAYBE_NotifyPerformance
C  258.197s Main  ../../base/test/test_suite.cc:85: Failure
C  258.197s Main  Value of: IsMarkedMaybe(test_info)
C  258.197s Main    Actual: true
C  258.197s Main  Expected: false
C  258.197s Main  Probably the OS #ifdefs don't include all of the necessary platforms.
C  258.197s Main  Please ensure that no tests have the MAYBE_ prefix after the code is preprocessed.
C  258.197s Main  [  FAILED  ] ObserverListPerfTest/1.MAYBE_NotifyPerformance, where TypeParam = <type> (1 ms)
C  258.197s Main  [----------] 1 test from ObserverListPerfTest/1 (30 ms total)
C  258.197s Main  
C  258.197s Main  [----------] 7 tests from TaskSchedulerPerfTest
C  258.197s Main  
C  258.198s Main  [UNKNOWN] SequenceManagerPerfTest.PostDelayedTasks_OneQueue/0:
C  258.198s Main  [ RUN      ] SequenceManagerPerfTest.PostDelayedTasks_OneQueue/0
C  258.198s Main  ********************************************************************************
C  258.198s Main  Summary
C  258.198s Main  ********************************************************************************
C  258.198s Main  [==========] 160 tests ran.
C  258.198s Main  [  PASSED  ] 157 tests.
C  258.198s Main  [  FAILED  ] 3 tests, listed below:
C  258.198s Main  [  FAILED  ] ObserverListPerfTest/0.MAYBE_NotifyPerformance (UNKNOWN)
C  258.198s Main  [  FAILED  ] ObserverListPerfTest/1.MAYBE_NotifyPerformance (UNKNOWN)
C  258.198s Main  [  FAILED  ] SequenceManagerPerfTest.PostDelayedTasks_OneQueue/0 (UNKNOWN)




logs from https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-nexus5x-perf/1062

C  272.490s Main  ********************************************************************************
C  272.490s Main  Detailed Logs
C  272.490s Main  ********************************************************************************
C  272.490s Main  [UNKNOWN] IntegratedPostTaskPerfTest.OneHundredTasksPerReload:
C  272.490s Main  [ RUN      ] IntegratedPostTaskPerfTest.OneHundredTasksPerReload
C  272.490s Main  
C  272.490s Main  [UNKNOWN] ObserverListPerfTest/0.MAYBE_NotifyPerformance:
C  272.490s Main  [ RUN      ] ObserverListPerfTest/0.MAYBE_NotifyPerformance
C  272.490s Main  ../../base/test/test_suite.cc:85: Failure
C  272.490s Main  Value of: IsMarkedMaybe(test_info)
C  272.490s Main    Actual: true
C  272.491s Main  Expected: false
C  272.491s Main  Probably the OS #ifdefs don't include all of the necessary platforms.
C  272.491s Main  Please ensure that no tests have the MAYBE_ prefix after the code is preprocessed.
C  272.491s Main  [  FAILED  ] ObserverListPerfTest/0.MAYBE_NotifyPerformance, where TypeParam = <type> (0 ms)
C  272.491s Main  [----------] 1 test from ObserverListPerfTest/0 (23 ms total)
C  272.491s Main  
C  272.491s Main  [----------] 1 test from ObserverListPerfTest/1, where TypeParam = <type>
C  272.491s Main  
C  272.491s Main  [UNKNOWN] ObserverListPerfTest/1.MAYBE_NotifyPerformance:
C  272.491s Main  [ RUN      ] ObserverListPerfTest/1.MAYBE_NotifyPerformance
C  272.491s Main  ../../base/test/test_suite.cc:85: Failure
C  272.491s Main  Value of: IsMarkedMaybe(test_info)
C  272.491s Main    Actual: true
C  272.491s Main  Expected: false
C  272.491s Main  Probably the OS #ifdefs don't include all of the necessary platforms.
C  272.491s Main  Please ensure that no tests have the MAYBE_ prefix after the code is preprocessed.
C  272.491s Main  [  FAILED  ] ObserverListPerfTest/1.MAYBE_NotifyPerformance, where TypeParam = <type> (0 ms)
C  272.491s Main  [----------] 1 test from ObserverListPerfTest/1 (18 ms total)
C  272.491s Main  
C  272.491s Main  [----------] 7 tests from TaskSchedulerPerfTest
C  272.491s Main  ********************************************************************************
C  272.491s Main  Summary
C  272.491s Main  ********************************************************************************
C  272.491s Main  [==========] 160 tests ran.
C  272.491s Main  [  PASSED  ] 157 tests.
C  272.491s Main  [  FAILED  ] 3 tests, listed below:
C  272.491s Main  [  FAILED  ] IntegratedPostTaskPerfTest.OneHundredTasksPerReload (UNKNOWN)
C  272.491s Main  [  FAILED  ] ObserverListPerfTest/0.MAYBE_NotifyPerformance (UNKNOWN)
C  272.491s Main  [  FAILED  ] ObserverListPerfTest/1.MAYBE_NotifyPerformance (UNKNOWN)
C  272.491s Main  
C  272.491s Main  3 FAILED TESTS
C  272.492s Main  ********************************************************************************

Comment 3 by crouleau@google.com, Nov 26

Labels: OS-Android

Comment 4 by crouleau@google.com, Nov 27

Labels: -Pri-2 Pri-1

Comment 5 by crouleau@chromium.org, Nov 27

r610618 doesn't work since

TYPED_TEST(ObserverListPerfTest, MAYBE_NotifyPerformance) {

is preprocessed to 

something_MAYBE_NotifyPerformance

So then the directive 

#define MAYBE_NotifyPerformance DISABLED_NotifyPerformance

is not applied.

Comment 6 by bugdroid1@chromium.org, Nov 27

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/158390508c619b9c04a79ec3868be57eff702d12

commit 158390508c619b9c04a79ec3868be57eff702d12
Author: Caleb Rouleau <crouleau@chromium.org>
Date: Tue Nov 27 23:19:05 2018

[go/perfbot-sheriff][base_perftests] Properly disable NotifyPerformance

This change should be committed right away per go/perfbot-sheriff
since this is failing on the waterfall.

r610618 didn't work since
TYPED_TEST(ObserverListPerfTest, MAYBE_NotifyPerformance) {
is preprocessed to
something_MAYBE_NotifyPerformance
So then the directive
#define MAYBE_NotifyPerformance DISABLED_NotifyPerformance
is not applied.

This will fix both 908490 and 906686.

Bug:  906686 , 908490 
Change-Id: Ib2adc016fbef79bb889a076e452534f893fb09b5
Reviewed-on: https://chromium-review.googlesource.com/c/1352481
Reviewed-by: Lei Zhang <thestig@chromium.org>
Reviewed-by: John Rummell <jrummell@chromium.org>
Commit-Queue: Caleb Rouleau <crouleau@chromium.org>
Cr-Commit-Position: refs/heads/master@{#611397}
[modify] https://crrev.com/158390508c619b9c04a79ec3868be57eff702d12/base/observer_list_perftest.cc

Comment 7 by skyostil@google.com, Nov 28

Status: Fixed (was: Assigned)
Thanks for investigating and fixing this! I completely overlooked the fact that this was a TYPED_TEST. Both Mac and Android bots look happy with this test now.

Comment 8 by skyostil@google.com, Nov 28

Status: Started (was: Fixed)
Actually SequenceManagerPerfTest seems to be still failing -- let me look into it.

Comment 10 by 42576172...@developer.gserviceaccount.com, Nov 28

Project Member
😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/151a6653e40000

All of the runs failed. The most common error (1/20 runs) was:
IOError: [Errno 2] No such file or directory: '/b/swarming/w/it1CcM9Q/tmpw2wjFgtelemetry/histograms.json'

Comment 11 by crouleau@chromium.org, Nov 28

Cc: crouleau@chromium.org
Components: Infra>Client>Chrome
Owner: jbudorick@chromium.org
Status: Assigned (was: Started)
It seems like the failure is a different test each time:

SequenceManagerPerfTest.PostDelayedTasks_FourQueues
MessageLoopPerfTest.PostTaskRate

but the stack trace is the same each time https://logs.chromium.org/logs/chrome/buildbucket/cr-buildbucket.appspot.com/8928617438908335456/+/steps/base_perftests_on_Android_device_Nexus_5X/0/stdout
https://logs.chromium.org/logs/chrome/buildbucket/cr-buildbucket.appspot.com/8928602146646224288/+/steps/base_perftests_on_Android_device_Nexus_5X/0/stdout



E  171.262s run_tests_on_device(020ad055c25b1129)  gtest shard timed out.
Traceback (most recent call last):
  File "/b/swarming/w/ir/build/android/pylib/local/device/local_device_gtest_run.py", line 190, in Run
    self._component, extras=extras, raw=False, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run
    error_log_func=error_log_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1340, in StartInstrumentation
    large_output=True)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1159, in RunShellCommand
    output = handle_large_output(cmd, large_output)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1124, in handle_large_output
    handle_large_command(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1116, in handle_large_command
    return handle_check_return('sh %s' % script.name_quoted)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1101, in handle_check_return
    return run(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1097, in run
    return self.adb.Shell(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 533, in Shell
    output = self._RunDeviceAdbCmd(args, timeout, retries, check_error=False)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd
    check_error=check_error)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 277, in _RunAdbCmd
    timeout, env=cls._ADB_ENV)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/cmd_helper.py", line 420, in GetCmdStatusAndOutputWithTimeout
    raise TimeoutError(output.getvalue())
CommandTimeoutError: Timeout


So I think therefore that this is a bug in devil https://github.com/catapult-project/catapult/tree/master/devil

John, what team owns Devil right now? Is it CCI team or just you personally?

Comment 12 by jbudorick@chromium.org, Nov 28

Juan and I are the primary owners at this point, though we're growing Ben as an owner too. I'm hesitant to say that CCI owns it.

I'm not sure I agree with your analysis, but I'll look into it more.

Comment 13 by jbudorick@chromium.org, Nov 28

Cc: perezju@chromium.org

Comment 14 by crouleau@chromium.org, Nov 28

My analysis could definitely be wrong :) . Thanks for looking into it though!

Comment 15 by skyostil@google.com, Nov 29

I've got a patch here to increase the timeout: https://chromium-review.googlesource.com/c/chromium/src/+/1353922

Comment 16 by bugdroid1@chromium.org, Nov 29

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/21bf578c37e818d29bd788bf720606f7069cefd3

commit 21bf578c37e818d29bd788bf720606f7069cefd3
Author: Sami Kyostila <skyostil@chromium.org>
Date: Thu Nov 29 16:41:48 2018

base_perftests: Increase shard timeout

The base perftests can take longer than the default 120 seconds to
execute -- especially on low end Android devices. Increase the timeout
on Android to 10 minutes.

Bug:  906686 
Change-Id: Iacfe08e3cbd1d6e7ef756a1a8c1028655a5d3918

TBR=fdoray@chromium.org

Change-Id: Iacfe08e3cbd1d6e7ef756a1a8c1028655a5d3918
Reviewed-on: https://chromium-review.googlesource.com/c/1353922
Reviewed-by: Sami Kyöstilä <skyostil@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Sami Kyöstilä <skyostil@chromium.org>
Cr-Commit-Position: refs/heads/master@{#612228}
[modify] https://crrev.com/21bf578c37e818d29bd788bf720606f7069cefd3/base/BUILD.gn

Comment 17 by skyostil@google.com, Nov 30

Status: Fixed (was: Assigned)
Timeout tweak seems to have worked!

Comment 18 by bugdroid1@chromium.org, Dec 14

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c61e1a789bd71a1bf03432218ebc127d9e252a09

commit c61e1a789bd71a1bf03432218ebc127d9e252a09
Author: Sami Kyostila <skyostil@chromium.org>
Date: Fri Dec 14 11:21:39 2018

Revert "base: Disable some perf tests that are failing on Android"

This reverts commit 15a28138fef95c32556efa516aae023dfc6a47ed.

Reason for revert: The test timeout has been increased with https://chromium-review.googlesource.com/c/1353922, so let's see if these pass reliably now.

Original change's description:
> base: Disable some perf tests that are failing on Android
> 
> The following perf tests are often timing out on Android:
> 
> - JSONPerfTest.StressTest
> - ObserverListPerfTest/1.NotifyPerformance
> - ScheduleWorkTest.*
> 
> This patch disables them on Android until they are be investigated.
> 
> TBR=gab@chromium.org
> 
> Bug:  906686 
> Change-Id: Iad78d71e1b3293a75834b06a2b7b4b13b7ac5e75
> Reviewed-on: https://chromium-review.googlesource.com/c/1349711
> Reviewed-by: Eric Seckler <eseckler@chromium.org>
> Commit-Queue: Sami Kyöstilä <skyostil@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#610618}

TBR=gab@chromium.org,skyostil@chromium.org,perezju@chromium.org,eseckler@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  906686 
Change-Id: I23a585253860366456fe7d8b7a27d5cd949840d9
Reviewed-on: https://chromium-review.googlesource.com/c/1377851
Reviewed-by: Sami Kyöstilä <skyostil@chromium.org>
Reviewed-by: Eric Seckler <eseckler@chromium.org>
Commit-Queue: Sami Kyöstilä <skyostil@chromium.org>
Cr-Commit-Position: refs/heads/master@{#616635}
[modify] https://crrev.com/c61e1a789bd71a1bf03432218ebc127d9e252a09/base/json/json_perftest.cc
[modify] https://crrev.com/c61e1a789bd71a1bf03432218ebc127d9e252a09/base/message_loop/message_pump_perftest.cc
[modify] https://crrev.com/c61e1a789bd71a1bf03432218ebc127d9e252a09/base/observer_list_perftest.cc

Comment 19 by bugdroid1@chromium.org, Dec 14

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dacd59c9179d75f89a27be539dde8892984612bc

commit dacd59c9179d75f89a27be539dde8892984612bc
Author: Sami Kyostila <skyostil@chromium.org>
Date: Fri Dec 14 18:32:52 2018

base: Disable JSONPerfTest.StressTest

Disable JSONPerfTest.StressTest as it is still timing out even after
increasing the timeout.

TBR=gab@chromium.org

Bug:  906686 
Change-Id: I67434e2405f86fd7f0fe9d82a901e618ac0b2b9e
Reviewed-on: https://chromium-review.googlesource.com/c/1378144
Reviewed-by: Sami Kyöstilä <skyostil@chromium.org>
Commit-Queue: Sami Kyöstilä <skyostil@chromium.org>
Cr-Commit-Position: refs/heads/master@{#616756}
[modify] https://crrev.com/dacd59c9179d75f89a27be539dde8892984612bc/base/json/json_perftest.cc

Sign in to add a comment