New issue
Advanced search Search tips

Issue 914390 link

Starred by 2 users
Status: Available
Owner: ----
Cc:
sullivan@google.com
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: ----



Sign in to add a comment

system_health.common_mobile/browse:chrome:newtab failing on chromium.perf/Android Nexus5 Perf

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, Dec 12 
Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of sullivan@google.com

system_health.common_mobile/browse:chrome:newtab in performance_test_suite failing on chromium.perf/Android Nexus5 Perf

Builders failed on: 
- Android Nexus5 Perf: 
  https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/Android%20Nexus5%20Perf

From the log:
(CRITICAL) 2018-12-12 12:10:56,037 reraiser_thread.JoinAll:200  Timed out. Dumping threads.
(CRITICAL) 2018-12-12 12:10:56,037 reraiser_thread.LogThreadStack:32  ********************************************************************************
(CRITICAL) 2018-12-12 12:10:56,037 reraiser_thread.LogThreadStack:33  Stack dump for thread 'TimeoutThread-1-for-MainThread'
(CRITICAL) 2018-12-12 12:10:56,037 reraiser_thread.LogThreadStack:34  ********************************************************************************
(CRITICAL) 2018-12-12 12:10:56,039 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/.swarming_module/lib/python2.7/threading.py", line 774, in __bootstrap
(CRITICAL) 2018-12-12 12:10:56,039 reraiser_thread.LogThreadStack:38    self.__bootstrap_inner()
(CRITICAL) 2018-12-12 12:10:56,039 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/.swarming_module/lib/python2.7/threading.py", line 801, in __bootstrap_inner
(CRITICAL) 2018-12-12 12:10:56,039 reraiser_thread.LogThreadStack:38    self.run()
(CRITICAL) 2018-12-12 12:10:56,039 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run
(CRITICAL) 2018-12-12 12:10:56,039 reraiser_thread.LogThreadStack:38    self._ret = self._func(*self._args, **self._kwargs)
(CRITICAL) 2018-12-12 12:10:56,039 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda>
(CRITICAL) 2018-12-12 12:10:56,039 reraiser_thread.LogThreadStack:38    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
(CRITICAL) 2018-12-12 12:10:56,039 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
(CRITICAL) 2018-12-12 12:10:56,039 reraiser_thread.LogThreadStack:38    return f(*args, **kwargs)
(CRITICAL) 2018-12-12 12:10:56,039 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/app_ui.py", line 243, in WaitForUiNode
(CRITICAL) 2018-12-12 12:10:56,039 reraiser_thread.LogThreadStack:38    return timeout_retry.WaitFor(node_found)
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 98, in WaitFor
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:38    result = condition()
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/app_ui.py", line 241, in node_found
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:38    return self.GetUiNode(**kwargs)
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/app_ui.py", line 223, in GetUiNode
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:38    return self._GetRootUiNode()._Find(**kwargs)
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:38    return impl()
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:38    return f(*args, **kwargs)
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/app_ui.py", line 201, in _GetRootUiNode
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:38    check_return=True)
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:38    return impl()
(CRITICAL) 2018-12-12 12:10:56,040 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:38    return f(*args, **kwargs)
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1161, in RunShellCommand
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:38    output = handle_large_output(cmd, large_output)
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1130, in handle_large_output
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:38    return handle_large_command(cmd)
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1112, in handle_large_command
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:38    return handle_check_return(cmd)
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1103, in handle_check_return
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:38    return run(cmd)
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1099, in run
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:38    return self.adb.Shell(cmd)
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 533, in Shell
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:38    output = self._RunDeviceAdbCmd(args, timeout, retries, check_error=False)
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 316, in _RunDeviceAdbCmd
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:38    check_error=check_error)
(CRITICAL) 2018-12-12 12:10:56,041 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
(CRITICAL) 2018-12-12 12:10:56,042 reraiser_thread.LogThreadStack:38    return impl()
(CRITICAL) 2018-12-12 12:10:56,042 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
(CRITICAL) 2018-12-12 12:10:56,042 reraiser_thread.LogThreadStack:38    return f(*args, **kwargs)
(CRITICAL) 2018-12-12 12:10:56,042 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 277, in _RunAdbCmd
(CRITICAL) 2018-12-12 12:10:56,042 reraiser_thread.LogThreadStack:38    timeout, env=cls._ADB_ENV)
(CRITICAL) 2018-12-12 12:10:56,042 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/cmd_helper.py", line 420, in GetCmdStatusAndOutputWithTimeout
(CRITICAL) 2018-12-12 12:10:56,042 reraiser_thread.LogThreadStack:38    for data in _IterProcessStdout(process, timeout=timeout):
(CRITICAL) 2018-12-12 12:10:56,042 reraiser_thread.LogThreadStack:36  File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/cmd_helper.py", line 282, in _IterProcessStdoutFcntl
(CRITICAL) 2018-12-12 12:10:56,042 reraiser_thread.LogThreadStack:38    [child_fd], [], [], iter_aware_poll_interval)
(CRITICAL) 2018-12-12 12:10:56,042 reraiser_thread.LogThreadStack:39  ********************************************************************************
(ERROR) 2018-12-12 12:10:56,042 story_runner.ProcessError:119  Possibly handleable error. Will try to restart shared state
Traceback (most recent call last):
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 145, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 304, in RunStory
    self._current_page.Run(self)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/__init__.py", line 99, in Run
    self.RunPageInteractions(action_runner)
  File "/b/swarming/w/ir/tools/perf/page_sets/system_health/system_health_story.py", line 123, in RunPageInteractions
    self._DidLoadDocument(action_runner)
  File "/b/swarming/w/ir/tools/perf/page_sets/system_health/chrome_stories.py", line 95, in _DidLoadDocument
    app_ui.WaitForUiNode(content_desc='New tab')
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run
    error_log_func=error_log_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 163, in _JoinAll
    (len(alive_threads), len(self._threads)))
CommandTimeoutError: Timed out waiting for 1 of 1 threads.

Comment 2 by sullivan@chromium.org, Dec 12 

Flakiness dashboard:
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=performance_test_suite&showAllRuns=true&builder=chromium.perf%3AAndroid%20Nexus5%20Perf

I'm going to say this is flaky starting at r601816:r601848, and it was flaky until it started failing hard at r603180:r603226. I'm going to kick off a wide failure bisect.
Project Member

Comment 3 by bugdroid1@chromium.org, Dec 12 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/22593cd3b809b4825a9d0b6c0aa08a71157e0c8b

commit 22593cd3b809b4825a9d0b6c0aa08a71157e0c8b
Author: Annie Sullivan <sullivan@chromium.org>
Date: Wed Dec 12 16:20:39 2018

Disable failing system_health.memory_mobile/browse:chrome:newtab.

Bug: 914390
Change-Id: Ib8d55b5782dd17b280440e87117654b5ede876be
TBR: charliea@chromium.org
NOTRY: true
Reviewed-on: https://chromium-review.googlesource.com/c/1374198
Reviewed-by: Annie Sullivan <sullivan@chromium.org>
Commit-Queue: Annie Sullivan <sullivan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#615913}
[modify] https://crrev.com/22593cd3b809b4825a9d0b6c0aa08a71157e0c8b/tools/perf/expectations.config

Comment 4 by perezju@chromium.org, Jan 3 

While fixing some inconsistencies in:
https://chromium-review.googlesource.com/c/chromium/src/+/1392734

we noted the disable in #3 above didn't quite work. Flakiness dashboard shows
the test still runs (and fails) on Nexus 5 for system_health.common_mobile.

Getting CL ready to fix these.
Project Member

Comment 5 by bugdroid1@chromium.org, Jan 3 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/37cc322c957f5a5ca742d1c817c0d1e5cdefb035

commit 37cc322c957f5a5ca742d1c817c0d1e5cdefb035
Author: Juan Antonio Navarro Perez <perezju@chromium.org>
Date: Thu Jan 03 19:05:51 2019

Fix story disables for browse:chrome:newtab

TBR=sullivan@chromium.org

Bug: 914390
Change-Id: I2cfe83fef58ea0f6c7a123c662ccfd5dd875aa5e
Reviewed-on: https://chromium-review.googlesource.com/c/1394393
Reviewed-by: Juan Antonio Navarro PĆ©rez <perezju@chromium.org>
Commit-Queue: Juan Antonio Navarro PĆ©rez <perezju@chromium.org>
Cr-Commit-Position: refs/heads/master@{#619708}
[modify] https://crrev.com/37cc322c957f5a5ca742d1c817c0d1e5cdefb035/tools/perf/expectations.config
►

Sign in to add a comment