Issue metadata
Sign in to add a comment
|
system_health.memory_desktop on ATI GPU on Mac on Mac-10.11 failing on chromium.perf/Mac Pro 10.11 Perf |
||||||||||||||||||||
Issue descriptionsystem_health.memory_desktop on ATI GPU on Mac on Mac-10.11 failing on chromium.perf/Mac Pro 10.11 Perf Builders failed on: - Mac Pro 10.11 Perf: https://build.chromium.org/p/chromium.perf/builders/Mac%20Pro%2010.11%20Perf Regression ranges: chromium: 451276 - 451303 Failed swarming task: https://chromium-swarm.appspot.com/user/task/349843479f19f210 Snippet: --->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8 2017-02-27 06:06:36.427 Google Chrome[72841:1075066] Call stack: ( "+callStackSymbols disabled for performance reasons" ) 2017-02-27 06:06:36.830 Google Chrome[72841:1075066] Errors logged by ksadmin: KSKeyedPersistentStore store directory does not exist. [com.google.UpdateEngine.CommonErrorDomain:501 - '/Library/Google/GoogleSoftwareUpdate/TicketStore' - 'KSKeyedPersistentStore.m:372'] KSPersistentTicketStore failed to load tickets. (productID: com.google.Chrome) [com.google.UpdateEngine.CoreErrorDomain:1051 - '/Library/Google/GoogleSoftwareUpdate/TicketStore/Keystone.ticketstore'] (KSKeyedPersistentStore store directory does not exist. - '/Library/Google/GoogleSoftwareUpdate/TicketStore' [com.google.UpdateEngine.CommonErrorDomain:501]) ksadmin cannot access the ticket store:<KSUpdateError:0x100514400 domain="com.google.UpdateEngine.CoreErrorDomain" code=1051 userInfo={ function = "-[KSProductKeyedStore(ProtectedMethods) errorForStoreError:productID:message:timeoutMessage:]"; date = 2017-02-27 14:06:36 +0000; productids = {( "com.google.Chrome" )}; filename = "KSProductKeyedStore.m"; line = 91; NSFilePath = "/Library/Google/GoogleSoftwareUpdate/TicketStore/Keystone.ticketstore"; NSUnderlyingError = <KSError:0x100510630 domain="com.google.UpdateEngine.CommonErrorDomain" code=501 userInfo={ date = 2017-02-27 14:06:36 +0000; line = 372; filename = "KSKeyedPersistentStore.m"; function = "-[KSKeyedPersistentStore(PrivateMethods) validateStorePath]"; NSFilePath = "/Library/Google/GoogleSoftwareUpdate/TicketStore"; NSLocalizedDescription = "KSKeyedPersistentStore store directory does not exist."; } >; NSLocalizedDescription = "KSPersistentTicketStore failed to load tickets."; } > INFO:root:*********** END OF BROWSER STANDARD OUTPUT ************ INFO:root:********************* BROWSER LOG ********************* INFO:root:No log file INFO:root:***************** END OF BROWSER LOG ****************** Traceback (most recent call last): File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 81, in _RunStoryAndProcessErrorIfNeeded state.WillRunStory(story) File "/b/s/w/irZFQlUs/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 234, in WillRunStory self.browser.tabs[0].WaitForDocumentReadyStateToBeComplete() File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/browser/tab_list.py", line 18, in __getitem__ return self._tab_list_backend.__getitem__(index) File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py", line 64, in __getitem__ return self.GetBackendFromContextId(context_id) File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py", line 78, in GetBackendFromContextId raise e TimeoutException: <unprintable TimeoutException object> INFO:root:Chrome version: 3026 INFO:root:Try printing formatted exception: None None None Exception raised when cleaning story run: Traceback (most recent call last): _RunStoryAndProcessErrorIfNeeded at /b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/story_runner.py:111 test.DidRunStory(state.platform) DidRunStory at /b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:315 platform.tracing_controller.StopTracing() StopTracing at /b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/core/tracing_controller.py:47 return self._tracing_controller_backend.StopTracing() StopTracing at /b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:140 '\n'.join(raised_exception_messages)) Error: Exceptions raised when trying to stop tracing: Traceback (most recent call last): File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 128, in StopTracing agent.CollectAgentTraceData(builder) File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 247, in CollectAgentTraceData '\n'.join(raised_exception_messages)) ChromeTracingStoppedError: Exceptions raised when trying to collect Chrome devtool tracing: Error when collecting Chrome tracing on devtools at port 49565: Traceback (most recent call last): File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 236, in CollectAgentTraceData client.CollectChromeTracingData(trace_data_builder) File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 370, in CollectChromeTracingData self._tracing_backend.CollectTraceData(trace_data_builder, timeout) File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 230, in CollectTraceData self._CollectTracingData(trace_data_builder, timeout) File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 268, in _CollectTracingData 'the timeout amount.' % elapsed_time) TracingTimeoutException: Only received partial trace data due to timeout after 30.0023741722 seconds. If the trace data is big, you may want to increase the timeout amount. Locals: agent : <telemetry.internal.platform.tracing_controller_backend.TracingControllerBackend object at 0x10fb82ed0> builder : <telemetry.timeline.trace_data.TraceDataBuilder object at 0x113be8490> raised_exception_messages : ['Traceback (most recent call last):\n File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 128, in StopTracing\n agent.CollectAgentTraceData(builder)\n File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 247, in CollectAgentTraceData\n \'\\n\'.join(raised_exception_messages))\nChromeTracingStoppedError: Exceptions raised when trying to collect Chrome devtool ... nds/chrome_inspector/tracing_backend.py", line 230, in CollectTraceData\n self._CollectTracingData(trace_data_builder, timeout)\n File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 268, in _CollectTracingData\n \'the timeout amount.\' % elapsed_time)\nTracingTimeoutException: Only received partial trace data due to timeout after 30.0023741722 seconds. If the trace data is big, you may want to increase the timeout amount.\n\n'] (truncated) [ FAILED ] load:news:nytimes (151239 ms) --->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8
,
Mar 1 2017
CL on review to increase the timeout: https://codereview.chromium.org/2727643002/
,
Mar 1 2017
nednguyen@, can you think of any way that we could get a TBMv2 metric around this? It seems like that'd be a useful thing to have. Is the Telemetry tracing agent already closed by this point?
,
Mar 1 2017
We can modify Telemetry tracing agent to be the last agent to collect data from in tracing_controller.Close() so that we can make sure it has data about how long does the collect trace step of each agent takes.
,
Mar 1 2017
Ah, yea. It looks like the steps are: Start tracing on Telemetry Start tracing on all other agents Record clock sync markers on all agents Stop tracing on all other agents Stop tracing on Telemetry Collect traces on all agents And, because Telemetry is also an agent, tracing on it has stopped by the time that we start collecting traces from other agents. In order to create a metric around this, we'd have to move "Stop tracing on Telemetry" until after "Collect traces on all agents"
,
Mar 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/45bc57b54c6a1e20ed41254327a06de44ea7bb1d commit 45bc57b54c6a1e20ed41254327a06de44ea7bb1d Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Fri Mar 03 14:08:46 2017 Roll src/third_party/catapult/ 7ef761733..1b305da0d (14 commits) https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/7ef761733d29..1b305da0d509 $ git log 7ef761733..1b305da0d --date=short --no-merges --format='%ad %ae %s' 2017-03-03 perezju Revert of [Telemetry][Wpr] Remove old version of archive info. (patchset #3 id:40001 of https://codereview.chromium.org/2694843002/ ) 2017-03-02 bsheedy Fix HasRoot and NeedsSU on sailfish/marlin 2017-03-02 dtu [dashboard] Suppress logging. 2017-03-02 sunjian Add logic for identifying Location::setHash. 2017-03-02 ccraik Limit checking --async-stop to platform versions without the fix 2017-03-02 dtu [dashboard] Suppress deployment confirmation prompt. 2017-03-02 jessimb Updating the table to accept legacy units. 2017-03-02 ccraik Add rs back as default categories, work around breakage 2017-03-02 hjd [tracing] Have breakdown_span.html depend on ChromeProcessNames FixedColorScheme 2017-03-02 rnephew [Telemetry][Wpr] Remove old version of archive info. 2017-03-02 simonhatch Dashboard - Clarify difference when comparing metrics. 2017-03-02 simonhatch Pinpoint - Listen for iron-select events and notify child pages. 2017-03-02 perezju [Telemetry] JS API: Remove redundant v2 method names 2017-03-01 perezju [Telemetry] Increase timeout to collect trace to 60 seconds Created with: roll-dep src/third_party/catapult BUG=618698,689587,618698, 696675 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2730903002 Cr-Commit-Position: refs/heads/master@{#454577} [modify] https://crrev.com/45bc57b54c6a1e20ed41254327a06de44ea7bb1d/DEPS
,
Mar 3 2017
The fix for that timeout is in that catapult roll above. Feel free to re-open if we keep seeing issues despite the fix. |
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by perezju@chromium.org
, Feb 28 2017