New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 696675 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Mar 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 1
Type: Bug-Regression

Blocking:
issue 664505



Sign in to add a comment

system_health.memory_desktop on ATI GPU on Mac on Mac-10.11 failing on chromium.perf/Mac Pro 10.11 Perf

Project Member Reported by robert...@chromium.org, Feb 27 2017

Issue description

system_health.memory_desktop on ATI GPU on Mac on Mac-10.11 failing on chromium.perf/Mac Pro 10.11 Perf

Builders failed on: 
- Mac Pro 10.11 Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Mac%20Pro%2010.11%20Perf

Regression ranges:
chromium:
451276 - 451303

Failed swarming task:
https://chromium-swarm.appspot.com/user/task/349843479f19f210

Snippet:
--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8
2017-02-27 06:06:36.427 Google Chrome[72841:1075066] Call stack:
(
    "+callStackSymbols disabled for performance reasons"
)
2017-02-27 06:06:36.830 Google Chrome[72841:1075066] Errors logged by ksadmin: KSKeyedPersistentStore store directory does not exist. [com.google.UpdateEngine.CommonErrorDomain:501 - '/Library/Google/GoogleSoftwareUpdate/TicketStore' - 'KSKeyedPersistentStore.m:372']
KSPersistentTicketStore failed to load tickets. (productID: com.google.Chrome) [com.google.UpdateEngine.CoreErrorDomain:1051 - '/Library/Google/GoogleSoftwareUpdate/TicketStore/Keystone.ticketstore'] (KSKeyedPersistentStore store directory does not exist. - '/Library/Google/GoogleSoftwareUpdate/TicketStore' [com.google.UpdateEngine.CommonErrorDomain:501])
ksadmin cannot access the ticket store:<KSUpdateError:0x100514400
	domain="com.google.UpdateEngine.CoreErrorDomain"
	code=1051
	userInfo={
		function = "-[KSProductKeyedStore(ProtectedMethods) errorForStoreError:productID:message:timeoutMessage:]";
		date = 2017-02-27 14:06:36 +0000;
		productids = {(
			"com.google.Chrome"
		)};
		filename = "KSProductKeyedStore.m";
		line = 91;
		NSFilePath = "/Library/Google/GoogleSoftwareUpdate/TicketStore/Keystone.ticketstore";
		NSUnderlyingError = <KSError:0x100510630
			domain="com.google.UpdateEngine.CommonErrorDomain"
			code=501
			userInfo={
				date = 2017-02-27 14:06:36 +0000;
				line = 372;
				filename = "KSKeyedPersistentStore.m";
				function = "-[KSKeyedPersistentStore(PrivateMethods) validateStorePath]";
				NSFilePath = "/Library/Google/GoogleSoftwareUpdate/TicketStore";
				NSLocalizedDescription = "KSKeyedPersistentStore store directory does not exist.";
			}
		>;
		NSLocalizedDescription = "KSPersistentTicketStore failed to load tickets.";
	}
>

INFO:root:*********** END OF BROWSER STANDARD OUTPUT ************
INFO:root:********************* BROWSER LOG *********************
INFO:root:No log file
INFO:root:***************** END OF BROWSER LOG ******************
Traceback (most recent call last):
  File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 81, in _RunStoryAndProcessErrorIfNeeded
    state.WillRunStory(story)
  File "/b/s/w/irZFQlUs/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 234, in WillRunStory
    self.browser.tabs[0].WaitForDocumentReadyStateToBeComplete()
  File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/browser/tab_list.py", line 18, in __getitem__
    return self._tab_list_backend.__getitem__(index)
  File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py", line 64, in __getitem__
    return self.GetBackendFromContextId(context_id)
  File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py", line 78, in GetBackendFromContextId
    raise e
TimeoutException: <unprintable TimeoutException object>

INFO:root:Chrome version: 3026
INFO:root:Try printing formatted exception: None None None

Exception raised when cleaning story run: 

Traceback (most recent call last):
  _RunStoryAndProcessErrorIfNeeded at /b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/story_runner.py:111
    test.DidRunStory(state.platform)
  DidRunStory at /b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:315
    platform.tracing_controller.StopTracing()
  StopTracing at /b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/core/tracing_controller.py:47
    return self._tracing_controller_backend.StopTracing()
  StopTracing at /b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:140
    '\n'.join(raised_exception_messages))
Error: Exceptions raised when trying to stop tracing:
Traceback (most recent call last):
  File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 128, in StopTracing
    agent.CollectAgentTraceData(builder)
  File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 247, in CollectAgentTraceData
    '\n'.join(raised_exception_messages))
ChromeTracingStoppedError: Exceptions raised when trying to collect Chrome devtool tracing:
Error when collecting Chrome tracing on devtools at port 49565:
Traceback (most recent call last):
  File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 236, in CollectAgentTraceData
    client.CollectChromeTracingData(trace_data_builder)
  File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 370, in CollectChromeTracingData
    self._tracing_backend.CollectTraceData(trace_data_builder, timeout)
  File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 230, in CollectTraceData
    self._CollectTracingData(trace_data_builder, timeout)
  File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 268, in _CollectTracingData
    'the timeout amount.' % elapsed_time)
TracingTimeoutException: Only received partial trace data due to timeout after 30.0023741722 seconds. If the trace data is big, you may want to increase the timeout amount.

Locals:
  agent                     : <telemetry.internal.platform.tracing_controller_backend.TracingControllerBackend object at 0x10fb82ed0>
  builder                   : <telemetry.timeline.trace_data.TraceDataBuilder object at 0x113be8490>
  raised_exception_messages : ['Traceback (most recent call last):\n  File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 128, in StopTracing\n    agent.CollectAgentTraceData(builder)\n  File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 247, in CollectAgentTraceData\n    \'\\n\'.join(raised_exception_messages))\nChromeTracingStoppedError: Exceptions raised when trying to collect Chrome devtool  ... nds/chrome_inspector/tracing_backend.py", line 230, in CollectTraceData\n    self._CollectTracingData(trace_data_builder, timeout)\n  File "/b/s/w/irZFQlUs/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 268, in _CollectTracingData\n    \'the timeout amount.\' % elapsed_time)\nTracingTimeoutException: Only received partial trace data due to timeout after 30.0023741722 seconds. If the trace data is big, you may want to increase the timeout amount.\n\n'] (truncated)

[  FAILED  ] load:news:nytimes (151239 ms)
--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8--->8
 
Blocking: 664505
Cc: -perezju@chromium.org sullivan@chromium.org charliea@chromium.org nednguyen@chromium.org
Owner: perezju@chromium.org
Status: Started (was: Available)
CL on review to increase the timeout: https://codereview.chromium.org/2727643002/
Cc: nedngu...@google.com
nednguyen@, can you think of any way that we could get a TBMv2 metric around this? It seems like that'd be a useful thing to have. Is the Telemetry tracing agent already closed by this point?
We can modify Telemetry tracing agent to be the last agent to collect data from in tracing_controller.Close() so that we can make sure it has data about how long does the collect trace step of each agent takes.
Ah, yea. It looks like the steps are:

Start tracing on Telemetry
Start tracing on all other agents
Record clock sync markers on all agents
Stop tracing on all other agents
Stop tracing on Telemetry
Collect traces on all agents

And, because Telemetry is also an agent, tracing on it has stopped by the time that we start collecting traces from other agents. In order to create a metric around this, we'd have to move "Stop tracing on Telemetry" until after "Collect traces on all agents"
Project Member

Comment 6 by bugdroid1@chromium.org, Mar 3 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/45bc57b54c6a1e20ed41254327a06de44ea7bb1d

commit 45bc57b54c6a1e20ed41254327a06de44ea7bb1d
Author: catapult-deps-roller <catapult-deps-roller@chromium.org>
Date: Fri Mar 03 14:08:46 2017

Roll src/third_party/catapult/ 7ef761733..1b305da0d (14 commits)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/7ef761733d29..1b305da0d509

$ git log 7ef761733..1b305da0d --date=short --no-merges --format='%ad %ae %s'
2017-03-03 perezju Revert of [Telemetry][Wpr] Remove old version of archive info. (patchset #3 id:40001 of https://codereview.chromium.org/2694843002/ )
2017-03-02 bsheedy Fix HasRoot and NeedsSU on sailfish/marlin
2017-03-02 dtu [dashboard] Suppress logging.
2017-03-02 sunjian Add logic for identifying Location::setHash.
2017-03-02 ccraik Limit checking --async-stop to platform versions without the fix
2017-03-02 dtu [dashboard] Suppress deployment confirmation prompt.
2017-03-02 jessimb Updating the table to accept legacy units.
2017-03-02 ccraik Add rs back as default categories, work around breakage
2017-03-02 hjd [tracing] Have breakdown_span.html depend on ChromeProcessNames FixedColorScheme
2017-03-02 rnephew [Telemetry][Wpr] Remove old version of archive info.
2017-03-02 simonhatch Dashboard - Clarify difference when comparing metrics.
2017-03-02 simonhatch Pinpoint - Listen for iron-select events and notify child pages.
2017-03-02 perezju [Telemetry] JS API: Remove redundant v2 method names
2017-03-01 perezju [Telemetry] Increase timeout to collect trace to 60 seconds

Created with:
  roll-dep src/third_party/catapult
BUG=618698,689587,618698, 696675 

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=catapult-sheriff@chromium.org

Review-Url: https://codereview.chromium.org/2730903002
Cr-Commit-Position: refs/heads/master@{#454577}

[modify] https://crrev.com/45bc57b54c6a1e20ed41254327a06de44ea7bb1d/DEPS

Status: Fixed (was: Started)
The fix for that timeout is in that catapult roll above. Feel free to re-open if we keep seeing issues despite the fix.

Sign in to add a comment