[system-health] load:media:soundcloud ChromeClockSyncError in system_health.memory_desktop.reference on Mac HDD Perf (3) |
||||||||
Issue descriptionThe error occurred twice in the last ~10 runs: https://build.chromium.org/p/chromium.perf/builders/Mac%20HDD%20Perf%20%283%29/builds/1531/steps/system_health.memory_desktop.reference/logs/stdio https://build.chromium.org/p/chromium.perf/builders/Mac%20HDD%20Perf%20%283%29/builds/1535/steps/system_health.memory_desktop.reference/logs/stdio Error message: [ RUN ] load:media:soundcloud (INFO) 2016-05-14 20:14:37,197 chrome_tracing_agent._CreateTraceConfigFile:243 Trace config file string: {"trace_config":{"excluded_categories": ["*"], "included_categories": ["disabled-by-default-memory-infra"], "record_mode": "record-as-much-as-possible"}} (INFO) 2016-05-14 20:14:37,205 tracing_backend.StartTracing:124 Start Tracing Request: {'params': {'transferMode': 'ReturnAsStream', 'options': 'record-as-much-as-possible', 'categories': 'disabled-by-default-memory-infra,-*'}, 'method': 'Tracing.start'} (INFO) 2016-05-14 20:14:37,210 cache_temperature.EnsurePageCacheTemperature:29 PageCacheTemperature: any (ERROR) 2016-05-14 20:14:46,014 system_health_stories._TakeMemoryMeasurement:319 Unable to get a memory dump for load:media:soundcloud. Traceback (most recent call last): File "/b/build/slave/Mac_HDD_Perf__3_/build/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 86, in _RunStoryAndProcessErrorIfNeeded test.Measure(state.platform, results) File "/b/build/slave/Mac_HDD_Perf__3_/build/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py", line 279, in Measure trace_result = platform.tracing_controller.StopTracing() File "/b/build/slave/Mac_HDD_Perf__3_/build/src/third_party/catapult/telemetry/telemetry/core/tracing_controller.py", line 39, in StopTracing return self._tracing_controller_backend.StopTracing() File "/b/build/slave/Mac_HDD_Perf__3_/build/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 107, in StopTracing self._IssueClockSyncMarker() File "/b/build/slave/Mac_HDD_Perf__3_/build/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 197, in _IssueClockSyncMarker self._RecordIssuerClockSyncMarker) File "/b/build/slave/Mac_HDD_Perf__3_/build/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 181, in RecordClockSyncMarker sync_id, record_controller_clock_sync_marker_callback) File "/b/build/slave/Mac_HDD_Perf__3_/build/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 164, in _RecordClockSyncMarkerAsyncEvent raise ChromeClockSyncError() ChromeClockSyncError (INFO) 2016-05-14 20:14:46,109 exception_formatter.PrintFormattedException:19 Try printing formatted exception: None None None Exception raised when cleaning story run: Traceback (most recent call last): _RunStoryAndProcessErrorIfNeeded at /b/build/slave/Mac_HDD_Perf__3_/build/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:108 test.DidRunStory(state.platform) DidRunStory at /b/build/slave/Mac_HDD_Perf__3_/build/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:293 platform.tracing_controller.StopTracing() StopTracing at /b/build/slave/Mac_HDD_Perf__3_/build/src/third_party/catapult/telemetry/telemetry/core/tracing_controller.py:39 return self._tracing_controller_backend.StopTracing() StopTracing at /b/build/slave/Mac_HDD_Perf__3_/build/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:107 self._IssueClockSyncMarker() _IssueClockSyncMarker at /b/build/slave/Mac_HDD_Perf__3_/build/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:197 self._RecordIssuerClockSyncMarker) RecordClockSyncMarker at /b/build/slave/Mac_HDD_Perf__3_/build/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py:181 sync_id, record_controller_clock_sync_marker_callback) _RecordClockSyncMarkerAsyncEvent at /b/build/slave/Mac_HDD_Perf__3_/build/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py:164 raise ChromeClockSyncError() ChromeClockSyncError Locals: has_clock_synced : False record_controller_clock_sync_marker_callback : <bound method TracingControllerBackend._RecordIssuerClockSyncMarker of <telemetry.internal.platform.tracing_controller_backend.TracingControllerBackend object at 0x1093223d0>> sync_id : '1ffe3f32-e55c-44a0-aefd-9766a25c6e69' [ FAILED ] load:media:soundcloud (8915 ms) Next action: I would wait for a few more builds to see whether this failure is a flake or not.
,
May 18 2016
,
May 18 2016
,
May 18 2016
Marking assigned to petrcermak@.
,
May 19 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/330cd00d2ce9cc10d34ba1264d4dcc66c40916b1 commit 330cd00d2ce9cc10d34ba1264d4dcc66c40916b1 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Thu May 19 06:37:01 2016 Roll src/third_party/catapult/ 73dc5f4ee..9085ddba7 (15 commits). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/73dc5f4ee8e4..9085ddba7182 $ git log 73dc5f4ee..9085ddba7 --date=short --no-merges --format='%ad %ae %s' BUG= 352807 , 352807 ,612140, 581716 , 602701 TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/1994023002 Cr-Commit-Position: refs/heads/master@{#394699} [modify] https://crrev.com/330cd00d2ce9cc10d34ba1264d4dcc66c40916b1/DEPS
,
Jun 16 2016
,
Jun 16 2016
Neither of the recent failures (see comment #7) contains a "Failed to record clock sync marker ..." message (https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py?l=163). This means that by the time the story runner tries to stop tracing, there are no more inspector backends. That in turn implies that the browser/renderer has crashed during the story. I won't get to try to reproduce the issue locally until next week. In the meantime, we have 2 options regarding the specific benchmark: 1. Disable system health benchmark on mac-reference. 2. Keep the benchmark enabled on mac-reference. and 2 options how to simplify triaging issues like this: 1. Modify story runner to print screenshots upon exceptions (the functionality is currently in SharedPageState). 2. Modify story runner to print browser standard output and log upon exceptions (similarly to https://codereview.chromium.org/2072713003/). My personal preference (definitely biased) would be to keep the benchmark enabled (there were 2 failures in the last 20 builds on each bot, which is below the 1/5 ratio suggested by the sheriffing guide for disabling tests: https://chromium.googlesource.com/chromium/src/+/master/tools/perf/docs/perf_bot_sheriffing.md#Handle-Test-Failures) and get screenshots+logs on all Telemetry failures. Ned: WDYT?
,
Jun 16 2016
Petr: keep the benchmark enabled & add more logging data sounds great to me as it also helps the tot one in the future.
,
Jun 20 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a604c7548e4b1ed89dfc8a8de3f0b5a501214b17 commit a604c7548e4b1ed89dfc8a8de3f0b5a501214b17 Author: petrcermak <petrcermak@chromium.org> Date: Mon Jun 20 15:50:32 2016 [memory-infra] Add DumpStateUponFailure method to MultiBrowserSharedState This is necessary for https://codereview.chromium.org/2074183002/ (Telemetry patch in the Catapult repository). BUG=612140 Review-Url: https://codereview.chromium.org/2085483002 Cr-Commit-Position: refs/heads/master@{#400687} [modify] https://crrev.com/a604c7548e4b1ed89dfc8a8de3f0b5a501214b17/tools/perf/page_sets/dual_browser_story.py
,
Jun 21 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/649d1016cfc6fcd4bcec9ade105f9c384c0fbdf4 commit 649d1016cfc6fcd4bcec9ade105f9c384c0fbdf4 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Tue Jun 21 09:32:29 2016 Roll src/third_party/catapult/ bed4eb00e..2c7c14263 (1 commit). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/bed4eb00e376..2c7c14263b4e $ git log bed4eb00e..2c7c14263 --date=short --no-merges --format='%ad %ae %s' BUG=612140 TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2083653004 Cr-Commit-Position: refs/heads/master@{#400930} [modify] https://crrev.com/649d1016cfc6fcd4bcec9ade105f9c384c0fbdf4/DEPS
,
Aug 15 2016
,
Oct 3 2016
,
Oct 5 2016
,
Oct 5 2016
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by petrcermak@chromium.org
, May 16 2016