page_cycler_v2.typical_25 is flaky, apparently exiting or crashing the browser early on |
|||||||
Issue descriptionThe newly added benchmark page_cycler_v2.typical_25 is flaky, apparently exiting or crashing the browser early on: https://build.chromium.org/p/chromium.perf/builders/Win%207%20Intel%20GPU%20Perf%20%282%29 It's failed in most runs on this bot since build 4199 where it was introduced: https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%207%20Intel%20GPU%20Perf%20%282%29/builds/4199 It's a problem that no stack trace is displayed for the failure. It's not clear to me why that's happening. If the browser's crashing a minidump should be created and symbolized. There are indications that there may be a race condition in minidump symbolization in Telemetry. kouhei@, assigning to you to attempt to reproduce on Windows and see what is happening with this benchmark. I will separately try to reproduce and fix any race conditions in Telemetry's minidump processing.
,
May 26 2016
This new benchmark should probably be disabled on all platforms until its problems have been more thoroughly debugged.
,
May 26 2016
kouhei@ is out, I can make the disable CL.
,
May 26 2016
Thanks Ned.
,
May 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c84885e83ecac137b93ed2b6c8b7323d8e91ffe1 commit c84885e83ecac137b93ed2b6c8b7323d8e91ffe1 Author: nednguyen <nednguyen@google.com> Date: Fri May 27 00:59:52 2016 [tools/perf] Disable page_cycler_v2.typical_25 benchmark TBR=kouhei@chromium.org BUG= 615178 Review-Url: https://codereview.chromium.org/2013233002 Cr-Commit-Position: refs/heads/master@{#396337} [modify] https://crrev.com/c84885e83ecac137b93ed2b6c8b7323d8e91ffe1/tools/perf/benchmarks/page_cycler_v2.py
,
May 27 2016
,
May 27 2016
Issue 615034 has been merged into this issue.
,
May 27 2016
,
May 27 2016
Now that Issue 615180 has been (I think) fixed, perhaps we should try briefly re-enabling this benchmark and see if a reasonable stack trace is created on the Perf bots. What do you think?
,
May 27 2016
I agree. I just make the CL to reland the benchmark.
,
May 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/997821c89eb567984eeea774934e955ddb4ec044 commit 997821c89eb567984eeea774934e955ddb4ec044 Author: nednguyen <nednguyen@google.com> Date: Fri May 27 18:56:02 2016 Revert of [tools/perf] Disable page_cycler_v2.typical_25 benchmark (patchset #1 id:1 of https://codereview.chromium.org/2013233002/ ) Reason for revert: Stack trace of crash should be enabled now. Reenable the benchmark to see what causes the failure. Original issue's description: > [tools/perf] Disable page_cycler_v2.typical_25 benchmark > > TBR=kouhei@chromium.org > BUG= 615178 > > Committed: https://crrev.com/c84885e83ecac137b93ed2b6c8b7323d8e91ffe1 > Cr-Commit-Position: refs/heads/master@{#396337} TBR= # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG= 615178 Review-Url: https://codereview.chromium.org/2017123004 Cr-Commit-Position: refs/heads/master@{#396525} [modify] https://crrev.com/997821c89eb567984eeea774934e955ddb4ec044/tools/perf/benchmarks/page_cycler_v2.py
,
May 28 2016
Oh, we have a devtool crash ('Inspector.targetCrashed') when trying to collect tracing, hence there is no crash stack:
[ RUN ] http://money.cnn.com/
(INFO) 2016-05-27 23:11:28,655 chrome_tracing_agent._CreateTraceConfigFile:245 Trace config file string: {"trace_config": {"enable_systrace": false, "included_categories": ["loading", "*", "navigation", "blink.user_timing", "blink.console"], "record_mode": "record-as-much-as-possible"}}
(INFO) 2016-05-27 23:11:28,655 tracing_backend.StartTracing:130 Start Tracing Request: {'params': {'transferMode': 'ReturnAsStream', 'traceConfig': {'enableSystrace': False, 'recordMode': 'recordAsMuchAsPossible', 'includedCategories': ['loading', '*', 'navigation', 'blink.user_timing', 'blink.console']}}, 'method': 'Tracing.start'}
(INFO) 2016-05-27 23:11:28,657 cache_temperature.EnsurePageCacheTemperature:55 PageCacheTemperature: pcv1-cold
(WARNING) 2016-05-27 23:11:29,802 desktop_browser_backend._GetMostRecentCrashpadMinidump:338 No path to crashpad_database_util found
(INFO) 2016-05-27 23:11:29,802 desktop_browser_backend._GetMostRecentMinidump:399 No minidump found via crashpad_database_util
Traceback (most recent call last):
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\story_runner.py", line 86, in _RunStoryAndProcessErrorIfNeeded
test.Measure(state.platform, results)
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\web_perf\timeline_based_measurement.py", line 279, in Measure
trace_result = platform.tracing_controller.StopTracing()
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\core\tracing_controller.py", line 39, in StopTracing
return self._tracing_controller_backend.StopTracing()
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\platform\tracing_controller_backend.py", line 131, in StopTracing
'\n'.join(raised_exception_messages))
TracingControllerStoppedError: Exceptions raised when trying to stop tracing:
Traceback (most recent call last):
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\platform\tracing_controller_backend.py", line 120, in StopTracing
agent.CollectAgentTraceData(builder)
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\platform\tracing_agent\chrome_tracing_agent.py", line 223, in CollectAgentTraceData
'\n'.join(raised_execption_messages))
ChromeTracingStoppedError: Exceptions raised when trying to stop Chrome devtool tracing:
Error when trying to stop Chrome tracing on devtools at port 59247:
Traceback (most recent call last):
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\platform\tracing_agent\chrome_tracing_agent.py", line 208, in CollectAgentTraceData
client.StopChromeTracing(trace_data_builder)
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\devtools_client_backend.py", line 353, in StopChromeTracing
"console.time.toString().indexOf('[native code]') != -1;")
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py", line 32, in inner
return func(inspector_backend, *args, **kwargs)
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py", line 203, in EvaluateJavaScript
return self._runtime.Evaluate(expr, context_id, timeout)
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_runtime.py", line 45, in Evaluate
res = self._inspector_websocket.SyncRequest(request, timeout)
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_websocket.py", line 110, in SyncRequest
res = self._Receive(timeout)
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_websocket.py", line 166, in _Receive
self._HandleNotification(result)
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_websocket.py", line 179, in _HandleNotification
self._domain_handlers[domain_name](result)
File "C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py", line 288, in _HandleInspectorDomainNotification
raise exception
DevtoolsTargetCrashException: Devtools target crashed
********************************************************************************
(C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py:341 _AddDebuggingInformation) Received a socket error in the browser connection and the tab no longer exists. The tab probably crashed.
********************************************************************************
(C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py:342 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:59247/devtools/page/ec56d6df-3ddb-4d63-806e-362c66900c07
Found Minidump: False
Stack Trace:
********************************************************************************
No crash dump found.
********************************************************************************
Standard output:
********************************************************************************
********************************************************************************
[ FAILED ] http://money.cnn.com/ (2173 ms)
*Andrey: this looks like a problem with devtool_tracing_handler, can you look into this?
,
May 28 2016
Sorry, I didn't look at the stack carefully. The exception happens when we try to validate whether console.time* methods in Chrome is monkey-patched, so this is happened before we collect tracing from devtool:
"C:\b\build\slave\Win_7_Intel_GPU_Perf__2_\build\src\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\devtools_client_backend.py", line 353, in StopChromeTracing
"console.time.toString().indexOf('[native code]') != -1;")
Still this may still be a devtool failure, since the stack at "catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py", line 288", line 179, in _HandleNotification" can only be reached when we receive 'Inspector.targetCrashed'.
,
May 30 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6ef05830840e2cc4a578c906a645d5cc0198836e commit 6ef05830840e2cc4a578c906a645d5cc0198836e Author: kouhei <kouhei@chromium.org> Date: Mon May 30 07:03:54 2016 [tools/perf] Disable page_cycler_v2.typical_25 benchmark again TBR=nednguyen@google.com BUG= 615178 CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq Review-Url: https://codereview.chromium.org/2025583003 Cr-Commit-Position: refs/heads/master@{#396691} [modify] https://crrev.com/6ef05830840e2cc4a578c906a645d5cc0198836e/tools/perf/benchmarks/page_cycler_v2.py
,
May 30 2016
I looked into the logs whole day, but don't have a good clue. I'll try to invest some time setting up win/android development environment to have a closer look.
,
Jun 1 2016
It looks like the failures are happening on Mac as well. I looked into https://build.chromium.org/p/chromium.perf/builders/Mac%20HDD%20Perf%20%281%29/ for an unrelated reason and found this test flaking again: https://build.chromium.org/p/chromium.perf/builders/Mac%20HDD%20Perf%20%281%29/builds/1693/steps/page_cycler.typical_25.reference/logs/stdio -------- (WARNING) 2016-05-31 16:50:58,074 desktop_browser_backend._GetMostRecentCrashpadMinidump:338 No path to crashpad_database_util found (INFO) 2016-05-31 16:50:58,074 desktop_browser_backend._GetMostRecentMinidump:399 No minidump found via crashpad_database_util (INFO) 2016-05-31 16:50:58,077 desktop_browser_backend.HasBrowserFinishedLaunching:229 Discovered ephemeral port 59471 Traceback (most recent call last): File "/b/build/slave/Mac_HDD_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 77, in _RunStoryAndProcessErrorIfNeeded state.WillRunStory(story) File "/b/build/slave/Mac_HDD_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 273, in WillRunStory self.browser.tabs[0].WaitForDocumentReadyStateToBeComplete() File "/b/build/slave/Mac_HDD_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/browser/tab_list.py", line 18, in __getitem__ return self._tab_list_backend.__getitem__(index) File "/b/build/slave/Mac_HDD_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py", line 64, in __getitem__ return self.GetBackendFromContextId(context_id) File "/b/build/slave/Mac_HDD_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py", line 78, in GetBackendFromContextId raise e TimeoutException: ******************************************************************************** (/b/build/slave/Mac_HDD_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:315 _ConvertExceptionFromInspectorWebsocket) The app is probably crashed: Found Minidump: False Stack Trace: ******************************************************************************** No crash dump found. -------- I can't guess why this particular benchmark would fail and the others wouldn't. My best guess is that this is the first benchmark that runs on the machine, and so it gets the cost of cold start of the browser. Perhaps Chrome takes so long to start on some hardware that it times out while doing so?
,
Jun 1 2016
Hmm. Is this only happening on reference? TBMv2 first_paint_metrics.html relies on new trace points added recently, so we may want to update the reference build. However, the timeout errors seems orthogonal. One hope is that the reference build update would fix that too.
,
Jun 1 2016
,
Jun 1 2016
Note that the all PCv2 benchmarks are disabled at ToT. We'd like to re-enable this on at least Linux, where we haven't seen this problem.
,
Jun 1 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3888b875185bfcbe94bad3a4708456d0c0b6e9af commit 3888b875185bfcbe94bad3a4708456d0c0b6e9af Author: kouhei <kouhei@chromium.org> Date: Wed Jun 01 03:58:44 2016 Re-enable PCv2 on known-to-work platforms Mac/Linux BUG= 615178 Review-Url: https://codereview.chromium.org/2024103005 Cr-Commit-Position: refs/heads/master@{#397048} [modify] https://crrev.com/3888b875185bfcbe94bad3a4708456d0c0b6e9af/tools/perf/benchmarks/page_cycler_v2.py
,
Jun 1 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b34eeae48ba79e80c4ec601922459d77af905f5a commit b34eeae48ba79e80c4ec601922459d77af905f5a Author: kouhei <kouhei@chromium.org> Date: Wed Jun 01 13:04:28 2016 re-enable PCv2 on non-ref Android BUG= 615178 Review-Url: https://codereview.chromium.org/2030523003 Cr-Commit-Position: refs/heads/master@{#397120} [modify] https://crrev.com/b34eeae48ba79e80c4ec601922459d77af905f5a/tools/perf/benchmarks/page_cycler_v2.py
,
Jun 1 2016
#17: yes, looking at https://build.chromium.org/p/chromium.perf/builders/Mac%20HDD%20Perf%20%281%29?numbuilds=200 it does look like page_cycler.typical_25.reference is the only flavor of the benchmark that's failing.
,
Jun 3 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e40c98d866412ae274204e989651132ea04e93d7 commit e40c98d866412ae274204e989651132ea04e93d7 Author: zhenw <zhenw@chromium.org> Date: Fri Jun 03 11:46:19 2016 Disable page_cycler_v2.typical_25 on Android One and Nexus 5X TBR=nednguyen@google.com BUG= 615178 ,616781 Review-Url: https://codereview.chromium.org/2030243003 Cr-Commit-Position: refs/heads/master@{#397684} [modify] https://crrev.com/e40c98d866412ae274204e989651132ea04e93d7/tools/perf/benchmarks/page_cycler_v2.py
,
Jun 24 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1eff267c107b3a0d1c7411df51d28125b0a0212a commit 1eff267c107b3a0d1c7411df51d28125b0a0212a Author: kouhei <kouhei@chromium.org> Date: Fri Jun 24 05:19:50 2016 [PCv2] Re-enable on desktops BUG= 615178 , 621473 CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq Review-Url: https://codereview.chromium.org/2091183002 Cr-Commit-Position: refs/heads/master@{#401818} [modify] https://crrev.com/1eff267c107b3a0d1c7411df51d28125b0a0212a/tools/perf/benchmarks/page_cycler_v2.py
,
Jun 27 2016
I think this is fixed by ksakamoto@. https://bugs.chromium.org/p/chromium/issues/detail?id=621473 |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by kbr@chromium.org
, May 26 2016