Some telemetry tests timed-out when receiving trace data |
|||||||||||||
Issue descriptionThe following is page cycler v2. It also happens on v8.browsing_desktop. https://uberchromegw.corp.google.com/i/chromeos/builders/chell-chrome-pfq/builds/1134 [ RUN ] http://www.rei.com/@{'cache_temperature': 'cold'} Traceback (most recent call last): File "/home/chromeos-test/images/chell-chrome-pfq/R63-10026.0.0-rc1/telemetry_src/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 106, in _RunStoryAndProcessErrorIfNeeded test.Measure(state.platform, results) File "/home/chromeos-test/images/chell-chrome-pfq/R63-10026.0.0-rc1/telemetry_src/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py", line 281, in Measure trace_result = platform.tracing_controller.StopTracing() File "/home/chromeos-test/images/chell-chrome-pfq/R63-10026.0.0-rc1/telemetry_src/src/third_party/catapult/telemetry/telemetry/core/tracing_controller.py", line 47, in StopTracing return self._tracing_controller_backend.StopTracing() File "/home/chromeos-test/images/chell-chrome-pfq/R63-10026.0.0-rc1/telemetry_src/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 140, in StopTracing '\n'.join(raised_exception_messages)) TracingException: Exceptions raised when trying to stop tracing: Traceback (most recent call last): File "/home/chromeos-test/images/chell-chrome-pfq/R63-10026.0.0-rc1/telemetry_src/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 128, in StopTracing agent.CollectAgentTraceData(builder) File "/home/chromeos-test/images/chell-chrome-pfq/R63-10026.0.0-rc1/telemetry_src/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 250, in CollectAgentTraceData '\n'.join(raised_exception_messages)) ChromeTracingStoppedError: Exceptions raised when trying to collect Chrome devtool tracing: Error when collecting Chrome tracing on devtools at port 43223: Traceback (most recent call last): File "/home/chromeos-test/images/chell-chrome-pfq/R63-10026.0.0-rc1/telemetry_src/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 239, in CollectAgentTraceData client.CollectChromeTracingData(trace_data_builder) File "/home/chromeos-test/images/chell-chrome-pfq/R63-10026.0.0-rc1/telemetry_src/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 394, in CollectChromeTracingData self._tracing_backend.CollectTraceData(trace_data_builder, timeout) File "/home/chromeos-test/images/chell-chrome-pfq/R63-10026.0.0-rc1/telemetry_src/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 231, in CollectTraceData self._CollectTracingData(trace_data_builder, timeout) File "/home/chromeos-test/images/chell-chrome-pfq/R63-10026.0.0-rc1/telemetry_src/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 269, in _CollectTracingData 'the timeout amount.' % elapsed_time) TracingTimeoutException: Only received partial trace data due to timeout after 60.0187599659 seconds. If the trace data is big, you may want to increase the timeout amount.
,
Oct 13 2017
,
Oct 13 2017
This is P0 because we need it to verify the solution to crbug.com/773561 .
,
Oct 14 2017
Ned provided the following CL, https://chromium-review.googlesource.com/#/c/catapult/+/719937 but it does not fix the issue. We see now that it exceeds a timeout of 180 seconds. We are skipping some of the benchmarks so that we can do some testing. The problem does not happen for all benchmarks.
,
Oct 14 2017
If 180s still timeout, I suspect this is related to chiniforooshan's work recently
,
Oct 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/a87fe74bb7075356af128b703077f35fce520bcc commit a87fe74bb7075356af128b703077f35fce520bcc Author: Ting-Yuan Huang <laszio@chromium.org> Date: Sat Oct 14 00:42:31 2017 telemetry_AFDOGenerate: update the benchmark set Most of the other benchmarks doesn't work due to some telemetry issue. BUG= chromium:774728 TEST=test_that Change-Id: I4738b8dd2830cc2dd70227b018ab66704f0e5dc9 Reviewed-on: https://chromium-review.googlesource.com/720247 Commit-Queue: Ting-Yuan Huang <laszio@chromium.org> Tested-by: Ting-Yuan Huang <laszio@chromium.org> Trybot-Ready: Ting-Yuan Huang <laszio@chromium.org> Reviewed-by: Luis Lozano <llozano@chromium.org> [modify] https://crrev.com/a87fe74bb7075356af128b703077f35fce520bcc/server/site_tests/telemetry_AFDOGenerate/telemetry_AFDOGenerate.py
,
Oct 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/catapult/+/8adfc30c1bfc46d919f5a62a3199334fa2854e05 commit 8adfc30c1bfc46d919f5a62a3199334fa2854e05 Author: Nghia Nguyen <nednguyen@google.com> Date: Sat Oct 14 00:47:58 2017 Increase the timeout for collecting Chrome trace This is only a bandaid fix, once crbug.com/774728 is resolved, this CL should be reverted. Bug: chromium:774728 Change-Id: I24d94a7c520da27788c19c5083877a299748041e TBR=achuith@chromium.org, bccheng@chromium.org Reviewed-on: https://chromium-review.googlesource.com/719937 Reviewed-by: Ned Nguyen <nednguyen@google.com> Commit-Queue: Ned Nguyen <nednguyen@google.com> [modify] https://crrev.com/8adfc30c1bfc46d919f5a62a3199334fa2854e05/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py
,
Oct 16 2017
Adding the gardener for this week. Maybe a bisect can reveal which CL is problematic?
,
Oct 16 2017
How can I reproduce the error locally?
,
Oct 16 2017
I was able to reproduce with run_benchmark, e.g.:
./run_benchmark run --remote=${IPADDR} --output-dir=results --output-format=histograms --browser=cros-chrome --pageset-repeat=1 smoothness.top_25_smooth
,
Oct 16 2017
Thanks. Do you get the error mentioned in comment #1? When I run smoothness.top_25_smooth I get a different error for some stories. The error is "Timed out while waiting 60s for IsJavaScriptExpressionTrue.", which is probably a different issue (crbug.com/769246 ?).
,
Oct 16 2017
For the first stories (https://www.google.com/#hl=en&q=barack+obama, https://mail.google.com/mail, https://www.google.com/calendar/) I get the error mentioned in comment #1. I haven't seen the alternative error you mention - so not really sure if that's related or not - also I can't see crbug.com/769246 as I don't have access.
,
Oct 16 2017
Thank you. It's probably related to the ARC tracing agent since I got the error when I used an ARC-enabled VM.
,
Oct 16 2017
Oh, it's a similar issue to crbug.com/773540 , for ARC tracing agents. I'm working on a fix...
,
Oct 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/catapult/+/b870971cbd5e07b9fd182e5f367f01fe79d7b09b commit b870971cbd5e07b9fd182e5f367f01fe79d7b09b Author: Ned Nguyen <nednguyen@google.com> Date: Mon Oct 16 21:58:45 2017 Revert "Increase the timeout for collecting Chrome trace" This reverts commit 8adfc30c1bfc46d919f5a62a3199334fa2854e05. Reason for revert: doesn't help fixing the issue. Original change's description: > Increase the timeout for collecting Chrome trace > > This is only a bandaid fix, once crbug.com/774728 is resolved, this CL should > be reverted. > > Bug: chromium:774728 > Change-Id: I24d94a7c520da27788c19c5083877a299748041e > TBR=achuith@chromium.org, bccheng@chromium.org > Reviewed-on: https://chromium-review.googlesource.com/719937 > Reviewed-by: Ned Nguyen <nednguyen@google.com> > Commit-Queue: Ned Nguyen <nednguyen@google.com> TBR=nednguyen@google.com # Not skipping CQ checks because original CL landed > 1 day ago. Bug: chromium:774728 Change-Id: I5834cbb04f04cb674d38b0bd9e4021b5c5fd198c Reviewed-on: https://chromium-review.googlesource.com/721803 Reviewed-by: Ned Nguyen <nednguyen@google.com> Commit-Queue: Ned Nguyen <nednguyen@google.com> [modify] https://crrev.com/b870971cbd5e07b9fd182e5f367f01fe79d7b09b/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py
,
Oct 16 2017
Hopefully https://chromium-review.googlesource.com/c/chromium/src/+/721967 fixes this.
,
Oct 16 2017
and we also need to get the fix in R63.
,
Oct 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/782c0ce4c1dab05e55293798b828580ff2ceb7d1 commit 782c0ce4c1dab05e55293798b828580ff2ceb7d1 Author: Ting-Yuan Huang <laszio@chromium.org> Date: Tue Oct 17 05:27:57 2017 Revert "telemetry_AFDOGenerate: update the benchmark set" This reverts commit a87fe74bb7075356af128b703077f35fce520bcc. Reason for revert: It was a temporary change so that we can verify the fix to chromium:773561. Reverting it since chromium:773561 is verified. Original change's description: > telemetry_AFDOGenerate: update the benchmark set > > Most of the other benchmarks doesn't work due to some telemetry issue. > > BUG= chromium:774728 > TEST=test_that > > Change-Id: I4738b8dd2830cc2dd70227b018ab66704f0e5dc9 > Reviewed-on: https://chromium-review.googlesource.com/720247 > Commit-Queue: Ting-Yuan Huang <laszio@chromium.org> > Tested-by: Ting-Yuan Huang <laszio@chromium.org> > Trybot-Ready: Ting-Yuan Huang <laszio@chromium.org> > Reviewed-by: Luis Lozano <llozano@chromium.org> Bug: chromium:774728 Change-Id: Ic2be12b0cfec76a3b0d536dbd2463ea6e3023acb Reviewed-on: https://chromium-review.googlesource.com/721861 Commit-Ready: Ting-Yuan Huang <laszio@chromium.org> Tested-by: Ting-Yuan Huang <laszio@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> [modify] https://crrev.com/782c0ce4c1dab05e55293798b828580ff2ceb7d1/server/site_tests/telemetry_AFDOGenerate/telemetry_AFDOGenerate.py
,
Oct 17 2017
,
Oct 17 2017
Just tested https://chromium-review.googlesource.com/c/chromium/src/+/721967 - can confirm that this fixes the issue for me.
,
Oct 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6588f9b2158846757d9da3ee30ed386e845e64cc commit 6588f9b2158846757d9da3ee30ed386e845e64cc Author: Ehsan Chiniforooshan <chiniforooshan@chromium.org> Date: Tue Oct 17 18:37:04 2017 Tracing: keep track of agents that are tracing Instead of asking all agents to keep track of whether they are currently tracing or not and bail out of StopAndFlush if they are not tracing, the coordinator only sends StopAndFlush to an agent if the agent has successfully started tracing. BUG= 774728 Change-Id: I79801d5c66a454ee3bb1ea4c6729039e81c61a82 Reviewed-on: https://chromium-review.googlesource.com/721967 Commit-Queue: Ehsan Chiniforooshan <chiniforooshan@chromium.org> Reviewed-by: oysteine <oysteine@chromium.org> Cr-Commit-Position: refs/heads/master@{#509439} [modify] https://crrev.com/6588f9b2158846757d9da3ee30ed386e845e64cc/content/browser/tracing/cros_tracing_agent.cc [modify] https://crrev.com/6588f9b2158846757d9da3ee30ed386e845e64cc/content/browser/tracing/etw_tracing_agent_win.cc [modify] https://crrev.com/6588f9b2158846757d9da3ee30ed386e845e64cc/services/resource_coordinator/tracing/agent_registry.cc [modify] https://crrev.com/6588f9b2158846757d9da3ee30ed386e845e64cc/services/resource_coordinator/tracing/agent_registry.h [modify] https://crrev.com/6588f9b2158846757d9da3ee30ed386e845e64cc/services/resource_coordinator/tracing/coordinator.cc
,
Oct 18 2017
Your change meets the bar and is auto-approved for M63. Please go ahead and merge the CL to branch 3239 manually. Please contact milestone owner if you have questions. Owners: cmasso@(Android), cmasso@(iOS), gkihumba@(ChromeOS), govind@(Desktop) For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Oct 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/096dedec7720e1208ef9e21e74758df0c5ad8cd8 commit 096dedec7720e1208ef9e21e74758df0c5ad8cd8 Author: Ting-Yuan Huang <laszio@chromium.org> Date: Wed Oct 18 18:45:50 2017 Revert "telemetry_AFDOGenerate: update the benchmark set" This reverts commit a87fe74bb7075356af128b703077f35fce520bcc. Reason for revert: It was a temporary change so that we can verify the fix to chromium:773561. Reverting it since chromium:773561 is verified. Original change's description: > telemetry_AFDOGenerate: update the benchmark set > > Most of the other benchmarks doesn't work due to some telemetry issue. > > BUG= chromium:774728 > TEST=test_that > > Change-Id: I4738b8dd2830cc2dd70227b018ab66704f0e5dc9 > Reviewed-on: https://chromium-review.googlesource.com/720247 > Commit-Queue: Ting-Yuan Huang <laszio@chromium.org> > Tested-by: Ting-Yuan Huang <laszio@chromium.org> > Trybot-Ready: Ting-Yuan Huang <laszio@chromium.org> > Reviewed-by: Luis Lozano <llozano@chromium.org> Bug: chromium:774728 Change-Id: Ic2be12b0cfec76a3b0d536dbd2463ea6e3023acb Reviewed-on: https://chromium-review.googlesource.com/721861 Commit-Ready: Ting-Yuan Huang <laszio@chromium.org> Tested-by: Ting-Yuan Huang <laszio@chromium.org> Reviewed-by: Manoj Gupta <manojgupta@chromium.org> (cherry picked from commit 782c0ce4c1dab05e55293798b828580ff2ceb7d1) Reviewed-on: https://chromium-review.googlesource.com/724525 Reviewed-by: Luis Lozano <llozano@chromium.org> Commit-Queue: Ting-Yuan Huang <laszio@chromium.org> Trybot-Ready: Ting-Yuan Huang <laszio@chromium.org> [modify] https://crrev.com/096dedec7720e1208ef9e21e74758df0c5ad8cd8/server/site_tests/telemetry_AFDOGenerate/telemetry_AFDOGenerate.py
,
Oct 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f47ffe18e1893497cea45a5da161950b3501a4ec commit f47ffe18e1893497cea45a5da161950b3501a4ec Author: Ehsan Chiniforooshan <chiniforooshan@chromium.org> Date: Wed Oct 18 18:51:26 2017 Tracing: keep track of agents that are tracing Instead of asking all agents to keep track of whether they are currently tracing or not and bail out of StopAndFlush if they are not tracing, the coordinator only sends StopAndFlush to an agent if the agent has successfully started tracing. BUG= 774728 Change-Id: I79801d5c66a454ee3bb1ea4c6729039e81c61a82 Reviewed-on: https://chromium-review.googlesource.com/721967 Commit-Queue: Ehsan Chiniforooshan <chiniforooshan@chromium.org> Reviewed-by: oysteine <oysteine@chromium.org> Cr-Original-Commit-Position: refs/heads/master@{#509439}(cherry picked from commit 6588f9b2158846757d9da3ee30ed386e845e64cc) Reviewed-on: https://chromium-review.googlesource.com/726539 Reviewed-by: Ehsan Chiniforooshan <chiniforooshan@chromium.org> Cr-Commit-Position: refs/branch-heads/3239@{#55} Cr-Branched-From: adb61db19020ed8ecee5e91b1a0ea4c924ae2988-refs/heads/master@{#508578} [modify] https://crrev.com/f47ffe18e1893497cea45a5da161950b3501a4ec/content/browser/tracing/cros_tracing_agent.cc [modify] https://crrev.com/f47ffe18e1893497cea45a5da161950b3501a4ec/content/browser/tracing/etw_tracing_agent_win.cc [modify] https://crrev.com/f47ffe18e1893497cea45a5da161950b3501a4ec/services/resource_coordinator/tracing/agent_registry.cc [modify] https://crrev.com/f47ffe18e1893497cea45a5da161950b3501a4ec/services/resource_coordinator/tracing/agent_registry.h [modify] https://crrev.com/f47ffe18e1893497cea45a5da161950b3501a4ec/services/resource_coordinator/tracing/coordinator.cc
,
Oct 18 2017
,
Jan 22 2018
,
Jan 23 2018
|
|||||||||||||
►
Sign in to add a comment |
|||||||||||||
Comment 1 by laszio@chromium.org
, Oct 13 2017