telemetry benchmarks failing on Chrome OS. |
||||
Issue description
Chrome Version: ToT
OS: Chrome
The AFDO collection process has been failing for several days.
We thought this was related to issue:
but that issue was fixed and this is still failing.
Command:
python /var/cache/chromeos-cache/distfiles/target/chrome-src-internal/src/tools/perf/run_benchmark --verbose --browser=cros-chrome --output-format=chartjson --output-dir=/var/cache/chromeos-cache/distfiles/target/chrome-src-internal/src --remote=100.96.193.32 --pageset-repeat=1 page_cycler_v2.typical_25
Note that a very similar error happens for benchmark "speedometer".
I have not tried others.
The error:
[ FAILED ] http://www.fda.gov@{'cache_temperature': 'cold'} (126941 ms)
[ RUN ] http://www.fda.gov@{'cache_temperature': 'warm'}
(INFO) 2017-09-19 17:56:59,969 tracing_backend.StartTracing:137 Start Tracing Request: {'params': {'transferMode': 'ReturnAsStream', 'traceConfig': {'recordMode': 'recordAsMuchAsPossible', 'includedCategories': ['trace_event_overhead', 'loading', 'webkit.console', 'benchmark', 'toplevel', 'blink.user_timing', 'naviga
tion', 'blink.console']}}, 'method': 'Tracing.start'}
(INFO) 2017-09-19 17:56:59,982 ts_proxy_server._IssueCommand:93 Issuing command to ts_proxy_server: set rtt 0
(INFO) 2017-09-19 17:56:59,983 ts_proxy_server._IssueCommand:93 Issuing command to ts_proxy_server: set inkbps 0
(INFO) 2017-09-19 17:56:59,983 ts_proxy_server._IssueCommand:93 Issuing command to ts_proxy_server: set outkbps 0
(INFO) 2017-09-19 17:56:59,984 cache_temperature.EnsurePageCacheTemperature:59 PageCacheTemperature: warm
(INFO) 2017-09-19 17:58:48,369 browser.DumpStateUponFailure:391 *************** BROWSER STANDARD OUTPUT ***************
(INFO) 2017-09-19 17:58:48,370 browser.DumpStateUponFailure:393 Cannot get standard output on CrOS
(INFO) 2017-09-19 17:58:48,370 browser.DumpStateUponFailure:396 *********** END OF BROWSER STANDARD OUTPUT ************
(INFO) 2017-09-19 17:58:48,370 browser.DumpStateUponFailure:398 ********************* BROWSER LOG *********************
(INFO) 2017-09-19 17:58:48,370 browser.DumpStateUponFailure:400 No log file
(INFO) 2017-09-19 17:58:48,370 browser.DumpStateUponFailure:403 ***************** END OF BROWSER LOG ******************
Traceback (most recent call last):
File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 105, in _RunStoryAndProcessErrorIfNeeded
state.RunStory(results)
File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 330, in RunStory
self._current_page.Run(self)
File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 108, in Run
shared_state.page_test.RunNavigateSteps(self, current_tab)
File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/page/legacy_page_test.py", line 152, in RunNavigateSteps
page.RunNavigateSteps(action_runner)
File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 117, in RunNavigateSteps
url, script_to_evaluate_on_commit=self.script_to_evaluate_on_commit)
File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 187, in Navigate
timeout_in_seconds=timeout_in_seconds))
File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 63, in _RunAction
action.RunAction(self._tab)
File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/actions/navigate.py", line 28, in RunAction
tab.WaitForFrameToBeDisplayed()
File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 119, in WaitForFrameToBeDisplayed
timeout=timeout)
File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 239, in WaitForJavaScriptCondition
return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs)
File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 291, in WaitForJavaScriptCondition
e.message + '\n' + debug_message)
TimeoutException: Timed out while waiting 90s for IsJavaScriptExpressionTrue.
Console output:
(INFO) 2017-09-19 17:58:58,095 chrome_tracing_agent.RecordClockSyncMarker:183 Chrome version: 3218
(INFO) 2017-09-19 17:58:58,514 trace_data.Serialize:191 Trace sizes in bytes: {'traceEvents': 1930472, 'telemetry': 41460, 'tabIds': 40}
[ FAILED ] http://www.fda.gov@{'cache_temperature': 'warm'} (124450 ms)
,
Sep 20 2017
,
Sep 20 2017
,
Sep 20 2017
I meet this issue too when I run below command on IA device R62-9954.0.0 :
./run_benchmark --remote=10.239.115.15 --browser=cros-chrome --reset-results --output-format=html page_cycler_v2.typical_25
or
./run_benchmark --remote=10.239.115.15 --browser=cros-chrome --reset-results --output-format=html smoothness.top_25_smooth
or
./run_benchmark --remote=10.239.115.15 --browser=cros-chrome --reset-results --output-format=html --also-run-disabled-tests tab_switching.typical_25
The error is below:
[ RUN ] multitab:misc:typical24@{'case': 'multitab', 'group': 'misc'}
(WARNING) 2017-09-20 15:12:55,240 multi_tab_stories.RunNavigateSteps:37 WaitForNetworkQuiescence() timeout, url[15]: http://premierleague.com
Traceback (most recent call last):
File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 105, in _RunStoryAndProcessErrorIfNeeded
state.RunStory(results)
File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
return func(*args, **kwargs)
File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 331, in RunStory
self._current_page.Run(self)
File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 112, in Run
self.RunPageInteractions(action_runner)
File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/tools/perf/page_sets/system_health/system_health_story.py", line 142, in RunPageInteractions
self._DidLoadDocument(action_runner)
File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/tools/perf/page_sets/system_health/multi_tab_stories.py", line 42, in _DidLoadDocument
tab.WaitForFrameToBeDisplayed()
File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
return func(*args, **kwargs)
File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 104, in WaitForFrameToBeDisplayed
timeout=timeout)
File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
return func(*args, **kwargs)
File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 192, in WaitForJavaScriptCondition
return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs)
File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
return func(*args, **kwargs)
File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 287, in WaitForJavaScriptCondition
e.message + '\n' + debug_message)
TimeoutException: Timed out while waiting 90s for IsJavaScriptExpressionTrue.
Console output:
[ FAILED ] multitab:misc:typical24@{'case': 'multitab', 'group': 'misc'} (202283 ms)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] multitab:misc:typical24@{'case': 'multitab', 'group': 'misc'}
1 FAILED TEST
,
Sep 20 2017
Please add jayeeta.bandyopadhyay@intel.com, lingyun.cai@intel.com, rachna.doggalli@intel.com and me into the cc list. Thanks a lot.
,
Sep 20 2017
Tracked on crbug.com/763834. The bisection process figured out that the culprit is on the Chrome side. But the current bisection tool couldn't run the bisection process due to some command line argument change. Keep looking into it.
,
Sep 20 2017
,
Sep 25 2017
On R63-9971.0.0, this issue is fixed. Thanks a lot |
||||
►
Sign in to add a comment |
||||
Comment 1 by llozano@chromium.org
, Sep 20 2017