New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 766908 link

Starred by 2 users

Issue metadata

Status: Duplicate
Owner: ----
Closed: Sep 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

telemetry benchmarks failing on Chrome OS.

Project Member Reported by llozano@chromium.org, Sep 20 2017

Issue description

Chrome Version: ToT
OS: Chrome

The AFDO collection process has been failing for several days. 
We thought this was related to issue:

but that issue was fixed and this is still failing.

Command:

python /var/cache/chromeos-cache/distfiles/target/chrome-src-internal/src/tools/perf/run_benchmark --verbose --browser=cros-chrome --output-format=chartjson --output-dir=/var/cache/chromeos-cache/distfiles/target/chrome-src-internal/src --remote=100.96.193.32 --pageset-repeat=1 page_cycler_v2.typical_25

Note that a very similar error happens for benchmark "speedometer".
I have not tried others.

The error:

[  FAILED  ] http://www.fda.gov@{'cache_temperature': 'cold'} (126941 ms)
[ RUN      ] http://www.fda.gov@{'cache_temperature': 'warm'}
(INFO) 2017-09-19 17:56:59,969 tracing_backend.StartTracing:137  Start Tracing Request: {'params': {'transferMode': 'ReturnAsStream', 'traceConfig': {'recordMode': 'recordAsMuchAsPossible', 'includedCategories': ['trace_event_overhead', 'loading', 'webkit.console', 'benchmark', 'toplevel', 'blink.user_timing', 'naviga
tion', 'blink.console']}}, 'method': 'Tracing.start'}
(INFO) 2017-09-19 17:56:59,982 ts_proxy_server._IssueCommand:93  Issuing command to ts_proxy_server: set rtt 0
(INFO) 2017-09-19 17:56:59,983 ts_proxy_server._IssueCommand:93  Issuing command to ts_proxy_server: set inkbps 0
(INFO) 2017-09-19 17:56:59,983 ts_proxy_server._IssueCommand:93  Issuing command to ts_proxy_server: set outkbps 0
(INFO) 2017-09-19 17:56:59,984 cache_temperature.EnsurePageCacheTemperature:59  PageCacheTemperature: warm
(INFO) 2017-09-19 17:58:48,369 browser.DumpStateUponFailure:391  *************** BROWSER STANDARD OUTPUT ***************
(INFO) 2017-09-19 17:58:48,370 browser.DumpStateUponFailure:393  Cannot get standard output on CrOS
(INFO) 2017-09-19 17:58:48,370 browser.DumpStateUponFailure:396  *********** END OF BROWSER STANDARD OUTPUT ************
(INFO) 2017-09-19 17:58:48,370 browser.DumpStateUponFailure:398  ********************* BROWSER LOG *********************
(INFO) 2017-09-19 17:58:48,370 browser.DumpStateUponFailure:400  No log file
(INFO) 2017-09-19 17:58:48,370 browser.DumpStateUponFailure:403  ***************** END OF BROWSER LOG ******************
Traceback (most recent call last):
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 105, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 330, in RunStory
    self._current_page.Run(self)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 108, in Run
    shared_state.page_test.RunNavigateSteps(self, current_tab)
  File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/page/legacy_page_test.py", line 152, in RunNavigateSteps
    page.RunNavigateSteps(action_runner)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 117, in RunNavigateSteps
    url, script_to_evaluate_on_commit=self.script_to_evaluate_on_commit)
  File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 187, in Navigate
    timeout_in_seconds=timeout_in_seconds))
  File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 63, in _RunAction
    action.RunAction(self._tab)
  File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/actions/navigate.py", line 28, in RunAction
    tab.WaitForFrameToBeDisplayed()
  File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 119, in WaitForFrameToBeDisplayed
    timeout=timeout)
  File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 239, in WaitForJavaScriptCondition
    return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/ssd2/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 291, in WaitForJavaScriptCondition
    e.message + '\n' + debug_message)
TimeoutException: Timed out while waiting 90s for IsJavaScriptExpressionTrue.
Console output:


(INFO) 2017-09-19 17:58:58,095 chrome_tracing_agent.RecordClockSyncMarker:183  Chrome version: 3218
(INFO) 2017-09-19 17:58:58,514 trace_data.Serialize:191  Trace sizes in bytes: {'traceEvents': 1930472, 'telemetry': 41460, 'tabIds': 40}
[  FAILED  ] http://www.fda.gov@{'cache_temperature': 'warm'} (124450 ms)

 
Cc: deanliao@chromium.org
isn't the Chrome OS performance detective seen these failures too?

cc'ing last perf detective 
deanliao@
Labels: Build-Toolchain
Cc: cros-perf-detectives@google.com
I meet this issue too when I run below command on IA device R62-9954.0.0 :

./run_benchmark --remote=10.239.115.15 --browser=cros-chrome --reset-results --output-format=html page_cycler_v2.typical_25

or 

./run_benchmark --remote=10.239.115.15 --browser=cros-chrome --reset-results --output-format=html smoothness.top_25_smooth

or 

./run_benchmark --remote=10.239.115.15 --browser=cros-chrome --reset-results --output-format=html --also-run-disabled-tests  tab_switching.typical_25

The error is below:

[ RUN      ] multitab:misc:typical24@{'case': 'multitab', 'group': 'misc'}
(WARNING) 2017-09-20 15:12:55,240 multi_tab_stories.RunNavigateSteps:37  WaitForNetworkQuiescence() timeout, url[15]: http://premierleague.com
Traceback (most recent call last):
  File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 105, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 331, in RunStory
    self._current_page.Run(self)
  File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 112, in Run
    self.RunPageInteractions(action_runner)
  File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/tools/perf/page_sets/system_health/system_health_story.py", line 142, in RunPageInteractions
    self._DidLoadDocument(action_runner)
  File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/tools/perf/page_sets/system_health/multi_tab_stories.py", line 42, in _DidLoadDocument
    tab.WaitForFrameToBeDisplayed()
  File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 104, in WaitForFrameToBeDisplayed
    timeout=timeout)
  File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 192, in WaitForJavaScriptCondition
    return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs)
  File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/home/stang23/chromeos/chromiumos-3/chrome_root/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 287, in WaitForJavaScriptCondition
    e.message + '\n' + debug_message)
TimeoutException: Timed out while waiting 90s for IsJavaScriptExpressionTrue.
Console output:


[  FAILED  ] multitab:misc:typical24@{'case': 'multitab', 'group': 'misc'} (202283 ms)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ]  multitab:misc:typical24@{'case': 'multitab', 'group': 'misc'}

1 FAILED TEST


Please add jayeeta.bandyopadhyay@intel.com, lingyun.cai@intel.com, rachna.doggalli@intel.com and me into the cc list. Thanks a lot.

Comment 6 by hctsai@chromium.org, Sep 20 2017

Tracked on crbug.com/763834. 

The bisection process figured out that the culprit is on the Chrome side. But the current bisection tool couldn't run the bisection process due to some command line argument change. Keep looking into it.
Mergedinto: 763834
Status: Duplicate (was: Untriaged)
On R63-9971.0.0, this issue is fixed.  Thanks a lot

Sign in to add a comment