We have recently started seeing many (but not all) of our attempts to run the page_cycler_v2.typical_25 benchmark fail with the following types of errors:


[ RUN      ] page_cycler_v2.typical_25/http://www.nick.com/games_cold@{'cache_temperature': 'cold'}
(WARNING) 2018-12-13 15:13:26,641 inspector_log._OnMessage:26  DevTools console [network]: Failed to load resource: net::ERR_FAILED chrome-search://local-ntp/promo.js
(WARNING) 2018-12-13 15:13:26,641 inspector_log._OnMessage:26  DevTools console [network]: Failed to load resource: net::ERR_FAILED chrome-search://local-ntp/one-google.js
(WARNING) 2018-12-13 15:13:26,764 inspector_log._OnMessage:26  DevTools console [network]: Failed to load resource: the server responded with a status of 404 (Not Found) http://does.not.exist/
(WARNING) 2018-12-13 15:13:27,849 inspector_log._OnMessage:26  DevTools console [network]: Failed to load resource: the server responded with a status of 404 (Not Found) http://ad.doubleclick.net/N1303/adj/nick.nol/atf_j_s/games/_mn;sec0=games;sec1=_mn;pos=atf;tag=adj;mtype=standard;demo=D;cat=1;site=nick;sz=6x6;tile=1;dcopt=ist;!category=nick;u=sec0_games-sec1__mn;ord=500815344087944060?
...
(ERROR) 2018-12-13 15:14:05,491 story_runner.ProcessError:119  Possibly handleable error. Will try to restart shared state
Traceback (most recent call last):
  File "/mnt/host/source/.cache/distfiles/target/chrome-src-internal/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 147, in _RunStoryAndProcessErrorIfNeeded
    test.Measure(state.platform, results)
  File "/mnt/host/source/.cache/distfiles/target/chrome-src-internal/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py", line 268, in Measure
    trace_result, _ = platform.tracing_controller.StopTracing()
  File "/mnt/host/source/.cache/distfiles/target/chrome-src-internal/src/third_party/catapult/telemetry/telemetry/core/tracing_controller.py", line 53, in StopTracing
    return self._tracing_controller_backend.StopTracing()
  File "/mnt/host/source/.cache/distfiles/target/chrome-src-internal/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 157, in StopTracing
    '\n'.join(raised_exception_messages))
TracingException: Exceptions raised when trying to stop tracing:
Traceback (most recent call last):
  File "/mnt/host/source/.cache/distfiles/target/chrome-src-internal/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 145, in StopTracing
    agent.CollectAgentTraceData(builder)
  File "/mnt/host/source/.cache/distfiles/target/chrome-src-internal/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 248, in CollectAgentTraceData
    '\n'.join(raised_exception_messages))
ChromeTracingStoppedError: Exceptions raised when trying to collect Chrome devtool tracing:
Error when collecting Chrome tracing on devtools at port 33681:
Traceback (most recent call last):
  File "/mnt/host/source/.cache/distfiles/target/chrome-src-internal/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 237, in CollectAgentTraceData
    client.CollectChromeTracingData(trace_data_builder)
  File "/mnt/host/source/.cache/distfiles/target/chrome-src-internal/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 489, in CollectChromeTracingData
    self._tracing_backend.CollectTraceData(trace_data_builder, timeout)
  File "/mnt/host/source/.cache/distfiles/target/chrome-src-internal/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 237, in CollectTraceData
    self._CollectTracingData(trace_data_builder, timeout)
  File "/mnt/host/source/.cache/distfiles/target/chrome-src-internal/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 265, in _CollectTracingData
    traceback.format_exc())
TracingUnrecoverableException: Exception raised while collecting tracing data:
Traceback (most recent call last):
  File "/mnt/host/source/.cache/distfiles/target/chrome-src-internal/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 258, in _CollectTracingData
    self._inspector_websocket.DispatchNotifications(timeout)
  File "/mnt/host/source/.cache/distfiles/target/chrome-src-internal/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 156, in DispatchNotifications
    self._Receive(timeout)
  File "/mnt/host/source/.cache/distfiles/target/chrome-src-internal/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 181, in _Receive
    raise WebSocketException(err)
WebSocketException: WebSocketException of type <class 'websocket._exceptions.WebSocketConnectionClosedException'>. Error message: Connection is already closed.

I am attaching a full log for one of our runs to this bug.

The errors occur whether we launch the tests remotely from a Chrome OS chroot: 

$ python ~/trunk/.cache/distfiles/target/chrome-src-internal/src/tools/perf/run_benchmark
 --browser=cros-chrome --remote=chromeos2-row9-rack7-host11.cros --output-dir="/tmp/test_that_results_B8ea0X/results-1-telemetry_Crosperf/telemetry_Crosperf/results" --output-format=chartjson  page_cycler_v2.typical_25

Or locally, directly on the Chromebook:

$ python /usr/local/telemetry/src/tools/perf/run_benchmark --browser=system --output-format=chartjson  page_cycler_v2.typical_25


 

Deleted: page_cycler_v2.typical_25.log 598 KB

page_cycler_v2.typical_25.log 598 KB View Download