It appears that some story errors may cause trace collection to be skipped, causing the next story to fail with an AssertionError, and the rest of the benchmark to bail out.


For example in:
https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus6_Perf%2F453%2F%2B%2Frecipes%2Fsteps%2Fsystem_health.common_mobile_on_Android%2F0%2Fstdout

We have:

[ RUN      ] browse:social:facebook_infinite_scroll@{'case': 'browse', 'group': 'social'}

... snip snip snip ...

  File "/b/swarming/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_socket.py", line 83, in recv
    raise WebSocketTimeoutException(message)
TimeoutException: 
********************************************************************************
(/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:470 _ConvertExceptionFromInspectorWebsocket) The app is probably crashed:
Found Minidump: True
Stack Trace:
********************************************************************************
	UI dump
	- (no package):
	  - (no id)
	- org.chromium.chrome:
	  - (no id)
	  - android:id/button1['Allow']
	  - android:id/button2['Block']
	  - android:id/content
	  - org.chromium.chrome:id/action_bar_root
	  - org.chromium.chrome:id/buttonPanel
	  - org.chromium.chrome:id/custom
	  - org.chromium.chrome:id/customPanel
	  - org.chromium.chrome:id/parentPanel
	  - org.chromium.chrome:id/text['m.facebook.com wants to send you notifications.']
	********************************************************************************

... snip snip snip ...

[  FAILED  ] browse:social:facebook_infinite_scroll@{'case': 'browse', 'group': 'social'} (143624 ms)
[ RUN      ] browse:media:flickr_infinite_scroll@{'case': 'browse', 'group': 'media'}

... snip snip snip ...

Traceback (most recent call last):
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 97, in _RunStoryAndProcessErrorIfNeeded
    test.WillRunStory(state.platform)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py", line 280, in WillRunStory
    platform.tracing_controller.StartTracing(self._tbm_options.config)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/core/tracing_controller.py", line 43, in StartTracing
    self._tracing_controller_backend.StartTracing(tracing_config, timeout)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 91, in StartTracing
    started = agent.StartAgentTracing(config, timeout)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 124, in StartAgentTracing
    started_devtools_tracing = self._StartDevToolsTracing(config, timeout)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 101, in _StartDevToolsTracing
    client.StartChromeTracing(config, timeout)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 361, in StartChromeTracing
    trace_config.chrome_trace_config, timeout)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend.py", line 116, in StartTracing
    assert not self._can_collect_data, 'Data not collected from last trace.'
AssertionError: Data not collected from last trace.