Oh whoops, found the test log.


Traceback (most recent call last):
  File "c:\b\s\w\irmpnipj\third_party\catapult\telemetry\telemetry\internal\story_runner.py", line 87, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "c:\b\s\w\irmpnipj\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "c:\b\s\w\irmpnipj\third_party\catapult\telemetry\telemetry\page\shared_page_state.py", line 299, in RunStory
    self._current_page.Run(self)
  File "c:\b\s\w\irmpnipj\third_party\catapult\telemetry\telemetry\page\__init__.py", line 111, in Run
    self.RunPageInteractions(action_runner)
  File "c:\b\s\w\irmpnipj\tools\perf\page_sets\system_health\long_running_stories.py", line 31, in RunPageInteractions
    action_runner.MeasureMemory()
  File "c:\b\s\w\irmpnipj\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "c:\b\s\w\irmpnipj\third_party\catapult\telemetry\telemetry\internal\actions\action_runner.py", line 160, in MeasureMemory
    raise exceptions.Error('Unable to obtain memory dump')
Error: Unable to obtain memory dump

c:\b\s\w\irmpnipj\third_party\catapult\telemetry\telemetry\internal\image_processing\_bitmap.py:152: UserWarning: Using pure python png decoder, which could be very slow. To speed up, consider installing numpy & cv2 (OpenCV).
  'Using pure python png decoder, which could be very slow. To speed up, '
INFO:root:Chrome version: 2938
ERROR:root:Error when deleting c:\b\s\w\itl0obwu\tmpwn5dql, will try again at exit.
Traceback (most recent call last):
  File "c:\b\s\w\irmpnipj\third_party\catapult\telemetry\telemetry\internal\platform\tracing_controller_backend.py", line 272, in CollectAgentTraceData
    os.remove(self._trace_log)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\b\\s\\w\\itl0obwu\\tmpwn5dql'
INFO:root:Successfully shut down browser cooperatively
[  FAILED  ] long_running:tools:gmail-foreground (49987 ms)

John, does this look like a test runner device issue?

No, this isn't an android test runner issue. It may be a telemetry issue of some kind?

Yeah, sorry. Not android related :) 

There are two runs of long_running:tools:gmail-foreground in the same step. Not sure why.

A later build fails:

INFO:root:*************** BROWSER STANDARD OUTPUT ***************
INFO:root:
INFO:root:*********** END OF BROWSER STANDARD OUTPUT ************
INFO:root:********************* BROWSER LOG *********************
INFO:root:No log file
INFO:root:***************** END OF BROWSER LOG ******************
WARNING:root:Either tab has crashed or browser does not support taking tab screenshot. Skip taking screenshot on failure.
Traceback (most recent call last):
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\story_runner.py", line 87, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\page\shared_page_state.py", line 299, in RunStory
    self._current_page.Run(self)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\page\__init__.py", line 111, in Run
    self.RunPageInteractions(action_runner)
  File "c:\b\s\w\irhbq1uk\tools\perf\page_sets\system_health\system_health_story.py", line 112, in RunPageInteractions
    self._DidLoadDocument(action_runner)
  File "c:\b\s\w\irhbq1uk\tools\perf\page_sets\system_health\browsing_stories.py", line 74, in _DidLoadDocument
    self._NavigateBack(action_runner)
  File "c:\b\s\w\irhbq1uk\tools\perf\page_sets\system_health\browsing_stories.py", line 44, in _NavigateBack
    self._WaitForNavigation(action_runner)
  File "c:\b\s\w\irhbq1uk\tools\perf\page_sets\system_health\browsing_stories.py", line 27, in _WaitForNavigation
    action_runner.WaitForNavigate()
  File "c:\b\s\w\irhbq1uk\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\actions\action_runner.py", line 187, in WaitForNavigate
    time_left_in_seconds)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\browser\web_contents.py", line 88, in WaitForDocumentReadyStateToBeInteractiveOrBetter
    'document.readyState == "complete"', timeout)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\browser\web_contents.py", line 129, in WaitForJavaScriptExpression
    py_utils.WaitFor(IsJavaScriptExpressionTrue, timeout)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\common\py_utils\py_utils\__init__.py", line 120, in WaitFor
    res = condition()
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\browser\web_contents.py", line 122, in IsJavaScriptExpressionTrue
    return bool(self.EvaluateJavaScript(expr))
  File "c:\b\s\w\irhbq1uk\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\browser\web_contents.py", line 193, in EvaluateJavaScript
    expr, context_id=None, timeout=timeout)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\browser\web_contents.py", line 221, in EvaluateJavaScriptInContext
    expr, context_id=context_id, timeout=timeout)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py", line 36, in inner
    return func(inspector_backend, *args, **kwargs)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py", line 211, in EvaluateJavaScript
    return self._runtime.Evaluate(expr, context_id, timeout)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_runtime.py", line 45, in Evaluate
    res = self._inspector_websocket.SyncRequest(request, timeout)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_websocket.py", line 110, in SyncRequest
    res = self._Receive(timeout)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_websocket.py", line 166, in _Receive
    self._HandleNotification(result)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_websocket.py", line 179, in _HandleNotification
    self._domain_handlers[domain_name](result)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py", line 365, in _HandleInspectorDomainNotification
    raise exception
DevtoolsTargetCrashException: Devtools target crashed
********************************************************************************
(c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py:418 _AddDebuggingInformation) Received a socket error in the browser connection and the tab no longer exists. The tab probably crashed.
********************************************************************************
(c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py:419 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:50138/devtools/page/427a905f-c0c0-4561-9960-ad103d1ed230
Found Minidump: False
Stack Trace:
********************************************************************************
	No crash dump found.
********************************************************************************
Standard output:
********************************************************************************
********************************************************************************
System log:
(Not implemented)

INFO:root:Chrome version: 2938
ERROR:root:Error when deleting c:\b\s\w\itxderej\tmpigue3w, will try again at exit.
Traceback (most recent call last):
  File "c:\b\s\w\irhbq1uk\third_party\catapult\telemetry\telemetry\internal\platform\tracing_controller_backend.py", line 272, in CollectAgentTraceData
    os.remove(self._trace_log)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\b\\s\\w\\itxderej\\tmpigue3w'
INFO:root:Successfully shut down browser cooperatively
[  FAILED  ] browse:news:reddit (24752 ms)

These look like flakes...

I'm seeing this on the following (most look like #3)

Builders failed on: 
- Win 8 Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Win%208%20Perf
- Win 7 Intel GPU Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Win%207%20Intel%20GPU%20Perf
- Win 7 ATI GPU Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Win%207%20ATI%20GPU%20Perf
- Win 10 Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Win%2010%20Perf
- Win 7 x64 Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Win%207%20x64%20Perf

I'm seeing these errors after these commit positions:
438483
438167
438319
438165

I got these numbers by going through builds, and looking at the build right before the test started failing. 

Nothing really seems to line up....

https://chromium.googlesource.com/chromium/src/+log/9095a6d305b5ce5d2e7a781c3e3ebd628410913e..HEAD is technically all the revisions at fault here, but it's a lot to go through. Let me try to start a bisect.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8993247675636070240

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_10_perf_bisect/builds/811
Failure reason: the build has failed due to infrastructure failure.

Blocking on crbug.omc/676416 for failing to post to dashboard.