📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/11a5136fe40000

😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/11a5136fe40000

All of the runs failed. The most common error (10/20 runs) was:
HTTPException: HTTP status code 400: {"error": {"message": "CIPD package path is required. Use \".\" to install to run dir."}}

Dave, Shutao, can you help with the failing bisect?

+Vadim for help with CIPD

This error is coming from Swarming. Pinpoint sent a task request to Swarming that had 'packages' field defined, in which at least one entry didn't have 'path' set.

Filed 908903 to fix Pinpoint issue. In the mean time I will investigate to see if we can figure out what is wrong with blink_perf.

The error is:

(INFO) 2018-11-27 11:17:56,324 action_runner._RunAction:61  START Page Action: NavigateAction(http://127.0.0.1:45240/events/hit-test-lots-of-layers.html)
(INFO) 2018-11-27 11:17:56,535 action_runner._RunAction:64  DONE Page Action: NavigateAction
(ERROR) 2018-11-27 11:19:18,470 story_runner.ProcessError:106  Handleable error
Traceback (most recent call last):
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 132, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 300, in RunStory
    self._current_page.Run(self)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/__init__.py", line 96, in Run
    shared_state.page_test.DidNavigateToPage(self, current_tab)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/tools/perf/benchmarks/blink_perf.py", line 247, in DidNavigateToPage
    tab.WaitForJavaScriptCondition('testRunner.isWaitingForTelemetry')
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 245, in WaitForJavaScriptCondition
    return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 307, in WaitForJavaScriptCondition
    self.GetCurrentConsoleOutputBuffer())
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 41, in Inner
    inspector_backend._ConvertExceptionFromInspectorWebsocket(e)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 38, in Inner
    return func(inspector_backend, *args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 196, in GetCurrentConsoleOutputBuffer
    return self._console.GetCurrentConsoleOutputBuffer(timeout)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_console.py", line 38, in GetCurrentConsoleOutputBuffer
    self._inspector_websocket.DispatchNotifications(timeout)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 156, in DispatchNotifications
    self._Receive(timeout)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 181, in _Receive
    raise WebSocketException(err)
TimeoutException: Timeout after 60s while waiting for JavaScript:testRunner.isWaitingForTelemetry

+some blink_perf Benchmark owners FYI.

Revision range link is http://test-results.appspot.com/revision_range?start=606083&end=607096

I will take a look

This is from screenshot on the bog:
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/6965035e-f3e7-11e8-837f-0242ac110003

The error shown is:
net::ERR_CONNECTION_REFUSED

I am suspecting that this is due to https://chromium-review.googlesource.com/c/chromium/src/+/1327801 (in the range of first failed build: https://chromium.googlesource.com/chromium/src/+log/b316cbc663d9db6673b2114f16735c2e1c88863e%5E..93925aac015056c95114b744e3fc27594be67a68?pretty=fuller&n=)

Weirdly I can't reproduce the problem with my Pixel2 device (userdebug 9 PQ2A) locally. 

After fetching the isolate, I run:
./testing/test_env.py third_party/catapult/devil/devil/android/tools/system_app.py remove --package com.android.webview com.google.android.webview -v -- tools/perf/run_benchmark --browser=android-webview --webview-embedder-apk=out/Release/apks/SystemWebViewShell.apk blink_perf.canvas

Given that I currently can't reproduce locally, getting bisect to work again is our safest choice. I filed  issue 910324  to block this bug.

Meanwhile, I will try to flash my phone to different Android version & trying to reproduce again

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/114b0997e40000

I restarted the pinpoint job since Dave says that Pinpoint should be working now in https://bugs.chromium.org/p/chromium/issues/detail?id=908903#c5

📍 Found a significant difference after 1 commit.
https://pinpoint-dot-chromeperf.appspot.com/job/114b0997e40000

Implicitly bypass localhost when proxying requests. by eroman@chromium.org
https://chromium.googlesource.com/chromium/src/+/da790f920bbc169a6805a4fb83b4c2ab09532d91
Failure rate: 0 → 1 (+1)

Understanding performance regressions:
  http://g.co/ChromePerformanceRegressions

Benchmark documentation link:
  https://bit.ly/blink-perf-benchmarks

eroman: will you be able to take a look at this soon? After your CL, all blink_perf benchmarks are failing on Nexus 5X. crouleau, will you be able to help if eroman has questions?

Is this still an issue? I expect this is the same root cause as  Issue 906540 , which should be fixed by https://crrev.com/612514

My fix was appears in build 3797. It shows that all the failures save a handful (4 unexpected failures, and 3 flakes) go away:

Build with fix fix (crrev.com/612514)

https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/Android%20Nexus5X%20WebView%20Perf/3797

So I believe the WebView impact of crrev.com/606112 has been addressed, and the bug as titled can be closed "All blink_perf test failing".

As for the remaining failures, could those be unrelated regressions that crept in during the outage period?

@nednguyen could you take a look at remainder of failures, and see if we can close this bug now?

Thanks!

Thanks! I think this bug can be safely closed now. The blink_perf.css/CustomPropertiesVarAlias.html is webview crash (https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/48581272-f479-11e8-942f-0242ac110003)

For documentation, the fix of this bug is in https://chromium.googlesource.com/chromium/src/+/4bc3f6d6ec942c97293e303f469ccdc4bf551fc2 by eroman@