First failure with this catapult roll:
https://codereview.chromium.org/2753063004

The only CL in that roll is this one:
https://codereview.chromium.org/2744373003

This CL only affects android, so this is likely to be a chrome problem.

First failing build on amd64:
https://uberchromegw.corp.google.com/i/chromiumos.chromium/builders/amd64-generic-telemetry/builds/11050

First failing build on x86:
https://uberchromegw.corp.google.com/i/chromiumos.chromium/builders/x86-generic-telemetry/builds/11993

The CLs they have in common are 178661-178686
It's one of the first 26 CLs in this list (anyone know how to do a commit log diff?):
https://chromium.googlesource.com/chromium/src/+log/bba421837cf80e6b786c736235f9ec10fa931302

Taking a look.

Thanks!

Looked at the build history at comment #2. Looks like the test which is actually failing is telemetry_UnitTests_perf, not telemetry_UnitTests_perf.

telemetry_UnitTests (the one mentioned in this bug description) seems somewhat flaky, but is mostly okay. OTOH, telemetry_UnitTests_perf always fails with the following two messages, probably depending on Chrome(?) version:


03/20 12:06:55.415 INFO |run_chromeos_tests:0052| benchmarks.system_health_smoke_test.load_tests() failed: ServiceException: 401 Anonymous users does not have storage.objects.get access to object chrome-partner-telemetry/7b49a582a4dcf33bd2cb7a7aebab3f25c70ee89e.
03/20 12:06:55.416 INFO |run_chromeos_tests:0052| 
03/20 12:06:55.504 WARNI|              test:0615| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 609, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/common_lib/test.py", line 817, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/common_lib/test.py", line 470, in execute
    dargs)
  File "/usr/local/autotest/common_lib/test.py", line 347, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/common_lib/test.py", line 380, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/tests/telemetry_UnitTests/telemetry_UnitTests.py", line 35, in run_once
    raise error.TestFail(error_str)
TestFail: The unit tests of /usr/local/telemetry/src/tools/perf failed.

 -or-

03/21 15:17:19.207 ERROR|      archive_info:0090| You either aren't authenticated or don't have permission to use the archives for this page set.
You may need to run gsutil config.
You can find instructions for gsutil config at: http://www.chromium.org/developers/telemetry/upload_to_cloud_storage
03/21 15:17:19.220 INFO |run_chromeos_tests:0052| benchmarks.system_health_smoke_test.load_tests() failed: Attempted to access a file from Cloud Storage but you have no configured credentials. To configure your credentials:
03/21 15:17:19.221 INFO |run_chromeos_tests:0052|   1. Run "HOME=/home/chromeos-test/ /usr/local/telemetry/src/third_party/catapult/third_party/gsutil/gsutil config" and follow its instructions.
03/21 15:17:19.222 INFO |run_chromeos_tests:0052|   2. If you have a @google.com account, use that account.
03/21 15:17:19.223 INFO |run_chromeos_tests:0052|   3. For the project-id, just enter 0.
03/21 15:17:19.318 WARNI|              test:0615| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 609, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/common_lib/test.py", line 817, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/common_lib/test.py", line 470, in execute
    dargs)
  File "/usr/local/autotest/common_lib/test.py", line 347, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/common_lib/test.py", line 380, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/tests/telemetry_UnitTests/telemetry_UnitTests.py", line 35, in run_once
    raise error.TestFail(error_str)
TestFail: The unit tests of /usr/local/telemetry/src/tools/perf failed.


To check if this is actually a Chrome issue, I built older Chrome (7b7a5330535b472ee9590af26b684a8b5dcefed6, r457794) which is outside the bad range mentioned in comment #2, downloaded x86-generic VM image from the builder, started the VM on my machine, deployed old Chrome, and ran telemetry_UnitTests_perf on VM, but it still failed with the same server error. I'm not familiar with telemetry, but I guess this is a server side or configuration issue.

+rnephew@ who fixed a similar issue recently at https://bugs.chromium.org/p/chromium/issues/detail?id=692278 . Do you have any idea on the cloud storage failure?

typo: I meant to say "the test which is actually failing is telemetry_UnitTests_perf, not telemetry_UnitTests."

> First failing build on amd64:
> https://uberchromegw.corp.google.com/i/chromiumos.chromium/builders/amd64-generic-telemetry/builds/11050
>
> First failing build on x86:
> https://uberchromegw.corp.google.com/i/chromiumos.chromium/builders/x86-generic-telemetry/builds/11993

Both builds (#11050 and #11993) introduced the same set of Chrome OS autotest CLs:
 https://chromium-review.googlesource.com/c/457096/
 https://chromium-review.googlesource.com/c/455331/
 https://chromium-review.googlesource.com/c/456592/
 https://chromium-review.googlesource.com/c/456591/

But the first 2 are only for ARC, and the rest are just reverts. They are probably unrelated to this failure.

Based on https://bugs.chromium.org/p/chromium/issues/detail?id=692278#c8 , let me reassign this to rnephew@. Randy, could you check if this is a Telemetry side issue?

These bots have been red for a week now

achuit@ - mephew@ is presumably out or swamped, is there anyone else who could investigate these?

I'm planning to take a look today or tomorrow.

video_ChromeCameraMJpegHWDecodeUsed autotest has a failure like this in tot (auron_paine, 9415.0.0).

03/30 15:13:30.305 WARNI|              test:0615| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 609, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/common_lib/test.py", line 823, in _call_test_function
    raise error.UnhandledTestFail(e)
UnhandledTestFail: Unhandled TimeoutException: 
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:477 _ConvertExceptionFromInspectorWebsocket) The app is probably crashed:

Found Minidump: False
Stack Trace:
********************************************************************************
	Cannot get stack trace on CrOS
********************************************************************************
Standard output:
********************************************************************************
	Cannot get standard output on CrOS
********************************************************************************
System log:
(Not implemented)
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:482 _ConvertExceptionFromInspectorWebsocket) Original exception:
timed out
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:503 _AddDebuggingInformation) Received a socket error in the browser connection and the tab no longer exists. The tab probably crashed.
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:504 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:60473/devtools/page/d788f5eb-dfc2-44e4-a904-ea0a76a9eec1
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 817, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/common_lib/test.py", line 470, in execute
    dargs)
  File "/usr/local/autotest/common_lib/test.py", line 347, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/common_lib/test.py", line 380, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/tests/video_ChromeRTCHWDecodeUsed/video_ChromeRTCHWDecodeUsed.py", line 79, in run_once
    histogram_verifier.verify(cr, histogram_name, histogram_bucket_val)
  File "/usr/local/autotest/cros/video/histogram_verifier.py", line 44, in verify
    sleep_interval=1)
  File "/usr/local/autotest/common_lib/site_utils.py", line 965, in poll_for_condition
    value = condition()
  File "/usr/local/autotest/cros/video/histogram_verifier.py", line 35, in loaded
    tab.Navigate('chrome://histograms/%s' % histogram_name)
  File "/usr/local/telemetry/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 228, in Navigate
    self._inspector_backend.Navigate(url, script_to_evaluate_on_commit, timeout)
  File "/usr/local/telemetry/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 40, in inner
    inspector_backend._ConvertExceptionFromInspectorWebsocket(e)
  File "/usr/local/telemetry/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 37, in inner
    return func(inspector_backend, *args, **kwargs)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 176, in Navigate
    self._page.Navigate(url, script_to_evaluate_on_commit, timeout)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_page.py", line 125, in Navigate
    self.WaitForNavigate(timeout)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_page.py", line 94, in WaitForNavigate
    self._inspector_websocket.DispatchNotifications(remaining_time)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 134, in DispatchNotifications
    self._Receive(timeout)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 149, in _Receive
    data = self._socket.recv()
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 293, in recv
    opcode, data = self.recv_data()
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 310, in recv_data
    opcode, frame = self.recv_data_frame(control_frame)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 323, in recv_data_frame
    frame = self.recv_frame()
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 357, in recv_frame
    return self.frame_buffer.recv_frame()
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 336, in recv_frame
    self.recv_header()
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 286, in recv_header
    header = self.recv_strict(2)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 371, in recv_strict
    bytes_ = self.recv(min(16384, shortage))
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 427, in _recv
    return recv(self.sock, bufsize)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_socket.py", line 83, in recv
    raise WebSocketTimeoutException(message)
TimeoutException: 

Interesting. A similar telemetry socket timeout is what I was also trying to reproduce today to track down crbug.com/706939. The test it fails in is different though.

I'm not 100% sure how to read the results from this bot.. I cant find telemetry logs. The failures I see are 

======================================================================
ERROR: SimpleTestVerify (crostestutils.au_test_harness.au_test.AUTest)
Test that only verifies the target image.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/b/cbuild/shared_external/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_test.py", line 232, in SimpleTestVerify
    self.assertTrue(self.worker.VerifyImage(self))
  File "/b/cbuild/shared_external/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/vm_au_worker.py", line 161, in VerifyImage
    return self._VerifyImage(test)
  File "/b/cbuild/shared_external/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/vm_au_worker.py", line 197, in _VerifyImage
    capture_output=True)
  File "/b/cbuild/shared_external/chromite/lib/cros_build_lib.py", line 593, in RunCommand
    (cmd_result.output, cmd_result.error) = proc.communicate(input)
  File "/usr/lib/python2.7/subprocess.py", line 751, in communicate
    self.wait()
  File "/usr/lib/python2.7/subprocess.py", line 1291, in wait
    pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
  File "/usr/lib/python2.7/subprocess.py", line 478, in _eintr_retry_call
    return func(*args)
  File "/b/cbuild/shared_external/chromite/lib/timeout_util.py", line 62, in kill_us
    raise TimeoutError(error_message % {'time': max_run_time})
TimeoutError: Timeout occurred- waited 4800 seconds.

I"m guessing they are somewhere in "vm_test_results_1" but I cant find them in that.

I'm looking at failures from 
https://uberchromegw.corp.google.com/i/chromiumos.chromium/builders/x86-generic-telemetry/builds/12094

35 failures in telemetry_UnitTests:
telemetry.core.memory_cache_http_server_unittest.MemoryCacheHTTPServerTest.testAbsoluteAndRelativePathsYieldSameURL
telemetry.internal.actions.action_runner_unittest.ActionRunnerMeasureMemoryTest.testDeterministicMode
telemetry.internal.actions.action_runner_unittest.ActionRunnerMeasureMemoryTest.testRealisticMode
telemetry.internal.actions.action_runner_unittest.ActionRunnerMeasureMemoryTest.testWithoutTracing
telemetry.internal.actions.action_runner_unittest.ActionRunnerTest.testWaitForJavaScriptCondition_returnsValue
telemetry.internal.actions.action_runner_unittest.ActionRunnerTest.testWaitForNetworkQuiescenceSmoke
telemetry.internal.actions.key_event_unittest.KeyPressActionTest.testPressEndAndHome
telemetry.internal.actions.load_media_unittest.LoadMediaActionTest.testLoadWithNoSelector
telemetry.internal.actions.loop_unittest.LoopActionTest.testLoopWaitForLoopTimeout
telemetry.internal.actions.loop_unittest.LoopActionTest.testLoopWithAllSelector
telemetry.internal.actions.page_action_unittest.PageActionTest.testEvaluateCallbackWithElement
telemetry.internal.actions.pinch_unittest.PinchActionTest.testPinchByApiCalledWithCorrectArguments
telemetry.internal.actions.play_unittest.PlayActionTest.testPlayWaitForEndedTimeout
telemetry.internal.actions.play_unittest.PlayActionTest.testPlayWaitForPlayTimeout
telemetry.internal.actions.play_unittest.PlayActionTest.testPlayWithAllSelector
telemetry.internal.actions.play_unittest.PlayActionTest.testPlayWithoutWaitForEnded
telemetry.internal.actions.repeatable_scroll_unittest.RepeatableScrollActionTest.testRepeatableScrollActionTwoRepeats
telemetry.internal.actions.scroll_to_element_unittest.ScrollToElementActionTest.testScrollContainerToElement
telemetry.internal.actions.scroll_to_element_unittest.ScrollToElementActionTest.testScrollToElement
telemetry.internal.actions.scroll_unittest.ScrollActionTest.testDiagonalScrollAction
telemetry.internal.actions.scroll_unittest.ScrollActionTest.testScrollAction
telemetry.internal.actions.seek_unittest.SeekActionTest.testSeekWithNoSelector
telemetry.internal.actions.tap_unittest.TapActionTest.testTapNavigate
telemetry.internal.backends.chrome_inspector.inspector_memory_unittest.InspectorMemoryTest.testGetDOMStats
telemetry.internal.backends.chrome_inspector.inspector_page_unittest.InspectorPageTest.testCustomActionToNavigate
telemetry.internal.backends.chrome_inspector.inspector_runtime_unittest.InspectorRuntimeTest.testIFrame
telemetry.internal.backends.chrome_inspector.inspector_runtime_unittest.InspectorRuntimeTest.testRuntimeEvaluateThatFails
telemetry.internal.browser.tab_unittest.TabTest.testActivateTab
telemetry.internal.browser.tab_unittest.TabTest.testGetRendererThreadFromTabId
telemetry.internal.browser.tab_unittest.TabTest.testNavigateAndWaitForInteractiveState
telemetry.internal.browser.tab_unittest.TabTest.testRendererCrash
telemetry.record_wpr_unittest.RecordWprUnitTests.testRecorderPageTest_BrowserMethods
telemetry.core.platform_unittest.PlatformScreenshotTest.testScreenshot
telemetry.core.tracing_controller_unittest.TracingControllerTest.testGotTrace
telemetry.core.tracing_controller_unittest.TracingControllerTest.testStartupTracingOnCrOS

All of these tests navigates to about:blank.

The perf tests seem to have a problem with accessing google cloud storage.

This is a bug with chrome. 

I have TOT chrome, and chrome from Mar 16, and TOT catapult, and catapult from Mar 16. chrome from Mar 16 passes, and TOT chrome fails, regardless of version of catapult.

Any news? This is still failing. In gardener handoff jennyz@ said you were looking into it.

achuith, can you take a look at this, or find someone who can?  I'm seeing a variety of different failures here:

https://build.chromium.org/p/chromiumos.chromium/builders/amd64-generic-telemetry/builds/11445

R60-9538.0.0-b11445/vm_test_results_2/telemetry_suite/test_harness/all/SimpleTestVerify/1_autotest_tests/results-1-telemetry_UnitTests/telemetry_UnitTests.user/debug

Several like this:

05/10 08:11:13.104 INFO |run_chromeos_tests:0052| [186/1075] telemetry.internal.actions.action_runner_unittest.ActionRunnerTest.testScrollToElement failed unexpectedly 13.3953s:
05/10 08:11:13.104 INFO |run_chromeos_tests:0052|   Traceback (most recent call last):
05/10 08:11:13.105 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/testing/browser_test_case.py", line 41, in WrappedMethod
05/10 08:11:13.106 INFO |run_chromeos_tests:0052|       method(self)
05/10 08:11:13.106 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner_unittest.py", line 351, in testScrollToElement
05/10 08:11:13.107 INFO |run_chromeos_tests:0052|       self.assertEqual(viewport_comparator(top_bottom_element), 0)
05/10 08:11:13.108 INFO |run_chromeos_tests:0052|   AssertionError: -1 != 0

and several like this:

05/10 08:40:00.467 INFO |run_chromeos_tests:0052| [795/1075] telemetry.util.color_histogram_unittest.HistogramTest.testHistogram failed unexpectedly 0.0020s:
05/10 08:40:00.467 INFO |run_chromeos_tests:0052|   Traceback (most recent call last):
05/10 08:40:00.468 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/util/color_histogram_unittest.py", line 59, in testHistogram
05/10 08:40:00.468 INFO |run_chromeos_tests:0052|       hist = image_util.GetColorHistogram(bmp)
05/10 08:40:00.468 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/util/image_util.py", line 121, in GetColorHistogram
05/10 08:40:00.469 INFO |run_chromeos_tests:0052|       return impl.GetColorHistogram(image, ignore_color, tolerance)
05/10 08:40:00.469 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/image_processing/image_util_bitmap_impl.py", line 50, in GetColorHistogram
05/10 08:40:00.469 INFO |run_chromeos_tests:0052|       return bitmap.ColorHistogram(ignore_color, tolerance)
05/10 08:40:00.469 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/image_processing/_bitmap.py", line 240, in ColorHistogram
05/10 08:40:00.470 INFO |run_chromeos_tests:0052|       return self._PrepareTools().Histogram(ignore_color, tolerance)
05/10 08:40:00.470 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/image_processing/_bitmap.py", line 120, in _PrepareTools
05/10 08:40:00.470 INFO |run_chromeos_tests:0052|       self._pixels)
05/10 08:40:00.470 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/image_processing/_bitmap.py", line 34, in __init__
05/10 08:40:00.471 INFO |run_chromeos_tests:0052|       platform.GetHostPlatform().GetOSName())
05/10 08:40:00.471 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/util/binary_manager.py", line 63, in FetchPath
05/10 08:40:00.471 INFO |run_chromeos_tests:0052|       return _binary_manager.FetchPath(binary_name, os_name, arch, os_version)
05/10 08:40:00.471 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/third_party/catapult/common/py_utils/py_utils/binary_manager.py", line 38, in FetchPath
05/10 08:40:00.472 INFO |run_chromeos_tests:0052|       os_version)
05/10 08:40:00.472 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/third_party/catapult/common/py_utils/py_utils/binary_manager.py", line 60, in _WrapDependencyManagerFunction
05/10 08:40:00.472 INFO |run_chromeos_tests:0052|       return function(binary_name, platform)
05/10 08:40:00.472 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/third_party/catapult/dependency_manager/dependency_manager/manager.py", line 130, in FetchPath
05/10 08:40:00.473 INFO |run_chromeos_tests:0052|       path, _ = self.FetchPathWithVersion(dependency, platform)
05/10 08:40:00.473 INFO |run_chromeos_tests:0052|     File "/usr/local/telemetry/src/third_party/catapult/dependency_manager/dependency_manager/manager.py", line 89, in FetchPathWithVersion
05/10 08:40:00.473 INFO |run_chromeos_tests:0052|       raise exceptions.NoPathFoundError(dependency, platform)
05/10 08:40:00.473 INFO |run_chromeos_tests:0052|   NoPathFoundError: No file could be found locally, and no file to download from cloud storage for bitmaptools on platform chromeos_x86_64

Maybe related to  issue 669849  "Performance tests cannot be run on architectures which don't have android_tools binaries stored in cloud storage" ?

Ping?

The builder is currently offline. Most recent build was on May 12 which failed with the following:

05/12 12:04:01.805 ERROR|      archive_info:0093| You either aren't authenticated or don't have permission to use the archives for this page set.
You may need to run gsutil config.
You can find instructions for gsutil config at: http://www.chromium.org/developers/telemetry/upload_to_cloud_storage
05/12 12:04:01.827 INFO |run_chromeos_tests:0052| benchmarks.system_health_smoke_test.load_tests() failed: Attempted to access a file from Cloud Storage but you have no configured credentials. To configure your credentials:
05/12 12:04:01.827 INFO |run_chromeos_tests:0052|   1. Run "HOME=/home/chromeos-test/ /usr/local/telemetry/src/third_party/catapult/third_party/gsutil/gsutil config" and follow its instructions.
05/12 12:04:01.827 INFO |run_chromeos_tests:0052|   2. If you have a @google.com account, use that account.
05/12 12:04:01.828 INFO |run_chromeos_tests:0052|   3. For the project-id, just enter 0.
05/12 12:04:01.843 WARNI|              test:0615| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 609, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/common_lib/test.py", line 817, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/common_lib/test.py", line 470, in execute
    dargs)
  File "/usr/local/autotest/common_lib/test.py", line 347, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/common_lib/test.py", line 380, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/tests/telemetry_UnitTests/telemetry_UnitTests.py", line 35, in run_once
    raise error.TestFail(error_str)
TestFail: The unit tests of /usr/local/telemetry/src/tools/perf failed.


Do we need to run the above instructions on that builder?

Yes, I believe this is the bug with the cloud storage issue:
https://bugs.chromium.org/p/chromium/issues/detail?id=712349