Looking at the logs for the most recent failing build
https://ci.chromium.org/buildbot/chromium.perf/mac-10_13_laptop_high_end-perf/138


 it says we are unauthorized: Error uploading histogram data: HTTP Response 401: Unauthorized
Duration of blink_perf.css upload time: 3 seconds
Duration of system_health.common_desktop.reference results merging: 0 seconds
Uploading perf results from system_health.common_desktop.reference benchmark
Sending histogram result 2 of 2 to dashboard.
Error uploading histogram data: HTTP Response 401: Unauthorized
Duration of system_health.common_desktop.reference upload time: 38 seconds
Uploading perf results from smoothness.tough_path_rendering_cases benchmark
Sending histogram result 2 of 2 to dashboard.
Error uploading histogram data: HTTP Response 401: Unauthorized
Duration of smoothness.tough_path_rendering_cases upload time: 8 seconds
Duration of loading.desktop results merging: 2 seconds
Uploading perf results from loading.desktop benchmark
Sending histogram result 2 of 2 to dashboard.
Error uploading histogram data: HTTP Response 401: Unauthorized
Duration of loading.desktop upload time: 74 seconds
Uploading perf results from dromaeo benchmark
Sending histogram result 2 of 2 to dashboard.
Error uploading histogram data: HTTP Response 401: Unauthorized
Duration of dromaeo upload time: 2 seconds
Uploading perf results from kraken benchmark
Sending histogram result 2 of 2 to dashboard.
Error uploading histogram data: HTTP Response 401: Unauthorized
Duration of kraken upload time: 2 seconds
Uploading perf results from blink_perf.owp_storage.reference benchmark
Sending histogram result 2 of 2 to dashboard.
Error uploading histogram data: HTTP Response 401: Unauthorized
Duration of blink_perf.owp_storage.reference upload time: 2 seconds
Uploading perf results from smoothness.gpu_rasterization_and_decoding.image_decoding_cases benchmark
Sending histogram result 2 of 2 to dashboard.
Error uploading histogram data: HTTP Response 401: Unauthorized
Duration of smoothness.gpu_rasterization_and_decoding.image_decoding_cases upload time: 2 seconds
Uploading perf results from smoothness.key_desktop_move_cases benchmark
Sending histogram result 2 of 2 to dashboard.
Error uploading histogram data: HTTP Response 401: Unauthorized
Duration of smoothness.key_desktop_move_cases upload time: 2 seconds
Uploading perf results from scheduler.tough_scheduling_cases benchmark
Sending histogram result 2 of 2 to dashboard.
Error uploading histogram data: HTTP Response 401: Unauthorized
Duration of scheduler.tough_scheduling_cases upload time: 3 seconds
Duration of Uploading results to perf dashboard: 2647 seconds
Duration of Merging json test results: 0 seconds
Duration of Total process_perf_results: 2649 seconds
WARNING:root:merge_cmd had non-zero return code: 1
step returned non-zero exit code: 1

So oddly enough this seems to have corrected itself: 
https://uberchromegw.corp.google.com/i/chromium.perf/builders/mac-10_13_laptop_high_end-perf

After three failed builds thi sis showing green.  And I couldn't find any 401 errors in the dashboard logs.  

Simon do you think you could investigate what happened here?  Maybe I am just missing something in the logs.  WE want to make sure it doesnt happen again,

I seem to find these in the dashboard's logs just fine, how are you searching?

From poking through the waterfall stdio links you provided, it looks like some upload properly while others get hit with an unauthorized error. Unfortunately the dashboard isn't very specific about the error, I think we could split this a bit better to try to get a more specific oauth error, I'll put up a CL to do that.

Is there a chance the oauth token is expiring before you use it or something like that? How/when is that generated.

This seems similar to https://bugs.chromium.org/p/chromium/issues/detail?id=854162

Yeah let's dupe these together.