Nodir, Robbie: any thoughts about why luci-auth fails with "failed to refresh the token" flakily?

You are using too large timeout:
https://cs.chromium.org/chromium/src/tools/perf/core/results_dashboard.py?q=LuciAuthTokenGeneratorCallback&sq=package:chromium&g=0&l=37

see comment in https://cs.chromium.org/chromium/infra/recipes-py/recipe_modules/service_account/api.py?l=37

rational: our goal is to have tokens short lived, so that after task completes, bad task after that doesn't have lots of time to abuse the system.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/73e83f95cd896a7e4f75b4eb4b76802a69349131

commit 73e83f95cd896a7e4f75b4eb4b76802a69349131
Author: Ned Nguyen <nednguyen@google.com>
Date: Wed Aug 29 23:02:13 2018

[perf] Do not specify timeout limit for luci-auth's token generation

We misunderstood how authentication token works in the past. For one time request,
the token expiration time don't matter much.

Setting a high timeout expiration is making luci-auth fail flakily. Since we
generate a new token for every perf dashboard upload attempt, using the default
value of luci-auth should be fine

Bug:878916
Cq-Include-Trybots: master.tryserver.chromium.perf:obbs_fyi
Change-Id: Ie9025fb0fc2467dda02cd8aa0aa3e2136ca190a8

NOTRY=true # all tests passed, speed up landing to green up perf waterfall uploading

Change-Id: Ie9025fb0fc2467dda02cd8aa0aa3e2136ca190a8
Reviewed-on: https://chromium-review.googlesource.com/1195930
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Simon Hatch <simonhatch@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#587352}
[modify] https://crrev.com/73e83f95cd896a7e4f75b4eb4b76802a69349131/tools/perf/core/results_dashboard.py

This is fixed: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-go_webview-perf