The LUCI perf builders:
https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-builder-perf/175
https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win64-builder-perf/772
https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win32-builder-perf/722

this looks like an ugly bug in how the test specs are handled for cross-master triggers.

fix on review at https://chromium-review.googlesource.com/c/chromium/tools/build/+/1182675

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/78015f7ebcba26dd97fef1f1279a9dad979d4c10

commit 78015f7ebcba26dd97fef1f1279a9dad979d4c10
Author: John Budorick <jbudorick@chromium.org>
Date: Wed Aug 22 00:02:29 2018

chromium_tests: support src-side test configs for cross-master triggers.

Bug:  875581 
Change-Id: Ic1cd04c1c02be97c1d622d20de2552cc5e7c2794
Reviewed-on: https://chromium-review.googlesource.com/1182675
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/78015f7ebcba26dd97fef1f1279a9dad979d4c10/scripts/slave/README.recipes.md
[modify] https://crrev.com/78015f7ebcba26dd97fef1f1279a9dad979d4c10/scripts/slave/recipe_modules/chromium_tests/bot_config_and_test_db.py
[modify] https://crrev.com/78015f7ebcba26dd97fef1f1279a9dad979d4c10/scripts/slave/recipe_modules/chromium_tests/tests/api/prepare_checkout.py

Good news: android-builder-perf now does attempt to isolate tests. Bad news: it fails in doing so. https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-builder-perf/457

I'm about to head offline but can look at this more tomorrow.

I think I found out where is the problem in #10:

We are passing  '--service-account-json', '/creds/service_accounts/service-account-chrome-perf-buildbot.json' (https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/chromium_perf.py?rcl=d5ce8d85317b8f31fd6ba94974e7efb3be96ac91&l=72) but we should pass chrome-ci-builder instead?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/4af1ac7ce6993ad69f0f92dfc9c08f87a6e6c85f

commit 4af1ac7ce6993ad69f0f92dfc9c08f87a6e6c85f
Author: Nghia Nguyen <nednguyen@google.com>
Date: Thu Aug 23 19:55:05 2018

Add is_luci_builder param to chromium_perf recipe and use it to branch service account

Bug:875581
Change-Id: Iae7c1d03344a9a3f087ee3b9f9dcabcb1610e63d
Recipe-Manual-Change: release_scripts
Recipe-Manual-Change: build_limited_scripts_slave
Reviewed-on: https://chromium-review.googlesource.com/1187073
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>

[modify] https://crrev.com/4af1ac7ce6993ad69f0f92dfc9c08f87a6e6c85f/scripts/slave/recipe_modules/chromium_tests/chromium_perf_fyi.py
[modify] https://crrev.com/4af1ac7ce6993ad69f0f92dfc9c08f87a6e6c85f/scripts/slave/recipe_modules/chromium_tests/chromium_perf.py
[modify] https://crrev.com/4af1ac7ce6993ad69f0f92dfc9c08f87a6e6c85f/scripts/slave/recipe_modules/chromium_tests/tryserver_chromium_perf.py

hmhh, with #12, we still have problem with the service account file:

[E2018-08-23T14:03:31.716227-07:00 12504 0 service.go:91] Failed to load private key JSON - open /creds/service_accounts/service-account-chrome-ci-builder.json: no such file or directory

tandrii@, vadimsh@: any idea about what service account we should use for LUCI perf builder in https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-builder-perf/597? 

(also see #12 & #13)

You should use the same service account configured for this builder in cr-buildbucket.cfg. You can mint tokens for it automatically in recipe by using 

https://cs.chromium.org/chromium/infra/recipes-py/recipe_modules/service_account/api.py?g=0&l=37

if you need to generate this token outside of recipe, run the same cmd that recipe does yourself.

$15: thanks Tandrii. Do you know if I can just use the token generated there, save it to a file & pass that into swarming service_account_json (the "--auth-service-account-json" flag)?

Or do we use the "--auth-tokens-cache" flag of swarming.py instead?

So what tool is the ultimate consumer of the service account?

Whatever this tool it is, it should be able to use luci-auth.
If the tool is swarming.py, it'd have expected it to be the case already.

Re #16: no, saving auth token as a private key won't work. I dunno how that cache works, but I don't think it's a good solution anyhow. The tool may need to retry a request and may need to refresh a token.

#17: swarming.py & isolate.py

+Ben/Marc in case if they know both tools support consuming the authorization token

I think the solution here is that we shouldn't pass a service account to swarming or isolate at all when running under LUCI. I think doing so will let them both use the local auth server.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/ca47f586feda1ab2b01879f54e40e52e3b329e23

commit ca47f586feda1ab2b01879f54e40e52e3b329e23
Author: Nghia Nguyen <nednguyen@google.com>
Date: Fri Aug 24 23:57:07 2018

[perf test recipe] Only set swarming and isolate service account for non luci build

See https://bugs.chromium.org/p/chromium/issues/detail?id=875581#c19

Bug:875581
Change-Id: Ib1873bf9af09885a554b3279b72708b1b1d12a1a
Recipe-Manual-Change: build_limited_scripts_slave
Recipe-Manual-Change: release_scripts
Reviewed-on: https://chromium-review.googlesource.com/1188632
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/ca47f586feda1ab2b01879f54e40e52e3b329e23/scripts/slave/recipe_modules/chromium_tests/chromium_perf.py

Not attempting to pass the account json appears to have worked.

https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-builder-perf/722 failed to isolate tests again, but differently: chrome-isolated logs show chrome-ci-builder@chops-service-accounts being denied access.

I added project-chrome-ci-task-accounts to chrome-swarming-users on chrome-infra-auth prior to https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/android-builder-perf/723.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/35ff77e8c01663c24c9199842ac0498e7e957789

commit 35ff77e8c01663c24c9199842ac0498e7e957789
Author: John Budorick <jbudorick@chromium.org>
Date: Mon Aug 27 20:54:54 2018

chromium_tests: don't use service accounts for swarming or isolate on LUCI.

Bug:  875581 
Change-Id: Ic84b6b9fdd5584c28fbd80bb6dfc6ff62f764fe7
Reviewed-on: https://chromium-review.googlesource.com/1188633
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/35ff77e8c01663c24c9199842ac0498e7e957789/scripts/slave/README.recipes.md
[modify] https://crrev.com/35ff77e8c01663c24c9199842ac0498e7e957789/scripts/slave/recipe_modules/chromium_tests/api.py
[modify] https://crrev.com/35ff77e8c01663c24c9199842ac0498e7e957789/scripts/slave/recipe_modules/chromium_tests/tests/api/set_up_swarming.py
[modify] https://crrev.com/35ff77e8c01663c24c9199842ac0498e7e957789/scripts/slave/recipe_modules/chromium_tests/tests/api/set_up_swarming.expected/basic.json