Can you describe in detail what builders would be uploading files and what tasks will be fetching them? 

IIUC, the existing Buildbot builders will be keep uploading to isolateserver.appspot.com, but start triggering isolated tasks on chrome-swarming in pool:chrome.tests?

Chatted with Vadim offline. 

The perf buildbot builders (all public) will be keep uploading to isolateserver.appspot.com. The isolated tasks will only be triggered from either existing public perf buildbot builders or new internal LUCI-based builders

Chatted with Vadim offline, here are the concrete steps:
1. Pick a name for the new internal perf pool.
2. Find few initial bots to add there.
3. Setup bots.cfg and pools.cfg configs for it.
4. Add 'system_service_account' used by these new bots to isolateserver ACL.

We discussed offline, the 3 new names we will use for internal perf pool are:

Chrome-perf --> chrome.tests.perf 
Chrome-perf-fyi --> chrome.tests.perf-fyi
Chrome-perf-pinpoint --> chrome.tests.pinpoint

Vadim, can you create those 3 new pools in chrome-swarming.appspot.com? 

Once that is done, I can file bug with lab team to add few sample machines into each of those.

Ping Vadim, can you create those 3 pools in chrome-swarming.appspot.com? 

I'm unavailable until 5PM today. Andrii can probably help too if it is needed earlier.

Thanks Vadim: we can certainly wait until 5 pm today :-)

Any updates on this? 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/e52da5efff3a6dd466c48c9d64700dbd58fd1b97

commit e52da5efff3a6dd466c48c9d64700dbd58fd1b97
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Mar 20 21:05:41 2018

Forgot to commit the CL that sets up pools.

So now Ned can move machines to pools chrome.tests.perf, chrome.tests.perf-fyi, chrome.tests.pinpoint.

Added bots-chrome@chrome-swarming.iam.gserviceaccount.com to isolateserver-full-access group, this should allow bots in chrome.tests.* pools to access isolateserver.appspot.com.

Thanks Vadim! I will try migrating a few bots & validate they have no credential issues before marking this bug as fixed

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/c08eb07ab817939ed3c0a434c40b8290f4ffbb74

commit c08eb07ab817939ed3c0a434c40b8290f4ffbb74
Author: David Tu <dtu@google.com>
Date: Wed Mar 21 00:13:01 2018

Hi Vadim, 

We have some credential problem with our first builder converted to use private swarming server. Can you take a look?

Build:
https://ci.chromium.org/buildbot/chromium.perf.fyi/Android%20Go/452

Log:
Running Swarming with args:
['trigger', '--swarming', 'https://chrome-swarming.appspot.com', '--isolate-server', 'https://isolateserver.appspot.com', '--priority', '25', '--task-name', 'performance_test_suite/Android/3a28b0f9dd/Android Go/452', '--expiration', '36000', '--io-timeout', '1800', '--hard-timeout', '36000', '--dimension', 'os', 'Android', '--dimension', 'pool', 'chrome.tests.perf-fyi', '--named-cache', 'swarming_module_cache_vpython', '.swarming_module_cache/vpython', '--tag', 'buildername:Android Go', '--tag', 'buildnumber:452', '--tag', 'data:3a28b0f9ddc795e7e23b75968204aaf73e1323e4', '--tag', 'master:chromium.perf.fyi', '--tag', 'name:performance_test_suite', '--tag', 'project:chromium', '--tag', 'purpose:CI', '--tag', 'purpose:post-commit', '--tag', 'slavename:slave340-c1', '--tag', 'stepname:performance_test_suite on Android', '--cipd-package', '.swarming_module:infra/python/cpython/${platform}:version:2.7.14.chromium14', '--cipd-package', '.swarming_module:infra/tools/luci/logdog/butler/${platform}:git_revision:e1abc57be62d198b5c2f487bfb2fa2d2eb0e867c', '--cipd-package', '.swarming_module:infra/tools/luci/vpython-native/${platform}:git_revision:e1abc57be62d198b5c2f487bfb2fa2d2eb0e867c', '--cipd-package', '.swarming_module:infra/tools/luci/vpython/${platform}:git_revision:e1abc57be62d198b5c2f487bfb2fa2d2eb0e867c', '--env-prefix', 'PATH', '.swarming_module', '--env-prefix', 'PATH', '.swarming_module/bin', '--env-prefix', 'VPYTHON_VIRTUALENV_ROOT', '.swarming_module_cache/vpython', '--isolated', '3a28b0f9ddc795e7e23b75968204aaf73e1323e4', '--dump-json', '/tmp/base_trigger_dimensions_cZgOE.json', '--env', 'GTEST_SHARD_INDEX', '0', '--env', 'GTEST_TOTAL_SHARDS', '14', '--dimension', 'id', 'build30-a7--device1', '--', '-v', '--browser=android-chromium', '--isolated-script-test-output=${ISOLATED_OUTDIR}/output.json', '--isolated-script-test-perf-output=${ISOLATED_OUTDIR}/perftest-output.json']
2953 2018-03-27 17:19:28.520 E: Request to https://chrome-swarming.appspot.com/api/swarming/v1/tasks/new failed with HTTP status code 403: Access is denied.
Sending the report ... done.
Report URL: https://chrome-swarming.appspot.com/restricted/ereporter2/errors/5653263235940352
Failed to trigger task performance_test_suite/Android/3a28b0f9dd/Android Go/452
Failed to trigger a task, aborting
step returned non-zero exit code: 1