Assigning @robertma to be the owner for now

I'm just getting started learning more about LUCI migration, let me know if I can help :-)

Downgrading the LUCI migration issues to P3 until we're prodded to migrate as part of a larger migration.

cc tandrii@

Copying some questions from a CL (https://crrev.com/c/1070489):

qyearsley@:
> When we migrate the importer/exporter to LUCI, we won't use puppet to put credentials on the bots anymore, right? AFAIK, the main thing blocking LUCI migration for importer/exporter is making sure that all of the credentials are available.
>
> (I think that these credentials include: refresh token for git cl try for importer; github username and token, and gerrit token)

robertma@:
> Context: the Gerrit token is the same as the random token used in .gitcookies. We need that because we are using some Gerrit REST APIs that are not provided by depot_tools. For `git cl try`, does it also default to the GCE service account if no additional argument is given? If that's the case, we may simply give the service account try bot access instead of forcing `git cl try` to use the blink-...@chromium.org account.

1. YOu should use a dedicated task service account configured for your builder in cr-buildbucket.cfg (e.g., [1]). Unfortunately, it can't be @chromium.org account.

2. Then, all invocations of git, git cl, gsutil et all will automatically (and transparently for you) authenticate with google services using provided service account.

3. This likely won't suffice to cover github, but you can still use puppet to deploy github credentials to the bot.

+vadimsh@ to correct me.


[1] https://chromium.googlesource.com/chromium/src/+/master/infra/config/global/cr-buildbucket.cfg#159

Actually +vadimsh@ to verify & correct my statement #7 above.

Oh, actually one more thing: we have another service account (on the chromium.org domain) in /creds/service_accounts/... which has access to Monorail APIs.

So IIUC #7, we need
1. Set up a task service account for importer & exporter on LUCI. And give this service account proper access to cr-buildbucket & Gerrit (via git-cl).

2. Continue distributing other credentials needed by HTTP APIs using Puppet (e.g. GitHub token & the other chromium.org service account to access Monorail API).

Re 3: a better approach would be using Cloud KMS, but puppet is OK for now, assuming this builder runs on a predefined set of machines (so we can put their hostnames in puppet config).

I think monorail API can be used with LUCI task service account. You can request a token:
  in recipes, like this https://cs.chromium.org/chromium/infra/recipes-py/recipe_modules/service_account/examples/full.py?l=21
  in scripts, $ luci-auth token (that's what recipe above does https://cs.chromium.org/chromium/infra/recipes-py/recipe_modules/service_account/api.py?l=70)

robertma@ we've made basic setup for two builders in new luci.infra.cron bucket. Please own these two bugs or find another relevant owner. 

+efoo@

Thank you, Andrii! Yes migrating these two bots to LUCI is a P1 for me in Q3.

I will first go over go/migrate2luci and familiarize myself with various concepts. Meanwhile, is there any builder in chromium.infra.cron that has been migrated LUCI? I found  issue 790404 . Would it be a good example?

Some things in that example will be similar, but some might not be. Another possible builder to look at might be "chromium-lkgr-finder", since I think the configs might look similar (triggered based on time, not considered CI for build or infra repos).

When looking at go/migrate2luci, of course the stuff related to trybots can be ignored, so that simplifies things.

I think Andrii's going to be away in the near future, I'll also be available to review CLs :-)

Monorail side has been sorted out, too ( https://crbug.com/monorail/4028 ).

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/6dbd02f564b2293d8958d159dfe148992d5aa70b

commit 6dbd02f564b2293d8958d159dfe148992d5aa70b
Author: Robert Ma <robertma@google.com>
Date: Wed Sep 19 14:51:38 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c1cf0c3de41b056131ee7af13061ff0ab7f38101

commit c1cf0c3de41b056131ee7af13061ff0ab7f38101
Author: Robert Ma <robertma@chromium.org>
Date: Mon Sep 24 19:09:43 2018

Dry run wpt-{importer,exporter} on LUCI

The two recipes now check for `api.runtime.is_experimental` to switch to
dry run mode (no import CL/export PR will be created) and turn on
verbose logging on LUCI, in preparation for migration.

New tests are added to ensure 100% code coverage.

Bug:  803111 
Change-Id: I1a98a83af701028a42ff6fb124ed8424cf3a2aea
Reviewed-on: https://chromium-review.googlesource.com/1231777
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#17831}
[copy] https://crrev.com/c1cf0c3de41b056131ee7af13061ff0ab7f38101/recipes/recipes/wpt_export.expected/wpt-export_experimental.json
[modify] https://crrev.com/c1cf0c3de41b056131ee7af13061ff0ab7f38101/recipes/recipes/wpt_import.expected/wpt-import-with-issue.json
[copy] https://crrev.com/c1cf0c3de41b056131ee7af13061ff0ab7f38101/recipes/recipes/wpt_import.expected/wpt-import-without-issue_luci.json
[modify] https://crrev.com/c1cf0c3de41b056131ee7af13061ff0ab7f38101/recipes/recipes/wpt_export.py
[modify] https://crrev.com/c1cf0c3de41b056131ee7af13061ff0ab7f38101/recipes/README.recipes.md
[modify] https://crrev.com/c1cf0c3de41b056131ee7af13061ff0ab7f38101/recipes/recipes/wpt_import.py
[rename] https://crrev.com/c1cf0c3de41b056131ee7af13061ff0ab7f38101/recipes/recipes/wpt_import.expected/wpt-import-without-issue_buildbot_experimental.json

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/66ff6d94070e59863d29386f07aecf3672299ee8

commit 66ff6d94070e59863d29386f07aecf3672299ee8
Author: Robert Ma <robertma@chromium.org>
Date: Tue Sep 25 15:05:27 2018

Set recipes and the service account for wpt-{importer,exporter} on LUCI

The dedicated service account was requested at https://crbug.com/863229

Bug:  803111 
Change-Id: I5acedd0af52b85c482470cceaea158a36463f570
Reviewed-on: https://chromium-review.googlesource.com/1234837
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>

[modify] https://crrev.com/66ff6d94070e59863d29386f07aecf3672299ee8/cr-buildbucket.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/b66d1393d49c97c0e064f8ff9b36036b96c97f77

commit b66d1393d49c97c0e064f8ff9b36036b96c97f77
Author: Robert Ma <robertma@chromium.org>
Date: Wed Sep 26 14:38:24 2018

Schedule wpt-{importer,exporter} on luci.infra.cron

I forgot to modify the scheduler configuration in the previous CL
https://crrev.com/c/1234837 . This CL schedules wpt-{importer,exporter}
to run continuously (with 1min interval) on LUCI, and gives wpt-importer
a long timeout as it needs to try and land patches (the no-op runs
currently take up to 4 hours according to
https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-importer/?limit=200).

The configuration is roughly based on their counterparts in
chromium.infra.cron:
/build/masters/master.chromium.infra.cron/master.cfg

Bug:  803111 
Change-Id: I422c08b843b913266680bf1a59aa21797cfec20a
Reviewed-on: https://chromium-review.googlesource.com/1244216
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>

[modify] https://crrev.com/b66d1393d49c97c0e064f8ff9b36036b96c97f77/cr-buildbucket.cfg
[modify] https://crrev.com/b66d1393d49c97c0e064f8ff9b36036b96c97f77/luci-scheduler.cfg

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/2eb452b3554b8c2d256014cbe0db92576333b0c0

commit 2eb452b3554b8c2d256014cbe0db92576333b0c0
Author: Robert Ma <robertma@google.com>
Date: Thu Sep 27 19:02:35 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/659eb2cb1014f076323f81125a968d635fbce486

commit 659eb2cb1014f076323f81125a968d635fbce486
Author: Robert Ma <robertma@chromium.org>
Date: Mon Oct 01 15:36:42 2018

[WPT sync] Add pinned depot_tools in checkout to PATH

This change appends the pinned depot_tools in Chromium checkout (at
//src/third_party/depot_tools) to PATH so that the wpt_{import,export}
scripts always work as long as we have a `gclient sync`ed checkout.

The main reason for the change is to fix the scripts on LUCI, where
depot_tools is no longer in PATH by default.

Bug:  803111 
Change-Id: I9242befa3dbc9e735ae3a618dcd5b8f5eff5579d
Reviewed-on: https://chromium-review.googlesource.com/1252532
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#595457}
[modify] https://crrev.com/659eb2cb1014f076323f81125a968d635fbce486/third_party/blink/tools/blinkpy/common/path_finder.py
[modify] https://crrev.com/659eb2cb1014f076323f81125a968d635fbce486/third_party/blink/tools/wpt_export.py
[modify] https://crrev.com/659eb2cb1014f076323f81125a968d635fbce486/third_party/blink/tools/wpt_import.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/f95affaec68632840871056d7e9d674554fe8403

commit f95affaec68632840871056d7e9d674554fe8403
Author: Robert Ma <robertma@chromium.org>
Date: Mon Oct 01 19:05:44 2018

Add mastername to wpt-{importer,exporter} on LUCI

This is needed by the migration app to set the is_experimental flag and
will be removed when LUCI is prod.

Bug:  803111 
Change-Id: I37a80d310ce6c2e05ee4e0e3724218e67f5b1677
Reviewed-on: https://chromium-review.googlesource.com/c/1254846
Auto-Submit: Robert Ma <robertma@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/f95affaec68632840871056d7e9d674554fe8403/cr-buildbucket.cfg

https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-exporter/ seems to have started passing --verbose to wpt_export.py a while ago, so we haven't been exporting CLs for a few days now. Is this part of the migration plan?

Er, s/--verbose/--dry-run/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/34af3640d1d6aaf1030fbc972848628e2a3e6f24

commit 34af3640d1d6aaf1030fbc972848628e2a3e6f24
Author: Robert Ma <robertma@chromium.org>
Date: Wed Oct 03 18:19:41 2018

[WPT export] Set git user in tmp WPT checkout

Builders don't (and shouldn't) have global git user name/email
configured, and the recipe only configures git user in the Chromium
checkout (the recipe doesn't know about the tmp WPT checkout). When git
user cannot be found (neither in the repo or globally), `git commit`
will fail. (Previously, we didn't notice the issue on BuildBot because
the bot had some leftover global configurations from an earlier version
of the recipe.)

To make sure we always have a git user configured, this change
explicitly sets git user name & email in the tmp WPT checkout to the bot
account. It is still safe to run the script manually because GitHub
allows committers to be different from the uploader.

Besides, remove an unnecessary pair of quotes in a format string for
`git show`; we don't use shell to start subprocesses so spaces don't
need to be quoted.

Bug:  803111 
Change-Id: Ife020c8a18120ff607b8802c32cf9539bf6a36c9
Reviewed-on: https://chromium-review.googlesource.com/1258977
Commit-Queue: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#596290}
[modify] https://crrev.com/34af3640d1d6aaf1030fbc972848628e2a3e6f24/third_party/blink/tools/blinkpy/w3c/chromium_commit.py
[modify] https://crrev.com/34af3640d1d6aaf1030fbc972848628e2a3e6f24/third_party/blink/tools/blinkpy/w3c/common.py
[modify] https://crrev.com/34af3640d1d6aaf1030fbc972848628e2a3e6f24/third_party/blink/tools/blinkpy/w3c/local_wpt.py
[modify] https://crrev.com/34af3640d1d6aaf1030fbc972848628e2a3e6f24/third_party/blink/tools/blinkpy/w3c/local_wpt_unittest.py
[modify] https://crrev.com/34af3640d1d6aaf1030fbc972848628e2a3e6f24/third_party/blink/tools/blinkpy/w3c/test_exporter_unittest.py

Re #27:

Yes, the wpt-exporter has been migrated to LUCI (https://ci.chromium.org/p/infra/builders/luci.infra.cron/wpt-exporter), and the legacy BuildBot version is kept in experimental mode now, which means --dry-run and --verbose, to serve as a baseline.

The LUCI wpt-exporter was running smoothly a few days ago, but it's now having a weird problem ( issue 891831 ) and I'm debugging. Meanwhile, I just ran the exporter script manually.

Oops, sorry for the confusion, I thought the Buildbot URL was still the production one. Thanks for the manual intervention in the meantime!

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/24575b1477737d5038d00a9933a7c75bd01ba860

commit 24575b1477737d5038d00a9933a7c75bd01ba860
Author: Robert Ma <robertma@chromium.org>
Date: Wed Oct 10 18:13:58 2018

Fix wpt-importer on LUCI and tweak git user setup

* depot_tools isn't in PATH on LUCI, so we use the depot_tools.git_cl
  recipe module instead to get access to `git cl`.
* wpt-importer uses another service account on LUCI, and LUCI already
  sets user.email in the checkout, so stop setting user.email on LUCI.
* wpt-exporter does not commit in the Chromium checkout, so we don't
  need to configure git user at all.

Bug:  803111 
Change-Id: If6a2231d8daacd9bd3589a1aaea50035a5a869fb
Reviewed-on: https://chromium-review.googlesource.com/c/1255945
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18218}
[modify] https://crrev.com/24575b1477737d5038d00a9933a7c75bd01ba860/recipes/recipes/wpt_export.expected/wpt-export_experimental.json
[modify] https://crrev.com/24575b1477737d5038d00a9933a7c75bd01ba860/recipes/recipes/wpt_import.expected/wpt-import-with-issue.json
[modify] https://crrev.com/24575b1477737d5038d00a9933a7c75bd01ba860/recipes/recipes/wpt_import.expected/wpt-import-without-issue_luci.json
[modify] https://crrev.com/24575b1477737d5038d00a9933a7c75bd01ba860/recipes/recipes/wpt_export.py
[modify] https://crrev.com/24575b1477737d5038d00a9933a7c75bd01ba860/recipes/README.recipes.md
[modify] https://crrev.com/24575b1477737d5038d00a9933a7c75bd01ba860/recipes/recipes/wpt_import.py
[modify] https://crrev.com/24575b1477737d5038d00a9933a7c75bd01ba860/recipes/recipes/wpt_import.expected/wpt-import-without-issue_buildbot_experimental.json
[modify] https://crrev.com/24575b1477737d5038d00a9933a7c75bd01ba860/recipes/recipes/wpt_export.expected/wpt-export.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/e8fb42c104e4315d6da9b9ea05c084ef13d411c8

commit e8fb42c104e4315d6da9b9ea05c084ef13d411c8
Author: Robert Ma <robertma@chromium.org>
Date: Thu Nov 08 20:19:15 2018

Tear down wpt-{exporter,importer} on BuildBot

Bug:  803111 
Change-Id: I90832ecbdedc0e089c7e611b9133d6282e566e42
Reviewed-on: https://chromium-review.googlesource.com/c/1327322
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>

[modify] https://crrev.com/e8fb42c104e4315d6da9b9ea05c084ef13d411c8/masters/master.chromium.infra.cron/slaves.cfg
[modify] https://crrev.com/e8fb42c104e4315d6da9b9ea05c084ef13d411c8/masters/master.chromium.infra.cron/master.cfg

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/4c34cab79cf513d92ced393e24a8f400ec38cd6d

commit 4c34cab79cf513d92ced393e24a8f400ec38cd6d
Author: Robert Ma <robertma@google.com>
Date: Thu Nov 08 22:36:09 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/64a216f2e8b6599f427f7eb1c7404520595ca235

commit 64a216f2e8b6599f427f7eb1c7404520595ca235
Author: Robert Ma <robertma@chromium.org>
Date: Thu Nov 08 22:45:21 2018

Remove special code paths for BuildBot in wpt recipes

Migration has been completed and we are tearing down
wpt-{importer,exporter} on BuildBot, so we no longer need these special
code paths (check for experimental builds & hacks for BuildBot).

Bug:  803111 
Change-Id: Id6805e0ad889b99f5bc7e2fa591e06c71f94564b
Reviewed-on: https://chromium-review.googlesource.com/c/1327283
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18879}
[delete] https://crrev.com/55984032195ca1d4e9a5320438260f3b9e68bf26/recipes/recipes/wpt_export.expected/wpt-export_experimental.json
[modify] https://crrev.com/64a216f2e8b6599f427f7eb1c7404520595ca235/recipes/recipes/wpt_import.expected/wpt-import-with-issue.json
[modify] https://crrev.com/64a216f2e8b6599f427f7eb1c7404520595ca235/recipes/recipes/wpt_export.py
[modify] https://crrev.com/64a216f2e8b6599f427f7eb1c7404520595ca235/recipes/README.recipes.md
[modify] https://crrev.com/64a216f2e8b6599f427f7eb1c7404520595ca235/recipes/recipes/wpt_import.py
[delete] https://crrev.com/55984032195ca1d4e9a5320438260f3b9e68bf26/recipes/recipes/wpt_import.expected/wpt-import-without-issue_buildbot_experimental.json
[rename] https://crrev.com/64a216f2e8b6599f427f7eb1c7404520595ca235/recipes/recipes/wpt_import.expected/wpt-import-without-issue.json