New issue
Advanced search Search tips
Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Nov 8
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Task

Blocking:
issue 803111



Sign in to add a comment
link

Issue 790503: Migrate "wpt-importer" to LUCI

Reported by luci-mig...@appspot.gserviceaccount.com, Nov 30 2017 Project Member

Issue description

Migrate builder chromium.infra.cron:wpt-importer to LUCI.

Buildbot: https://ci.chromium.org/buildbot/chromium.infra.cron/wpt-importer
LUCI: https://ci.chromium.org/buildbucket/luci.infra.cron/wpt-importer

Migration app will be posting updates on changes of the migration status.
For the latest status, see
https://luci-migration.appspot.com/masters/chromium.infra.cron/builders/wpt-importer

Migration app will close this bug when the builder is entirely migrated from Buildbot to LUCI.
 

Comment 1 by robertma@chromium.org, Jan 17 2018

Blocking: 803111
Cc: robertma@chromium.org foolip@chromium.org
Components: Blink>Infra>Ecosystem

Comment 2 by no...@chromium.org, Jan 18 2018

Labels: -Restrict-View-Google

Comment 3 by efoo@chromium.org, Jan 30 2018

Labels: LUCI-Blocker-P1Builders

Comment 4 by efoo@chromium.org, Feb 13 2018

Labels: -LUCI-Blocker-P1Builders LUCI-Chromium

Comment 5 by efoo@chromium.org, Feb 21 2018

Labels: Pri-2

Comment 6 by ajuma@chromium.org, Apr 23 2018

Comment 7 by qyears...@chromium.org, Apr 30 2018

This would most likely be migrated together with wpt-exporter ( bug 790570 ).

First step for both of these is to modify the recipe to check the api.runtime.is_experimental recipe attribute, and if the builder is experimental, just do a dry run. Then we'll add the LUCI builder, check that it runs as expected with dry runs and then flip "LUCI is prod". This will make it easy to roll back if something unexpected happens.

Comment 8 by foolip@chromium.org, May 19 2018

Labels: -Pri-2 Pri-3

Comment 9 by tandrii@chromium.org, Jul 10 2018

Owner: robertma@chromium.org
Status: Assigned (was: Available)

Comment 10 by tandrii@chromium.org, Jul 10 2018

Description: Show this description

Comment 11 by efoo@chromium.org, Jul 10 2018

Labels: LUCI-Chromium-Mass

Comment 12 by efoo@chromium.org, Jul 10 2018

Labels: Pri-1

Comment 13 by efoo@chromium.org, Sep 8

Labels: LUCI-Mirror-Red

Comment 14 by robertma@chromium.org, Oct 1

Status: Started (was: Assigned)

Comment 15 by bugdroid1@chromium.org, Oct 4

Project Member
Labels: merge-merged-config
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/2f1e7a9d203304f6e14570ff17fc332d62649800

commit 2f1e7a9d203304f6e14570ff17fc332d62649800
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Oct 04 22:26:17 2018

luci.infra.cron: connect to migration app

Connect luci.infra.cron to migration app because not all builders were migrated
to LUCI yet.

Bug:  790503 
Change-Id: I0ae2a67d17f50955ee869d3540f80dabae20454b
Reviewed-on: https://chromium-review.googlesource.com/c/1262919
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/2f1e7a9d203304f6e14570ff17fc332d62649800/cr-buildbucket.cfg

Comment 16 by bugdroid1@chromium.org, Oct 12

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/167121ea09987452572723a78b2d8987b770b751

commit 167121ea09987452572723a78b2d8987b770b751
Author: Robert Ma <robertma@chromium.org>
Date: Fri Oct 12 13:49:45 2018

[WPT import] Add a flag to upload but not submit CL

We used to have only one flag, --auto-update, which controls both
uploading and landing the import CL. This makes testing new setup of the
bot (i.e. LUCI in this case) hard.

This change adds one more flag, --auto-upload, to exercise more steps
(namely uploading and rebaselining), which will be used on the
experimental LUCI builder.

Bug:  790503 
Change-Id: I4ce07fc9ddf15a34c9cb25ba4723d250cad26977
Reviewed-on: https://chromium-review.googlesource.com/c/1277587
Commit-Queue: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#599197}
[modify] https://crrev.com/167121ea09987452572723a78b2d8987b770b751/third_party/blink/tools/blinkpy/w3c/test_importer.py

Comment 17 by bugdroid1@chromium.org, Oct 12

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/ec5a8d649238fad935a09d6ae1f260bde7171b6b

commit ec5a8d649238fad935a09d6ae1f260bde7171b6b
Author: Robert Ma <robertma@chromium.org>
Date: Fri Oct 12 13:51:45 2018

[WPT import] Add --auto-upload to importer on LUCI

To test two more critical steps in the importer: uploading CL and
rebaselining. The flag is being added in https://crrev.com/c/1277587 .

Bug:  790503 
Change-Id: Id3abeafe573e876019da0c5e06236728053a1168
Reviewed-on: https://chromium-review.googlesource.com/c/1277625
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18280}
[modify] https://crrev.com/ec5a8d649238fad935a09d6ae1f260bde7171b6b/recipes/recipes/wpt_import.expected/wpt-import-without-issue_buildbot_experimental.json
[modify] https://crrev.com/ec5a8d649238fad935a09d6ae1f260bde7171b6b/recipes/README.recipes.md
[modify] https://crrev.com/ec5a8d649238fad935a09d6ae1f260bde7171b6b/recipes/recipes/wpt_import.py

Comment 18 by efoo@google.com, Oct 30

Build looks to be working as intended and matches Buildbot results. Can the LUCI builder be flipped to prod?

Comment 19 by robertma@chromium.org, Oct 30

Hi Eric, sorry I was travelling in the last two weeks. Yes, could you please flip the switch to set LUCI as prod? Thanks!

Comment 20 by bugdroid1@chromium.org, Oct 31

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/72b85dbf39aee46060072351c2057bd18913356e

commit 72b85dbf39aee46060072351c2057bd18913356e
Author: Robert Ma <robertma@chromium.org>
Date: Wed Oct 31 19:51:15 2018

[WPT Import] Stop creating CLs on experimental bot

The experimental builds on LUCI have proven to work as intended and we
are flipping prod to LUCI, so we don't need to create CLs from the
experimental bot anymore. This will save some unnecessary try jobs and
reduce the wait time of *-blink-rel bots.

Bug:  790503 
Change-Id: I7e983165a4df3c327278df0ec77618ce96a7ca45
Reviewed-on: https://chromium-review.googlesource.com/c/1310523
Auto-Submit: Robert Ma <robertma@chromium.org>
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Cr-Commit-Position: refs/heads/master@{#18702}
[modify] https://crrev.com/72b85dbf39aee46060072351c2057bd18913356e/recipes/recipes/wpt_import.expected/wpt-import-without-issue_buildbot_experimental.json
[modify] https://crrev.com/72b85dbf39aee46060072351c2057bd18913356e/recipes/README.recipes.md
[modify] https://crrev.com/72b85dbf39aee46060072351c2057bd18913356e/recipes/recipes/wpt_import.py

Comment 21 by raphael....@intel.com, Nov 1

Imports are currently failing on chromium_presubmit due to the account being used. See e.g. https://chromium-review.googlesource.com/c/chromium/src/+/1312618/ and https://ci.chromium.org/p/chromium/builders/luci.chromium.try/chromium_presubmit/269758:

** Presubmit ERRORS **
wpt-autoroller@chops-service-accounts.iam.gserviceaccount.com is not in AUTHORS file. If you are a new contributor, please visit
https://www.chromium.org/developers/contributing-code and read the "Legal" section
If you are a chromite, verify the contributor signed the CLA.

Comment 22 by raphael....@intel.com, Nov 2

The issue above seems to have been fixed, but the new account seems to be having trouble with Monorail. From https://ci.chromium.org/p/infra/builders/luci.infra.cron/wpt-importer/b8931004603591793696:

2018-11-01 22:18:25,607 - URL being requested: GET https://monorail-prod.appspot.com/_ah/api/discovery/v1/apis/monorail/v1/rest
2018-11-01 22:18:25,743 - URL being requested: POST https://monorail-prod.appspot.com/_ah/api/monorail/v1/projects/chromium/issues?alt=json
2018-11-01 22:18:25,743 - Attempting refresh to obtain initial access_token
2018-11-01 22:18:25,799 - Refreshing due to a 401 (attempt 1/2)
2018-11-01 22:18:25,827 - Refreshing due to a 401 (attempt 2/2)
Traceback (most recent call last):
  File "/b/swarming/w/ir/kitchen-workdir/src/third_party/blink/tools/wpt_import.py", line 26, in <module>
    main()
  File "/b/swarming/w/ir/kitchen-workdir/src/third_party/blink/tools/wpt_import.py", line 19, in main
    host.exit(importer.main())
  File "/b/swarming/w/ir/kitchen-workdir/src/third_party/blink/tools/blinkpy/w3c/test_importer.py", line 186, in main
    if not self.send_notifications(local_wpt, options.auto_file_bugs, options.monorail_auth_json):
  File "/b/swarming/w/ir/kitchen-workdir/src/third_party/blink/tools/blinkpy/w3c/test_importer.py", line 667, in send_notifications
    dry_run=not auto_file_bugs, service_account_key_json=monorail_auth_json)
  File "/b/swarming/w/ir/kitchen-workdir/src/third_party/blink/tools/blinkpy/w3c/import_notifier.py", line 73, in main
    self.file_bugs(bugs, dry_run, service_account_key_json)
  File "/b/swarming/w/ir/kitchen-workdir/src/third_party/blink/tools/blinkpy/w3c/import_notifier.py", line 264, in file_bugs
    response = api.insert_issue(bug)
  File "/b/swarming/w/ir/kitchen-workdir/src/third_party/blink/tools/blinkpy/w3c/monorail.py", line 137, in insert_issue
    return self.api.issues().insert(projectId=issue.project_id, body=body).execute()
  File "/b/swarming/w/ir/cache/vpython/ae31aa/lib/python2.7/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/b/swarming/w/ir/cache/vpython/ae31aa/lib/python2.7/site-packages/googleapiclient/http.py", line 840, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 401 when requesting https://monorail-prod.appspot.com/_ah/api/monorail/v1/projects/chromium/issues?alt=json returned "Auth error: Neither client ID '116719108335393041424' nor email 'swarming@chromecompute.google.com.iam.gserviceaccount.com' is whitelisted">
step returned non-zero exit code: 1

Comment 23 by bugdroid1@chromium.org, Nov 6

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d86841e2fe2b3eda485a350a7ef44a489f15009e

commit d86841e2fe2b3eda485a350a7ef44a489f15009e
Author: Robert Ma <robertma@chromium.org>
Date: Tue Nov 06 18:20:38 2018

[Import Notifier] Get Monorail OAuth token from luci-auth

The application default credential belongs to the default Swarming bot
account, not the dedicated service account for wpt-importer. As
suggested by LUCI task force, we need to use `luci-auth token` instead.

Bug:  790503 
Change-Id: I3257db9a49196fd427c6f2371d00dcabab7ee3a6
Reviewed-on: https://chromium-review.googlesource.com/c/1315931
Commit-Queue: Robert Ma <robertma@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#605746}
[add] https://crrev.com/d86841e2fe2b3eda485a350a7ef44a489f15009e/third_party/blink/tools/blinkpy/common/net/luci_auth.py
[modify] https://crrev.com/d86841e2fe2b3eda485a350a7ef44a489f15009e/third_party/blink/tools/blinkpy/w3c/import_notifier.py
[modify] https://crrev.com/d86841e2fe2b3eda485a350a7ef44a489f15009e/third_party/blink/tools/blinkpy/w3c/import_notifier_unittest.py
[modify] https://crrev.com/d86841e2fe2b3eda485a350a7ef44a489f15009e/third_party/blink/tools/blinkpy/w3c/monorail.py

Comment 24 by robertma@chromium.org, Nov 8

Status: Fixed (was: Started)

Comment 25 by luci-mig...@appspot.gserviceaccount.com, Dec 17

Project Member
Description: Show this description

Sign in to add a comment