New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 731091 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Sep 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocked on:
issue 529026

Blocking:
issue 600469



Sign in to add a comment

Migrate catapult from GitHub+Rietveld to Gerrit

Project Member Reported by nedngu...@google.com, Jun 8 2017

Issue description

Not sure if we misconfigure anything but anytime someone submit a catapult CL through rietveld, we can't +1 or trigger tryjob or commit.
 
Example CL would be useful here.
Sorry, an example CL is https://chromium-review.googlesource.com/c/527594/
Summary: gerrit is not usable for https://github.com/catapult-project/catapult project (was: rietveld is not usable for https://github.com/catapult-project/catapult project)
Components: -Infra>Codereview>Rietveld Infra>Codereview>Gerrit

Comment 5 by aga...@chromium.org, Jun 12 2017

Status: WontFix (was: Untriaged)
Yes, this is because catapult is a github repo. The repository on googlesource is just a mirror.

I thought[1] I had previously disallowed uploading changes; turns out I hadn't correctly done so. I have now fixed the glitch[2].

If you want to use gerrit to review catapult changes, talk to your org about moving catapult off of GitHub.

[1] https://chromium.googlesource.com/external/+/36b01fc56e1a0e21404badde89bdc6d746675b55
[2] https://chromium.googlesource.com/external/+/be6d08eda1c452480018d3621e4797ac7ac0b857
When I upload a CL in catapult, I get the message...

=====================================
NOTICE: Rietveld is being deprecated. You can upload changes to Gerrit with
  git cl upload --gerrit
or set Gerrit to be your default code review tool with
  git config gerrit.host true
=====================================

Is there anyway to disable this for catapult?
Cc: dpranke@chromium.org aga...@chromium.org sullivan@chromium.org
Status: Unconfirmed (was: WontFix)
Also if gerrit cannot support github repo, does that mean that the team will still keep rietveld around or we need to find a new code review client?
We're not going to keep rietveld around just for catapult. You'll either need to move to Gerrit's built in codereview tools, or move your source of truth to *.googlesource.com and set up a github mirror, I think.
"You'll either need to move to Gerrit's built in codereview tools" --> I thought Gerrit cannot support github. What do you mean by this?


It means I typed it wrong :). I meant "move to *GitHub's* built in codereview tools".

Sorry for the confusion.
For example, here's the bug tracking the transition of the LUCI repos from using github as the source of truth to using googlesource: https://bugs.chromium.org/p/chromium/issues/detail?id=726507

The work hasn't been started because the person running most of the migrations (i.e. me) is focused on src.git right now; but I'll be tackling all the other repos which continue to use Rietveld shortly after that main switch is complete.
Can we have some rationale behind the "If you want to use gerrit to review catapult changes, talk to your org about moving catapult off of GitHub." ?
That is the way the CQ did work on the past for Rietveld, so this seems a regression to me introduced by the Gerrit.
Why the CQ cannot keep reacting to "ticks" (now +1) and landing stuff to github like it did in the past?

Alternatively, how we'd get a CQ if catapult moved to GitHub? Is there any chrome infra integration with Github webhooks in place?
The rationale is mostly that Chrome Ops is trying hard to reduce the number of different usage patterns we need to support. 

If the Gerrit team has a story for how to integrate w/ GitHub, we can look into that, but we would prefer not to support doing additional work on our side to handle such an integration.

Yes, this is a change from how things worked previously, and one could certainly view such a change as a regression :).

I don't think we have any answer for CQ integration with a GH repo in a non-Rietveld, Gerrit-only world. We'd have to build something new to replace the existing mechanisms.

I suppose it's possible that we could continue to support Rietveld plus the Rietveld/CQ integration for GH uses; we'd have to evaluate the cost of that versus the cost of migrating GH-based projects like Catapult to googlesource.com.
Just replied on the other thread. I mistakenly though that we'd have lost the CQ in any case, which turned out to be just my misunderstanding. 

Labels: Milestone-Afterglow Proj-Gerrit-Migration
Status: Available (was: Unconfirmed)
Summary: Migrate catapult from GitHub+Rietveld to Gerrit (was: gerrit is not usable for https://github.com/catapult-project/catapult project)
Based on the content of the other thread, I'm repurposing this bug to track switching catapult source-of-truth from github to googlesource, and switching code-review systems from Rietveld to Gerrit at the same time.
Labels: -Milestone-Afterglow Milestone-Turndown
The "replication" plugin *may* be a good alternative to copybara for mirroring back to github: https://gerrit.googlesource.com/plugins/replication/+/master/src/main/resources/Documentation
Cc: estaab@chromium.org
Project Member

Comment 19 by sheriffbot@chromium.org, Jul 13 2017

Labels: Hotlist-Google
Blockedon: 529026
Blocking: 600469
Cc: achuith@chromium.org
https://docs.google.com/document/d/1_9SDp_7iIFXF4ASBCdH4JbcftYo6yq1wYS7dnEZce6s/edit provides the list of steps that the LUCI repos used to migrate from github to gerrit last night. Not all of them apply (updating go stuff for example) but basically these are the steps to use:

1) create a new repo (chromium.googlesource.com/catapult-project/catapult?) with the latest content (either via gob-ctl copy configs or via manual force-push)
2) update permissions on the new repo to match chromium/src
3) update infra/config/cq.cfg to add a gerrit cq
3a) update puppet to have the cq start running the new cq
3b) add a buildbucket.cfg listing the trybots syzygy uses
4) update codereview.settings to upload to gerrit by default
5) halt mirroring from github to external/github.com/catapult-project/catapult and mark both the github repo and the googlesource mirror as fully read-only
6) update rietveld to refuse uploads to the catapult project

I can do all these steps (except for making the github repo read-only) in about an hour. I can do them today, if you like, or next week when I return from my next block of vacation.

Note that these steps do *not* include mirroring back to the github repo. We're still working on solutions for that. If you're okay with having the github mirror go stale for a while we can do the migration immediately; if you strictly require that the github mirror stay up-to-date the entire time, we'll have to wait on this until after that solution is in place.
agable@ Thanks for taking this bug! 

However, would the CQ bots still work as-is with the migration?
The catapult CQ bots use bot_update, which knows how to apply gerrit patches. They should continue to work just fine. We would test this (in step 3b) before making the github repo read-only.
Thanks for the clear outline in #23, agable!

We'd strongly prefer to have the mirror stay up to date, so it would be better to wait until the solution for that is in place. When we're ready to migrate, we should set a clear date/time for the migration and announce it on relevant project mailing lists (tracing@chromium.org, telemetry-announce@chromium.org, chrome-speed-operations@google.com)
Cc: benjhayden@chromium.org benhenry@chromium.org
agable@:
1. Any update on mirroring googlesource to github?
2. How should catapult's bugs be moved to monorail?
3. Is there a way to make monorail bug priority more fine-grained than 0-3?

sullivan@: Is an up-to-date mirror still strongly preferred? Why?
If it's just about having a homepage on github, can it just link to googlesource like this? https://github.com/chromium/chromium

An up-to-date mirror is still strongly preferred. We have a lot of documentation on github, and lots of third-party tooling that we're not aware of pulls from the github repo. I'd really like to make as few changes as possible here.
Also re #27: is the bug tracker used by the project in scope for this bug? It'd be better to have separation of concerns if possible.
1) I just got back from vacation, and am focusing on the mirroring now. I expect to have the solution running within a week. The bug blocking this one will have the updates for you to follow along.
2-3) As sullivan@ suggested, bugs are a wholly separate concern.
Mirroring is now in place! I'd like to set a migration date of Tuesday, september 26. Does that work for you?
Lots of people will be OOO for blinkon and promo committee, which should be a good thing because there will be fewer attempts to submit CLs during migration. However I'm usually the point of contact on issues like this and I'll be OOO all day in committee. Maybe the following day?
I *just* got an email saying "Hey we put you on a committee go watch the training session that already happened" so yeah, we're going to move this to Wednesday instead.
Owner: aga...@chromium.org
Status: Started (was: Available)
I need one more thing from you: What would you like catapult's home on chromium.googlesource.com to be? We won't be using "external/github.com/catapult-project/catapult" since that repo is explicitly named to be a mirror (although it will continue to exist for historical purposes).

What repo path do you want me to create for you?
Can we just call it "catapult"?
Can do.
Project Member

Comment 37 by bugdroid1@chromium.org, Sep 27 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/e345b4a43bed5bbbf930b5e3ab0d79b72455c96a

commit e345b4a43bed5bbbf930b5e3ab0d79b72455c96a
Author: Aaron Gable <agable@chromium.org>
Date: Wed Sep 27 19:48:25 2017

Project Member

Comment 38 by bugdroid1@chromium.org, Sep 27 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/c0daba28f4d64e87ae53b7f7f492cb510562327a

commit c0daba28f4d64e87ae53b7f7f492cb510562327a
Author: Aaron Gable <agable@chromium.org>
Date: Wed Sep 27 20:19:07 2017

Fix typo in cq.cfg

TBR=tandrii@chromium.org

Bug:  chromium:731091 
Change-Id: I59106a1bec15818159eb474a9d106c4fb95abcbf
Reviewed-on: https://chromium-review.googlesource.com/687889
Reviewed-by: Aaron Gable <agable@chromium.org>

[modify] https://crrev.com/c0daba28f4d64e87ae53b7f7f492cb510562327a/infra/config/cq.cfg

Project Member

Comment 39 by bugdroid1@chromium.org, Sep 27 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/e5e3843f330ed7b84082c2e44c1b17bc6caa4487

commit e5e3843f330ed7b84082c2e44c1b17bc6caa4487
Author: Aaron Gable <agable@chromium.org>
Date: Wed Sep 27 20:57:06 2017

Make Gerrit the default code review system for Catapult

Bug:  chromium:731091 
Change-Id: Id14bf6b58e54df5ea21e8907083d81cfddee93cf
Reviewed-on: https://chromium-review.googlesource.com/687873
Reviewed-by: Annie Sullivan <sullivan@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>

[modify] https://crrev.com/e5e3843f330ed7b84082c2e44c1b17bc6caa4487/CONTRIBUTING.md
[modify] https://crrev.com/e5e3843f330ed7b84082c2e44c1b17bc6caa4487/codereview.settings

Status: Fixed (was: Started)
Done! Please update this bug or the PSA threads with any problems that are found.
Project Member

Comment 41 by bugdroid1@chromium.org, Sep 28 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3db1b3b3a7c8d164bab2f6501b8c44685b9b8df0

commit 3db1b3b3a7c8d164bab2f6501b8c44685b9b8df0
Author: Aaron Gable <agable@chromium.org>
Date: Thu Sep 28 03:17:10 2017

Update DEPS to reference new catapult repo location

R=sullivan@chromium.org

Bug:  731091 
Change-Id: Ie48518b6fdc4d9e5b671f117943978ee18ed5d0d
Reviewed-on: https://chromium-review.googlesource.com/688742
Reviewed-by: Annie Sullivan <sullivan@chromium.org>
Commit-Queue: Aaron Gable <agable@chromium.org>
Cr-Commit-Position: refs/heads/master@{#504882}
[modify] https://crrev.com/3db1b3b3a7c8d164bab2f6501b8c44685b9b8df0/DEPS

Project Member

Comment 42 by bugdroid1@chromium.org, Sep 28 2017

The following revision refers to this bug:
  https://webrtc.googlesource.com/src.git/+/78ba000d7e71554981c0caa4c4876181840af84e

commit 78ba000d7e71554981c0caa4c4876181840af84e
Author: Oleh Prypin <oprypin@webrtc.org>
Date: Thu Sep 28 13:27:39 2017

Change DEPS URL for catapult to match Chromium

See https://chromium-review.googlesource.com/688742

Bug:  chromium:731091 
Change-Id: I5904e87ac76b08bd3e71dff5ba791dc17de7240f
Reviewed-on: https://webrtc-review.googlesource.com/4424
Reviewed-by: Henrik Kjellander <kjellander@webrtc.org>
Commit-Queue: Oleh Prypin <oprypin@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#20016}
[modify] https://crrev.com/78ba000d7e71554981c0caa4c4876181840af84e/DEPS

Project Member

Comment 43 by bugdroid1@chromium.org, Sep 29 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/879ac56890d419b9ee82603f2aa587c56be276e3

commit 879ac56890d419b9ee82603f2aa587c56be276e3
Author: Michael Achenbach <machenbach@chromium.org>
Date: Fri Sep 29 15:08:12 2017

[build] Update DEPS to reference new catapult repo location

This ports:
https://chromium-review.googlesource.com/c/chromium/src/+/688742

TBR=agable@chromium.org

Bug:  chromium:731091 
Change-Id: I99b6bfcc769a5f3c429668c2d36518bcac80f3cf
Reviewed-on: https://chromium-review.googlesource.com/692634
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#48236}
[modify] https://crrev.com/879ac56890d419b9ee82603f2aa587c56be276e3/DEPS

I get the following error:
achuith@achuithz620:~/code/catapult/telemetry/telemetry$ git cl upload --gerrit
Credentials for the following hosts are required:
  github-review.com
  github.com
These are read from /usr/local/google/home/achuith/.gitcookies (or legacy /usr/local/google/home/achuith/.netrc)
You can (re)generate your credentials by visiting https://github-review.com/new-password

Are there some instructions on migrating my repo that were sent out that I missed?
Yep, instructions for that were included in one of the later messages on the internal PSA thread. Repeated here for your convenience:

"""
Please run in your repo:
`git config remote.origin.url https://chromium.googlesource.com/catapult`

This will update your repo to be pulling from (and pushing to) the new source of truth. For those of you who work on catapult from inside src/third_party/catapult, this update will happen for you when src.git updates its DEPS file to point at the new source of truth repo.
"""


Thanks Aaron, that worked!

This doc should probably be updated: https://catapult.gsrc.io/CONTRIBUTING.md

Sign in to add a comment